Info
Last Execution: 2026-08-19
| Package | Version |
|---|---|
| nnsight | 0.8 |
| Python | 3.12.13 |
| torch | 2.13.0+cu126 |
| transformers | 5.15.0 |
Massive Activations in Large Language Models¶
Introduction¶
📈 A handful of scalars inside a language model are enormous. Four of the 2048 dimensions of Llama-3.2-1B's residual stream sit at magnitude ≈ 200, while the median dimension never exceeds 0.45 anywhere in a 16,000-token corpus. Sun et al. (2024) name these massive activations: a scalar whose magnitude exceeds 100 and is at least 1,000× the median activation magnitude of its own hidden state.
Their paper makes four claims, and we reproduce each in turn:
- They exist, in a fixed place — a few feature dimensions, at a few token positions: the
first token, and delimiters such as
.and\n. - They are constants, not features. Their values barely depend on the input; they act as bias terms smuggled through the residual stream.
- They are indispensable. Zero them and the model collapses; replace them with their own corpus average and nothing happens.
- They cause the attention sink — the concentration of attention onto their tokens described by Xiao et al. (2023) in StreamingLLM.
The phenomenon is older than the name: Dettmers et al. (2022) hit the same dimensions while building LLM.int8() and called them emergent outlier features, channels whose dynamic range is wide enough to destroy a shared quantization scale.
We work on meta-llama/Llama-3.2-1B and finish on GPT-2, where the folklore claim ("GPT-2
doesn't really have them") fails in an instructive way. Two things separate this from a naive
reproduction, and both come out against the first hypothesis you would reach for: a
magnitude-matched control rather than only a count-matched one, and a test of whether the
damage is additive or a threshold.
📗 Primary paper: Sun, Chen, Kolter, Liu, Massive Activations in Large Language Models, COLM 2024.
Setup¶
If using Colab, install NNsight:
!pip install -U nnsight
try:
import google.colab
is_colab = True
except ImportError:
is_colab = False
if is_colab:
!pip install -U nnsight
from IPython.display import clear_output
import numpy as np
import torch
import nnsight
from nnsight import TransformersModel
import plotly.express as px
import plotly.io as pio
pio.renderers.default = "colab" if is_colab else "plotly_mimetype+notebook_connected+colab+notebook"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
/home/localjadenfk/miniconda3/envs/ndif2/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
We load with attn_implementation="eager" because section 4 needs attention probabilities, and
the default SDPA kernel never materialises them — self_attn.output[1] comes back as None,
silently. Eager costs nothing at 512 tokens, so we pay for it once rather than reloading later.
⚠️ On transformers 5 a decoder block's .output is a bare tensor: the residual stream after
that block. self_attn.output is still a tuple, (hidden_states, attention_weights). The old
blocks[i].output[0] idiom silently indexes the batch axis instead.
model = TransformersModel(
"meta-llama/Llama-3.2-1B",
task="text-generation",
attn_implementation="eager",
device_map=DEVICE,
dispatch=True,
)
layers = model.model.layers
n_layers = model.config.num_hidden_layers
d_model = model.config.hidden_size
bos = model.tokenizer.decode([model.tokenizer.bos_token_id])
clear_output()
print(f"Llama-3.2-1B: {n_layers} blocks, d_model={d_model}")
print(f"BOS token: {bos!r}")
Llama-3.2-1B: 16 blocks, d_model=2048 BOS token: '<|begin_of_text|>'
The corpus¶
Massive activations are a claim about typical text, so every number below is measured over a
corpus. We take documents from
NeelNanda/pile-10k and keep only those
long enough to fill the full 512-token context.
⚠️ That filter is load-bearing. Padding short documents would average every statistic — magnitudes, cross-entropy, the attention sink — over pad positions, and position 0 is exactly where the effect we are chasing lives. Selecting full-length documents means there is no padding anywhere in this notebook. The two splits use different seeds: locating the dimensions and testing them on the same text would let us fit noise.
from datasets import load_dataset
pile = load_dataset("NeelNanda/pile-10k", split="train")
SEQ_LEN = 512
def full_length_documents(n_docs, seed):
"""Tokenized documents that fill SEQ_LEN exactly, so no padding is ever needed."""
rng = np.random.RandomState(seed)
kept = []
for index in rng.permutation(len(pile)):
text = pile[int(index)]["text"]
if len(text) < SEQ_LEN * 5:
continue
ids = model.tokenizer(
text, return_tensors="pt", truncation=True, max_length=SEQ_LEN
).input_ids[0]
if ids.shape[0] < SEQ_LEN:
continue
kept.append(ids)
if len(kept) == n_docs:
break
return torch.stack(kept)
stats_tokens = full_length_documents(32, seed=0) # locate the phenomenon
held_tokens = full_length_documents(32, seed=1) # measure its causal effect
clear_output()
print(f"statistics corpus : {tuple(stats_tokens.shape)} = {stats_tokens.numel():,} tokens")
print(f"held-out corpus : {tuple(held_tokens.shape)} = {held_tokens.numel():,} tokens")
print(f"first three tokens of doc 0: {[model.tokenizer.decode([t]) for t in stats_tokens[0, :3]]}")
statistics corpus : (32, 512) = 16,384 tokens held-out corpus : (32, 512) = 16,384 tokens first three tokens of doc 0: ['<|begin_of_text|>', ' ', ' FILE']
1. Where they are¶
The first question is descriptive: over a real corpus, what is the largest magnitude each residual-stream dimension reaches, at which layer, and at which position?
One tracer.cache() per batch collects the embedding output and all 16 block outputs in a single
forward pass. modules=[...] is not optional in practice — caching everything would pull down
attention and MLP internals we have no use for — and device=None keeps the tensors on the GPU,
where the reduction is cheap, instead of paying a ~70 MB copy per batch. The torch.no_grad()
matters too: a read-only trace still builds an autograd graph otherwise.
The cache is keyed by module path, and every module handle carries its own key as .path — so
cache[layers[i].path] is the one form that works unchanged when we swap Llama for GPT-2 in
section 5, where the same modules live at transformer.h instead of model.layers.
cached_modules = [model.model.embed_tokens] + [layers[i] for i in range(n_layers)]
n_states = n_layers + 1 # embedding output, then one per block
max_abs = torch.zeros(n_states, SEQ_LEN, d_model, device=DEVICE) # max |x| over documents
bos_sum = torch.zeros(n_layers, d_model, device=DEVICE) # signed sum at position 0
median_abs = [] # median |x| per hidden state
BATCH = 4
for start in range(0, len(stats_tokens), BATCH):
batch = stats_tokens[start:start + BATCH].to(DEVICE)
with torch.no_grad(), model.trace(batch) as tracer:
cache = tracer.cache(modules=cached_modules, device=None)
hidden = torch.stack(
[cache[model.model.embed_tokens.path].output.float()]
+ [cache[layers[i].path].output.float() for i in range(n_layers)]
) # (n_states, batch, seq, d_model)
max_abs = torch.maximum(max_abs, hidden.abs().amax(dim=1))
bos_sum += hidden[1:, :, 0, :].sum(dim=1)
median_abs.append(hidden.abs().flatten(1).median(dim=1).values)
del cache, hidden
median_abs = torch.stack(median_abs).mean(0)
bos_mean = bos_sum / len(stats_tokens) # the corpus-average residual at BOS
print(f"peak GPU memory: {torch.cuda.max_memory_allocated() / 1e9:.2f} GB")
peak GPU memory: 3.34 GB
Ranking the dimensions halfway through the network, where the phenomenon is fully established:
MID = n_layers // 2 + 1 # index into max_abs: hidden state after block 8
per_dim_max = max_abs[MID].amax(dim=0) # (d_model,) max over positions and documents
median_dim_max = per_dim_max.median().item()
order = per_dim_max.argsort(descending=True)
print(f"after block {MID - 1}: median |activation| = {median_abs[MID]:.4f}"
f" median per-dimension max = {median_dim_max:.3f}\n")
print(f"{'dim':>6} {'max |x|':>10} {'at pos':>8} {'x median-dim':>14} {'x median act':>14}")
for dim in order[:6].tolist():
position = int(max_abs[MID, :, dim].argmax())
print(f"{dim:>6} {per_dim_max[dim]:>10.1f} {position:>8} "
f"{per_dim_max[dim] / median_dim_max:>13.0f}x {per_dim_max[dim] / median_abs[MID]:>13.0f}x")
MASSIVE = order[:4].tolist()
NEXT_LARGEST = order[4:8].tolist()
print(f"\nmassive dims : {MASSIVE}")
print(f"next-largest dims : {NEXT_LARGEST} (the magnitude-matched-by-rank control)")
after block 8: median |activation| = 0.0662 median per-dimension max = 0.447 dim max |x| at pos x median-dim x median act 400 217.0 0 485x 3277x 698 210.0 0 470x 3171x 2029 208.0 0 465x 3141x 1159 200.0 0 447x 3020x 2023 45.5 0 102x 687x 1107 34.5 0 77x 521x massive dims : [400, 698, 2029, 1159] next-largest dims : [2023, 1107, 1314, 781] (the magnitude-matched-by-rank control)
Four dimensions — 400, 698, 2029, 1159 — at magnitude ~200, all at position 0, and then a cliff: the fifth-ranked dimension is 45. Against Sun et al.'s definition these clear both bars, |x| > 100 and > 3,000× the median activation of their hidden state. Plotting every dimension on a log axis shows the gap between "massive" and "merely large".
fig = px.scatter(
x=np.arange(d_model),
y=per_dim_max.cpu().numpy(),
log_y=True,
labels=dict(x="residual-stream dimension", y="max |activation| over the corpus"),
title=f"Llama-3.2-1B, hidden state after block {MID - 1}: every dimension's peak magnitude",
height=420,
)
fig.update_traces(marker=dict(size=3))
fig.add_hline(y=median_dim_max, line_dash="dot",
annotation_text=f"median dimension ({median_dim_max:.2f})")
fig.show()
Four points at the top, one at 45, and 2043 dimensions in a band below 5 — two populations, not a heavy tail. Reducing over dimensions instead gives the other half of "where": the largest activation anywhere in the residual stream, per layer and position.
POSITIONS = 24
position_profile = max_abs.amax(dim=-1)[:, :POSITIONS].cpu().numpy()
fig = px.imshow(
np.log10(position_profile),
color_continuous_scale="Viridis",
labels=dict(x="token position", y="hidden state", color="log10 max |x|"),
y=["embed"] + [f"block {i}" for i in range(n_layers)],
title="Largest activation anywhere in the residual stream, by layer and position",
height=480,
)
fig.show()
print(f"after block {MID - 1}: max |x| at position 0 = {max_abs[MID, 0, :].max():.1f}")
print(f" max |x| at positions 1-511 = {max_abs[MID, 1:, :].max():.1f}")
after block 8: max |x| at position 0 = 217.0
max |x| at positions 1-511 = 3.6
A single bright column at position 0 from block 1 onward; everywhere else the residual stream stays
below 4. Llama always prepends <|begin_of_text|>, so position 0 is the same token in all 32
documents — which raises the question of whether this belongs to the BOS token or to the
first slot. One forward pass without the special token settles it.
for text in ["The capital of France is Paris.", "def fibonacci(n): return n if n < 2 else"]:
ids = model.tokenizer(text, return_tensors="pt", add_special_tokens=False).input_ids.to(DEVICE)
with torch.no_grad(), model.trace(ids) as tracer:
cache = tracer.cache(modules=[layers[MID - 1]])
hidden = cache[layers[MID - 1].path].output[0].float()
print(f"first token {model.tokenizer.decode(ids[0, :1])!r:>8}"
f" massive dims at position 0: {np.round(hidden[0, MASSIVE].cpu().numpy(), 0)}"
f" max |x| at positions > 0: {hidden[1:, MASSIVE].abs().max():.1f}")
first token 'The' massive dims at position 0: [ 446. -430. -426. -400.] max |x| at positions > 0: 0.5 first token 'def' massive dims at position 0: [ 360. -348. -344. -324.] max |x| at positions > 0: 0.5
Without BOS they do not disappear — they migrate to whatever token is first, and get bigger
(446 and 360, against 217 with the real BOS). This is a property of the first position, not of the
<|begin_of_text|> embedding. Sun et al. add a second home that Llama's dedicated start token
hides: delimiter tokens (., \n) in models without one.
2. When they appear, and when they vanish¶
A block's output is its input plus what its attention and MLP wrote, so caching self_attn, mlp
and the block itself for all 16 layers reads that decomposition off directly and attributes the
write to a component.
⚠️ Statements in a trace body must appear in the order the model executes them — asking for
layers[i].output before layers[i].self_attn.output raises OutOfOrderError. tracer.cache()
avoids the question by declaring everything up front, which is also why it has to be the first
statement in the trace.
probe = stats_tokens[:2].to(DEVICE)
with torch.no_grad(), model.trace(probe) as tracer:
cache = tracer.cache(
modules=[layers[i].self_attn for i in range(n_layers)]
+ [layers[i].mlp for i in range(n_layers)]
+ [layers[i] for i in range(n_layers)],
device=None,
)
def at_bos(values):
return values[0, 0, MASSIVE].float().cpu().numpy()
row = lambda v: " ".join(f"{x:+8.1f}" for x in v)
print(f"{'block':>5} | {'attention writes':^35} | {'MLP writes':^35} | {'residual after':^35}")
print("-" * 121)
trajectory = []
for i in range(n_layers):
attn = at_bos(cache[layers[i].self_attn.path].output[0])
mlp = at_bos(cache[layers[i].mlp.path].output)
resid = at_bos(cache[layers[i].path].output)
trajectory.append(resid)
if i < 3 or i > n_layers - 4:
print(f"{i:>5} | {row(attn)} | {row(mlp)} | {row(resid)}")
elif i == 3:
print(f"{'...':>5} | {'(all below 0.2)':^35} | {'(all below 1.2)':^35} | {'(unchanged)':^35}")
block | attention writes | MLP writes | residual after
-------------------------------------------------------------------------------------------------------------------------
0 | +0.0 +0.0 +0.0 +0.0 | +0.2 +0.0 +0.1 +0.0 | +0.2 +0.1 +0.1 +0.1
1 | +0.1 +0.0 +0.0 +0.0 | +219.0 -211.0 -209.0 -196.0 | +219.0 -211.0 -209.0 -196.0
2 | +0.0 -0.0 -0.0 -0.0 | +0.2 -0.2 -0.2 -0.3 | +219.0 -211.0 -209.0 -196.0
... | (all below 0.2) | (all below 1.2) | (unchanged)
13 | -0.1 +0.0 +0.0 -0.4 | +0.1 +0.1 +0.1 +0.3 | +219.0 -211.0 -209.0 -202.0
14 | +0.0 +0.0 -0.0 +0.1 | +2.5 -2.6 -2.7 -1.5 | +222.0 -214.0 -212.0 -204.0
15 | +0.0 +0.0 +0.0 +0.5 | -80.0 +109.0 +92.0 +71.0 | +142.0 -105.0 -120.0 -132.0
The whole thing is written by the MLP of block 1, in one step: +219, −211, −209, −196, out of a residual stream that was at 0.2 the layer before. Attention contributes nothing anywhere — below 0.2 at every layer — and blocks 2 through 13 add a few tenths each.
Then the last block's MLP deletes 40–50% of it: −80, +109, +92, +71, each opposing the value it acts on. This is a bias written once, carried, and partly cleared before the unembedding, not a feature being maintained.
fig = px.line(
np.stack(trajectory),
labels=dict(index="block", value="activation at BOS", variable="dimension"),
title="The four massive dimensions across depth (position 0)",
markers=True,
height=420,
)
fig.for_each_trace(lambda t: t.update(name=f"dim {MASSIVE[int(t.name)]}"))
fig.show()
Flat lines are the story: four values set at block 1, held to three significant figures for twelve blocks, partly retracted at block 15. If the network cared about them as a computed quantity we would expect it to defend them — erasing them at position 0 after block L tests that.
for L in [0, 1, 2, 4]:
with torch.no_grad(), model.trace(probe) as tracer:
cache = tracer.cache(modules=[layers[i] for i in range(L, n_layers)], device=None)
layers[L].output[:, 0, MASSIVE] = 0
after = [f"block {i}: {np.round(at_bos(cache[layers[i].path].output), 1)}"
for i in [L + 1, min(L + 2, n_layers - 1), n_layers - 1]]
print(f"erased after block {L} -> " + " ".join(after))
erased after block 0 -> block 1: [ 219. -211. -209. -196.] block 2: [ 219. -211. -209. -196.] block 15: [ 142. -105. -120. -132.] erased after block 1 -> block 2: [ 0.6 -0.6 -0.6 -0.8] block 3: [ 0.7 -0.8 -0.7 -1.1] block 15: [-64.5 86. 70. 56. ]
erased after block 2 -> block 3: [ 0.5 -0.7 -0.5 -0.8] block 4: [ 0.5 -0.9 -0.5 -1.2] block 15: [-61.2 83. 67. 54.2] erased after block 4 -> block 5: [ 0.2 -0.2 -0.4 -1.3] block 6: [ 0.4 -0.5 -0.5 -1.8] block 15: [-372. 372. 362. 356.]
Erase after block 0 and block 1 writes them straight back — the write is unconditional. Erase after block 1, 2 or 4 and no layer ever restores them; they stay under |3| for the rest of the network. There is no error correction here.
The last row shows block 15 is running a fixed subtraction rather than regulating anything: with nothing left to delete it fires anyway and drives the same dimensions to −372, +372, +362, +356.
3. What they do: the causal test¶
Everything so far is correlational. The causal question is what the model loses when these four numbers are removed — and, the part that decides whether the answer means anything, how that compares against controls.
The metric is next-token cross-entropy on the held-out corpus, reported as ΔCE against the clean run, plus top-1 agreement: the fraction of positions where the ablated model's argmax still matches the clean model's. Cross-entropy alone can move on a recalibration; agreement says whether the model is still doing the same thing.
The intervention is a Python for loop inside the trace — every block's output gets the same edit,
all applied in one forward pass, with no hook registry and nothing to clean up. We do not batch
the conditions themselves with tracer.invoke here: the metric is a full distribution over 512
positions and a 128k vocabulary, so each condition costs about half a gigabyte of logits. Batch the
axis that is cheap, which is documents.
def evaluate(edit=None, batch_size=2):
"""Cross-entropy and argmax over the held-out corpus, applying `edit(block)` in every block."""
nll_chunks, argmax_chunks = [], []
for start in range(0, len(held_tokens), batch_size):
batch = held_tokens[start:start + batch_size].to(DEVICE)
with torch.no_grad(), model.trace(batch) as tracer:
if edit is not None:
for block in range(n_layers):
edit(block)
log_probs = torch.log_softmax(model.output.logits[:, :-1].float(), dim=-1)
nll = (-log_probs.gather(-1, batch[:, 1:].unsqueeze(-1)).squeeze(-1)).save()
top1 = log_probs.argmax(dim=-1).save()
nll_chunks.append(nll.cpu())
argmax_chunks.append(top1.cpu())
torch.cuda.empty_cache()
return torch.cat(nll_chunks), torch.cat(argmax_chunks)
clean_nll, clean_top1 = evaluate()
clean_ce = clean_nll.mean()
print(f"clean: CE {clean_ce:.4f} perplexity {clean_ce.exp():.2f}")
results = {}
def report(name, edit):
nll, top1 = evaluate(edit)
results[name] = dict(
ce=nll.mean().item(),
dce=(nll.mean() - clean_ce).item(),
ppl=nll.mean().exp().item(),
agree=(top1 == clean_top1).float().mean().item(),
)
r = results[name]
print(f"{name:<40} dCE {r['dce']:>+9.4f} ppl {r['ppl']:>10.2f} top-1 agree {r['agree']:.3f}")
# Anchor: write each block's own value back into itself. If the intervention machinery is
# faithful this must be bit-exact, and drift here would contaminate every number below.
def self_assign(block):
layers[block].output[:, :, MASSIVE] = layers[block].output[:, :, MASSIVE]
report("no-op (self-assignment)", self_assign)
clean: CE 2.2749 perplexity 9.73
no-op (self-assignment) dCE +0.0000 ppl 9.73 top-1 agree 1.000
ΔCE +0.0000 and agreement 1.000 — bit-exact, so anything below is the edit and not the tooling.
Now zero the four massive dimensions at every position in every block output, against three controls at matched count: the four next-largest dimensions; three sets of four random dimensions; and the massive dimensions replaced by their corpus mean at BOS, which is Sun et al.'s test of whether they carry input-dependent information at all.
rng = np.random.RandomState(0)
pool = [d for d in range(d_model) if d not in MASSIVE]
RANDOM_SETS = [sorted(int(d) for d in rng.choice(pool, len(MASSIVE), replace=False)) for _ in range(3)]
def zero(dims, positions=slice(None)):
def edit(block):
layers[block].output[:, positions, dims] = 0
return edit
def set_to_corpus_mean(block):
layers[block].output[:, 0, MASSIVE] = bos_mean[block, MASSIVE].to(layers[block].output.dtype)
report("ABLATE massive (all positions)", zero(MASSIVE))
report(f"ctrl: next-largest dims {NEXT_LARGEST}", zero(NEXT_LARGEST))
for i, dims in enumerate(RANDOM_SETS):
report(f"ctrl: 4 random dims, seed {i}", zero(dims))
report("ctrl: massive set to their corpus mean", set_to_corpus_mean)
report("ctrl: massive zeroed at positions > 0", zero(MASSIVE, slice(1, None)))
report("ABLATE massive (position 0 only)", zero(MASSIVE, 0))
ABLATE massive (all positions) dCE +6.7852 ppl 8605.43 top-1 agree 0.060
ctrl: next-largest dims [2023, 1107, 1314, 781] dCE +0.0837 ppl 10.58 top-1 agree 0.851
ctrl: 4 random dims, seed 0 dCE +0.0075 ppl 9.80 top-1 agree 0.946
ctrl: 4 random dims, seed 1 dCE +0.0123 ppl 9.85 top-1 agree 0.940
ctrl: 4 random dims, seed 2 dCE +0.0066 ppl 9.79 top-1 agree 0.951
ctrl: massive set to their corpus mean dCE -0.0000 ppl 9.73 top-1 agree 0.985
ctrl: massive zeroed at positions > 0 dCE +0.0309 ppl 10.03 top-1 agree 0.915
ABLATE massive (position 0 only) dCE +6.4199 ppl 5971.74 top-1 agree 0.065
labels = ["massive<br>(all pos)", "next-largest<br>4 dims", "random<br>4 dims (mean of 3)",
"corpus mean<br>(pos 0)", "massive<br>at pos > 0", "massive<br>(pos 0 only)"]
random_mean = np.mean([results[f"ctrl: 4 random dims, seed {i}"]["dce"] for i in range(3)])
values = [results["ABLATE massive (all positions)"]["dce"],
results[f"ctrl: next-largest dims {NEXT_LARGEST}"]["dce"],
random_mean,
results["ctrl: massive set to their corpus mean"]["dce"],
results["ctrl: massive zeroed at positions > 0"]["dce"],
results["ABLATE massive (position 0 only)"]["dce"]]
fig = px.bar(
x=labels, y=np.maximum(values, 1e-4), log_y=True,
labels=dict(x="", y="ΔCE (nats, log scale)"),
title="Removing 4 of 2048 dimensions, against count-matched controls",
height=430,
)
fig.show()
print(f"massive / random-dimension ratio: {values[0] / random_mean:,.0f}x")
print(f"massive / next-largest ratio : {values[0] / values[1]:,.0f}x")
massive / random-dimension ratio: 769x massive / next-largest ratio : 81x
Zeroing 0.2% of the residual stream costs +6.79 nats — perplexity 9.7 → 8,600, with the model agreeing with its own clean argmax on 6% of tokens. Four random dimensions cost +0.009, roughly 770× less; the next-largest four, genuine outliers at |x| = 45 and 35, cost +0.084.
Two controls say more than the headline. Setting the massive dimensions to their corpus mean is free (ΔCE −0.0000, agreement 0.985) while zeroing them is catastrophic: the model reads nothing from them that varies with the input, which is why "bias term" is the right word and not "feature". And almost all the damage is at position 0 — zeroing the same dimensions everywhere except position 0 costs +0.030.
3.1 The controls that complicate the story¶
A factor of 770 is where a naive write-up stops, and it should not, because the obvious alternative is still standing: perhaps what matters is not these dimensions but that the BOS residual is enormous. Its norm at mid-depth is 426 against about 10 for an ordinary token, and 96% of that norm is these four numbers.
Two interventions separate "big in these directions" from "big". Norm-preserving deletion zeroes the four and rescales the remaining 2044 so that ‖h‖ at position 0 is exactly what it was — harmless if the model only needs a large vector there. Permutation moves the four values onto four random dimensions: same magnitudes, same norm, different directions. That is a genuine magnitude-matched control rather than a count-matched one, and we run it with three different draws of target dimensions, because there is no reason to expect one draw to be representative.
PERMUTE_SETS = [[int(d) for d in np.random.RandomState(seed).choice(pool, len(MASSIVE), replace=False)]
for seed in [7, 8, 9]]
def norm_preserving(block):
hidden = layers[block].output
bos = hidden[:, 0, :].float()
target_norm = bos.norm(dim=-1, keepdim=True)
stripped = bos.clone()
stripped[:, MASSIVE] = 0
rescaled = stripped * (target_norm / stripped.norm(dim=-1, keepdim=True))
layers[block].output[:, 0, :] = rescaled.to(hidden.dtype)
def permute_onto_random(targets):
def edit(block):
hidden = layers[block].output
bos = hidden[:, 0, :].float()
values = bos[:, MASSIVE].clone()
moved = bos.clone()
moved[:, MASSIVE] = 0
moved[:, targets] = moved[:, targets] + values
layers[block].output[:, 0, :] = moved.to(hidden.dtype)
return edit
def scale(factor):
def edit(block):
hidden = layers[block].output
layers[block].output[:, 0, MASSIVE] = (hidden[:, 0, MASSIVE].float() * factor).to(hidden.dtype)
return edit
with torch.no_grad(), model.trace(held_tokens[:2].to(DEVICE)) as tracer:
cache = tracer.cache(modules=[layers[MID - 1]], device=None)
bos_norm = cache[layers[MID - 1].path].output[:, 0, :].float().norm(dim=-1).mean()
print(f"clean ||h|| at position 0, after block {MID - 1}: {bos_norm:.1f}\n")
report("zero massive, norm preserved", norm_preserving)
for i, targets in enumerate(PERMUTE_SETS):
report(f"permute massive onto 4 random dims, seed {i}", permute_onto_random(targets))
for factor in [0.5, 2.0, 4.0]:
report(f"scale massive x{factor}", scale(factor))
clean ||h|| at position 0, after block 8: 426.4
zero massive, norm preserved dCE +9.2753 ppl 103803.88 top-1 agree 0.008
permute massive onto 4 random dims, seed 0 dCE +6.6214 ppl 7305.23 top-1 agree 0.038
permute massive onto 4 random dims, seed 1 dCE +6.2695 ppl 5138.15 top-1 agree 0.067
permute massive onto 4 random dims, seed 2 dCE +5.4483 ppl 2260.24 top-1 agree 0.133
scale massive x0.5 dCE +4.5940 ppl 961.94 top-1 agree 0.172
scale massive x2.0 dCE +0.3843 ppl 14.28 top-1 agree 0.706
scale massive x4.0 dCE +0.4679 ppl 15.53 top-1 agree 0.674
It is not a norm effect. Preserving the BOS norm while deleting the massive dimensions is worse than deleting them — +9.28 against +6.42, perplexity 10⁵, agreement 0.008. Scaling the 2044 ordinary components up ~20× to make good the missing norm is more destructive than leaving the position small. The model does not need BOS to be big; it needs BOS to be big in those four directions.
Direction is what matters, but the magnitude-matched control cannot put a number on it. Moving the identical values onto four random dimensions — same magnitudes, exact same norm — costs +5.45, +6.27 and +6.62 across the three draws, straddling the +6.42 of deleting them outright. Random directions plainly do not substitute for the real ones, and that is the finding. What this control cannot do is say how much better the real ones are: any large perturbation of the BOS residual is already catastrophic, and the seed-to-seed spread is wider than the difference we would be trying to measure. Reporting the count-matched 770× as though it were the specificity of these four directions would overstate the case by orders of magnitude.
The scaling sweep adds an asymmetry: halving them costs +4.59 nats, doubling +0.39, quadrupling +0.47. A sink four times too large is tolerable; one two times too small is not.
3.2 A threshold, not a sum¶
Four dimensions cost 6.4 nats. If each carried its own share, one should cost about 1.6. Removing them cumulatively tests that.
for k in range(1, len(MASSIVE) + 1):
report(f"zero the top-{k} massive dims (pos 0)", zero(MASSIVE[:k], 0))
print(f"\nfour independent contributions would predict roughly "
f"{4 * results['zero the top-1 massive dims (pos 0)']['dce']:.3f} nats")
print(f"the four taken together cost "
f"{results['zero the top-4 massive dims (pos 0)']['dce']:.3f} nats")
zero the top-1 massive dims (pos 0) dCE +0.0130 ppl 9.85 top-1 agree 0.940
zero the top-2 massive dims (pos 0) dCE +0.0623 ppl 10.35 top-1 agree 0.874
zero the top-3 massive dims (pos 0) dCE +0.2840 ppl 12.92 top-1 agree 0.759
zero the top-4 massive dims (pos 0) dCE +6.4199 ppl 5971.74 top-1 agree 0.065 four independent contributions would predict roughly 0.052 nats the four taken together cost 6.420 nats
+0.013, +0.062, +0.284, +6.42. The fourth dimension costs 23× what the first three cost together, and 120× what four independent contributions would predict. This is a threshold, not a sum.
Read next to the scaling result it pins the invariant down. Leaving one dimension at full strength (|x| = 200, three zeroed) keeps perplexity at 12.9; leaving all four at half strength — the same total mass at that position — gives 962. What the network requires is that some dimension at position 0 stays large in absolute terms, not that the position carries a given amount of mass.
4. Massive activations and the attention sink¶
Xiao et al. (2023) observed that transformers dump a large fraction of their attention onto the first few tokens regardless of content, and built StreamingLLM around never evicting them. Sun et al.'s claim is that the two phenomena are one: the massive activations make their token's key a fixed target. Reading the probabilities is what the eager attention implementation was for.
SINK_LEN, SINK_DOCS = 256, 4
sink = np.zeros((n_layers, SINK_LEN))
for doc in range(SINK_DOCS):
ids = stats_tokens[doc:doc + 1, :SINK_LEN].to(DEVICE)
with torch.no_grad(), model.trace(ids) as tracer:
cache = tracer.cache(
modules=[layers[i].self_attn for i in range(n_layers)],
device=None, dtype=torch.float32,
)
for i in range(n_layers):
weights = cache[layers[i].self_attn.path].output[1] # (1, heads, query, key)
sink[i] += weights[0].mean(dim=(0, 1)).cpu().numpy() # attention received per key
sink /= SINK_DOCS
fig = px.imshow(
np.log10(sink[:, :32]),
color_continuous_scale="Viridis",
labels=dict(x="key position", y="block", color="log10 attention received"),
y=[f"block {i}" for i in range(n_layers)],
title="Mean attention received per key position (over heads, queries, 4 documents)",
height=460,
)
fig.show()
print(f"{'block':>6} {'attn to pos 0':>15} {'median position':>17} {'ratio':>10}")
for i in [0, 1, 4, 8, 12, 15]:
median = np.median(sink[i])
print(f"{i:>6} {sink[i][0]:>14.1%} {median:>17.5f} {sink[i][0] / median:>9.0f}x")
block attn to pos 0 median position ratio
0 38.5% 0.00223 172x
1 75.0% 0.00091 822x
4 64.8% 0.00130 497x
8 49.7% 0.00176 282x
12 68.9% 0.00112 616x
15 64.0% 0.00132 484x
Position 0 receives 38–75% of all attention at every layer, against 0.001–0.002 for the median position — a 170–820× sink, exactly where the massive activations are, switching on at block 1, exactly when they are written.
On Llama this alignment is suggestive rather than decisive, and it is worth saying why: only one
position carries signal, so a rank correlation across positions would be computed over 255 near-zero
values and would measure noise. The ratio is the honest statistic. The decisive test needs a model
whose massive activations sit at a content-dependent index — Pythia puts them on the first \n
or ., at a different position per document, and there the top sink position tracks the top
massive-activation position document by document. What section 3 does add is the direction of the
arrow: deleting the massive activations is what breaks the model, at that one position.
5. Cross-family contrast: GPT-2¶
The folklore is that GPT-2 does not have massive activations. Re-running section 1 on it says otherwise, and the way the claim fails is more interesting than the claim. We free Llama first, to keep the notebook inside a Colab T4.
⚠️ The corpus has to be rebuilt, not reused. stats_tokens holds Llama token ids, which run up to
128k; feeding them to GPT-2's 50,257-entry embedding is an out-of-bounds index, and on CUDA that
surfaces as a device-side assert several operations later rather than as a clear error.
import gc
del model, layers, cache, max_abs, bos_sum, bos_mean
gc.collect()
torch.cuda.empty_cache()
model = TransformersModel("gpt2", task="text-generation", device_map=DEVICE, dispatch=True)
layers = model.transformer.h
n_layers = model.config.n_layer
d_model = model.config.n_embd
# Token ids belong to a tokenizer, not to a corpus: Llama's run to 128k and GPT-2's vocabulary
# stops at 50,257. `full_length_documents` reads the global `model`, so re-running it now
# retokenizes the same Pile documents for GPT-2.
stats_tokens = full_length_documents(32, seed=0)
held_tokens = full_length_documents(32, seed=1)
clear_output()
print(f"gpt2: {n_layers} blocks, d_model={d_model}, vocab={model.config.vocab_size}")
print(f"first token of doc 0: {model.tokenizer.decode(stats_tokens[0, :1])!r} (no BOS is prepended)")
gpt2: 12 blocks, d_model=768, vocab=50257 first token of doc 0: ' ' (no BOS is prepended)
gpt2_max_abs = torch.zeros(n_layers + 1, SEQ_LEN, d_model, device=DEVICE)
gpt2_median_abs = []
for start in range(0, len(stats_tokens), BATCH):
batch = stats_tokens[start:start + BATCH].to(DEVICE)
with torch.no_grad(), model.trace(batch) as tracer:
cache = tracer.cache(
modules=[model.transformer.wte] + [layers[i] for i in range(n_layers)], device=None
)
hidden = torch.stack(
[cache[model.transformer.wte.path].output.float()]
+ [cache[layers[i].path].output.float() for i in range(n_layers)]
)
gpt2_max_abs = torch.maximum(gpt2_max_abs, hidden.abs().amax(dim=1))
gpt2_median_abs.append(hidden.abs().flatten(1).median(dim=1).values)
del cache, hidden
gpt2_median_abs = torch.stack(gpt2_median_abs).mean(0)
GPT2_MID = n_layers // 2 + 1
gpt2_per_dim = gpt2_max_abs[GPT2_MID].amax(dim=0)
gpt2_median_dim = gpt2_per_dim.median().item()
gpt2_order = gpt2_per_dim.argsort(descending=True)
print(f"after block {GPT2_MID - 1}: median |activation| = {gpt2_median_abs[GPT2_MID]:.3f}"
f" median per-dimension max = {gpt2_median_dim:.2f}\n")
for dim in gpt2_order[:4].tolist():
position = int(gpt2_max_abs[GPT2_MID, :, dim].argmax())
print(f" dim {dim:>4} max |x| = {gpt2_per_dim[dim]:>7.0f} at pos {position}"
f" {gpt2_per_dim[dim] / gpt2_median_dim:>5.0f}x median dim")
GPT2_MASSIVE = gpt2_order[:2].tolist()
GPT2_NEXT = gpt2_order[2:4].tolist()
after block 6: median |activation| = 1.534 median per-dimension max = 10.20 dim 447 max |x| = 2984 at pos 0 293x median dim dim 138 max |x| = 807 at pos 0 79x median dim dim 378 max |x| = 68 at pos 0 7x median dim dim 373 max |x| = 43 at pos 74 4x median dim
GPT-2 has them, and its outliers are the largest of the two models in absolute terms: dim 447 peaks near 2,990 against Llama's 217. The claim survives because GPT-2's residual stream is uniformly hotter — its median dimension peaks at 10.2 where Llama's peaks at 0.45 — so the ratio is 293× rather than 485×. Absolute magnitude and relative outlier-ness point opposite ways, and only the ratio means anything.
GPT-2 also has no BOS token, so position 0 is a different token in every document. If these values are constants rather than features, that should not matter.
with torch.no_grad(), model.trace(stats_tokens[:8].to(DEVICE)) as tracer:
cache = tracer.cache(modules=[layers[GPT2_MID - 1]], device=None)
bos_values = cache[layers[GPT2_MID - 1].path].output[:, 0, :].float()
print(f"{'doc':>4} {'first token':>14} " + " ".join(f"dim {d}" for d in GPT2_MASSIVE))
for doc in range(8):
token = model.tokenizer.decode(stats_tokens[doc, :1])
values = " ".join(f"{bos_values[doc, d]:+8.0f}" for d in GPT2_MASSIVE)
print(f"{doc:>4} {token!r:>14} {values}")
spread = bos_values[:, GPT2_MASSIVE[0]]
print(f"\ndim {GPT2_MASSIVE[0]} across 8 different first tokens: "
f"{spread.min():.0f} to {spread.max():.0f} "
f"(+/-{100 * (spread.max() - spread.min()) / 2 / spread.mean():.1f}%)")
doc first token dim 447 dim 138 0 ' ' +2887 +756 1 '<' +2932 +788 2 'Let' +2914 +774 3 'Q' +2968 +801 4 'Menu' +2932 +781 5 'e' +2967 +801 6 'Robert' +2928 +780 7 '\n' +2954 +794 dim 447 across 8 different first tokens: 2887 to 2968 (+/-1.4%)
Eight different first tokens and the same two numbers to within ±1.4%. In a model with no dedicated start token that is a stronger statement of "fixed bias" than Llama's, where position 0 is always literally the same token. The causal test transfers too — two dimensions out of 768.
clean_nll, clean_top1 = evaluate()
clean_ce = clean_nll.mean()
results = {}
print(f"clean: CE {clean_ce:.4f} perplexity {clean_ce.exp():.2f}")
gpt2_pool = [d for d in range(d_model) if d not in GPT2_MASSIVE]
gpt2_rng = np.random.RandomState(0)
gpt2_random = [sorted(int(d) for d in gpt2_rng.choice(gpt2_pool, 2, replace=False)) for _ in range(3)]
report(f"ABLATE massive {GPT2_MASSIVE}", zero(GPT2_MASSIVE))
report(f"ctrl: next-largest {GPT2_NEXT}", zero(GPT2_NEXT))
for i, dims in enumerate(gpt2_random):
report(f"ctrl: 2 random dims, seed {i}", zero(dims))
clean: CE 3.1256 perplexity 22.77
ABLATE massive [447, 138] dCE +2.6092 ppl 309.44 top-1 agree 0.287
ctrl: next-largest [378, 373] dCE +0.0634 ppl 24.26 top-1 agree 0.880
ctrl: 2 random dims, seed 0 dCE +0.0049 ppl 22.89 top-1 agree 0.962
ctrl: 2 random dims, seed 1 dCE +0.0047 ppl 22.88 top-1 agree 0.958
ctrl: 2 random dims, seed 2 dCE +0.0068 ppl 22.93 top-1 agree 0.958
+2.61 nats for two dimensions out of 768, against +0.005 for two random ones — a factor of ~480. The anchor passes in GPT-2 as clearly as in Llama.
What does not transfer is the ordering. GPT-2's massive activations are 14× larger in absolute terms than Llama's and their causal effect is less than half (+2.61 against +6.79 nats; perplexity ×13.6 against ×880). Whatever makes these dimensions load-bearing is not their size, which is exactly what section 3.2 found from the other direction.
Caveats¶
- Two models, one corpus. 32 held-out Pile documents each. Sun et al. cover many more families and three corpora, and report the delimiter-token location that Llama's dedicated BOS hides. Read the cross-family contrast here as consistent with theirs, not as independent confirmation.
- Zero-ablation overstates importance. Setting a dimension to zero moves the model off its data manifold. Set-to-corpus-mean is the resample control that keeps it on, and that is the condition that comes back clean — which is the result, not a flaw. But +6.79 nats is an upper bound on "how much this matters", not an estimate of it.
- The magnitude-matched control has no headroom. Permuting the values onto random dimensions costs +5.45 to +6.62 nats against +6.42 for deleting them — the same ballpark, with a seed-to-seed spread wider than any difference between them. It establishes that random directions do not substitute; it cannot establish how much better the real directions are.
- Section 4 is correlational on Llama. With signal at exactly one position the rank statistic is uninformative and only the ratio is meaningful; the content-dependent test needs a model such as Pythia, whose massive activations move with the text.
- Numerics. Everything runs in bfloat16 with cross-entropy accumulated in float32. The no-op anchor is bit-exact, so the intervention path adds nothing, but ΔCE below ~0.001 is not meaningful.
Conclusion¶
🎉 Four numbers out of 2048, at one token position, written by one MLP in one step and carried unchanged through most of the network, are worth 6.8 nats of cross-entropy. Replacing them with their own average costs nothing; deleting them destroys the model; deleting them while preserving the norm is worse still; and three of the four can go before much happens at all.
The methodological content is in the controls. Count-matched controls give a 770× gap and a comfortable story; magnitude-matched controls land in the same range as deletion itself and give a much more careful one. The BOS residual is fragile in every direction, so the specificity of these four directions is real — nothing else substitutes — but this experiment cannot put a size on it.
In nnsight the whole study is two primitives: tracer.cache(modules=[...]) for the descriptive
half, and a for loop of assignments into layers[i].output inside a single trace for the causal
half. Each condition above — zeroing, scaling, permuting, rescaling to preserve a norm — is a
two-line change to that loop, which is what made six controls affordable instead of two.
Related: Activation Patching for the localising form of this intervention, and Caching for the statistics-collection primitive.
References¶
- Sun, Chen, Kolter, Liu, Massive Activations in Large Language Models, COLM 2024 · project page · code
- Xiao, Tian, Chen, Han, Lewis, Efficient Streaming Language Models with Attention Sinks, ICLR 2024
- Dettmers, Lewis, Belkada, Zettlemoyer, LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, NeurIPS 2022
- Gao et al., The Pile: An 800GB Dataset of Diverse Text for Language Modeling, 2020 — via
NeelNanda/pile-10k