Info

Last Execution: 2026-08-19

Package	Version
nnsight	0.8
Python	3.12
torch	2.13.0+cu126
transformers	5.15.0
datasets	5.0.0

Under-trained Tokens: Fishing for Magikarp in GPT-2¶

Introduction¶

🎣 A tokenizer is fitted before the model is trained, usually on a different corpus. Anything that reached the merge table but not the training data leaves the model with a vocabulary entry it has essentially never seen — famously SolidGoldMagikarp, a Reddit username that survived into GPT-2's vocabulary and that GPT-3 could not repeat, spell, or acknowledge.

These are under-trained tokens, or glitch tokens, and the question we reproduce is Land & Bartolo's: can you find them from the weights alone, before running a single prompt? Their answer is a weight-based indicator computed from the unembedding matrix, followed by a behavioural check on the top candidates only. Six findings follow: the indicator; the popular embedding-norm heuristic being at chance; why formulas copied from interpretability papers do not reproduce against raw HuggingFace weights; behavioural confirmation against a frequency-matched control; overlap with the published lists; and what the residual stream does at a glitch token.

📗 Primary paper: Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models (Land & Bartolo, EMNLP 2024). The phenomenon was first described in SolidGoldMagikarp (plus, prompt generation) (Rumbelow & Watkins, 2023).

⚠️ This mini-paper is unusual for the site in that most of it reads weights, not activations — there is no trace at all until Finding 4. In nnsight a model's parameters are the wrapped module's own nn.Parameters, reached by walking the envoy exactly as you would walk the HuggingFace module: model.transformer.wte.weight is the embedding matrix, full stop.

Setup¶

If using Colab, install NNsight and datasets (we need a small corpus to build the control set):

!pip install -U nnsight datasets

In [1]:

try:
    import google.colab
    is_colab = True
except ImportError:
    is_colab = False

if is_colab:
    !pip install -U nnsight datasets

try: import google.colab is_colab = True except ImportError: is_colab = False if is_colab: !pip install -U nnsight datasets

In [2]:

import warnings
warnings.filterwarnings("ignore", message=".*IProgress.*")

from IPython.display import clear_output
import numpy as np
import torch
import torch.nn.functional as F
import nnsight
from nnsight import TransformersModel

import plotly.express as px
import plotly.io as pio
pio.renderers.default = "colab" if is_colab else "plotly_mimetype+notebook_connected+colab+notebook"

import warnings warnings.filterwarnings("ignore", message=".*IProgress.*") from IPython.display import clear_output import numpy as np import torch import torch.nn.functional as F import nnsight from nnsight import TransformersModel import plotly.express as px import plotly.io as pio pio.renderers.default = "colab" if is_colab else "plotly_mimetype+notebook_connected+colab+notebook"

⚠️ Pass dispatch=True. A TransformersModel is built lazily, so until it dispatches every module lives on the meta device and a parameter read off it has the right shape, the right dtype, and no storage. norm, topk and argsort all succeed on a meta tensor and return another one, so the failure surfaces well below the mistake. An activation workload never notices, because the first trace dispatches; a weight workload hits it immediately.

In [3]:

model = TransformersModel("openai-community/gpt2", device_map="auto", dispatch=True)
clear_output()

tokenizer = model.tokenizer
V = model.config.vocab_size
print(model.transformer)
print(f"\nvocab size: {V}   dispatched: {model.dispatched}")

model = TransformersModel("openai-community/gpt2", device_map="auto", dispatch=True) clear_output() tokenizer = model.tokenizer V = model.config.vocab_size print(model.transformer) print(f"\nvocab size: {V} dispatched: {model.dispatched}")

GPT2Model(
  (wte): Embedding(50257, 768)
  (wpe): Embedding(1024, 768)
  (drop): Dropout(p=0.1, inplace=False)
  (h): ModuleList(
    (0-11): 12 x GPT2Block(
      (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True, bias=True)
      (attn): GPT2Attention(
        (c_attn): Conv1D()
        (c_proj): Conv1D()
        (attn_dropout): Dropout(p=0.1, inplace=False)
        (resid_dropout): Dropout(p=0.1, inplace=False)
      )
      (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True, bias=True)
      (mlp): GPT2MLP(
        (c_fc): Conv1D()
        (c_proj): Conv1D()
        (act): NewGELUActivation()
        (dropout): Dropout(p=0.1, inplace=False)
      )
    )
  )
  (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True, bias=True)
)

vocab size: 50257   dispatched: True

W_E is the input embedding (wte), W_U the unembedding (lm_head), and ln_f the final layer norm between them.

In [4]:

W_E = model.transformer.wte.weight
W_U = model.lm_head.weight
ln_g = model.transformer.ln_f.weight
ln_b = model.transformer.ln_f.bias

print(f"W_E {tuple(W_E.shape)}  {W_E.dtype}  device={W_E.device}  is_meta={W_E.is_meta}")
print(f"W_U {tuple(W_U.shape)}  lm_head bias: {model.lm_head.bias}")
print(f"tied (same storage)? {W_E.data_ptr() == W_U.data_ptr()}")

W_E   = W_E.detach().float()
W_U   = W_U.detach().float()
ln_g  = ln_g.detach().float()
ln_b  = ln_b.detach().float()

W_E = model.transformer.wte.weight W_U = model.lm_head.weight ln_g = model.transformer.ln_f.weight ln_b = model.transformer.ln_f.bias print(f"W_E {tuple(W_E.shape)} {W_E.dtype} device={W_E.device} is_meta={W_E.is_meta}") print(f"W_U {tuple(W_U.shape)} lm_head bias: {model.lm_head.bias}") print(f"tied (same storage)? {W_E.data_ptr() == W_U.data_ptr()}") W_E = W_E.detach().float() W_U = W_U.detach().float() ln_g = ln_g.detach().float() ln_b = ln_b.detach().float()

W_E (50257, 768)  torch.float32  device=cuda:0  is_meta=False
W_U (50257, 768)  lm_head bias: None
tied (same storage)? True

W_E.data_ptr() == W_U.data_ptr() is True: GPT-2 ties its input and output embeddings. They are one matrix, which turns out to be the whole explanation for Finding 2.

Finding 1: an unused token's output row gives it away¶

An input embedding row is touched only when its token appears in a batch. An output (unembedding) row is touched at every step, because the softmax pushes every non-target logit down everywhere. A token that is never a target therefore still gets a gradient — the same "you are not the answer" gradient as every other never-occurring token — so all unused rows drift in the same direction. Under-training shows up as direction, not magnitude.

Hence Land & Bartolo's indicator: name a few rows that are provably never targets, take their mean as the "unused" direction, and score every token by how closely its unembedding row aligns with it. No corpus, no labels, no forward pass. GPT-2 supplies the reference set for free — its byte-level BPE has a token per byte value, and bytes 0xF5–0xFF cannot occur in valid UTF-8, so those 11 rows can never have been a target.

⚠️ Two details decide whether this reproduces: score the unembedding, not the embedding, and fold ln_f's gain into it. The logit for token $i$ is $\langle \widehat{h},\, g \odot u_i \rangle + \langle b,\, u_i \rangle$, so the vector the residual stream sees is $g \odot u_i$, not $u_i$.

In [5]:

def bytes_to_unicode():
    """GPT-2's byte <-> printable-character map. transformers 5 no longer exports this."""
    bs = (list(range(ord("!"), ord("~") + 1))
          + list(range(ord("\xa1"), ord("\xac") + 1))
          + list(range(ord("\xae"), ord("\xff") + 1)))
    cs, n = bs[:], 0
    for byte in range(256):
        if byte not in bs:
            bs.append(byte)
            cs.append(256 + n)
            n += 1
    return dict(zip(bs, [chr(c) for c in cs]))

byte_to_char = bytes_to_unicode()

# bytes 0xF5-0xFF are illegal in UTF-8: these 11 rows are provably never a training target
REFERENCE = [tokenizer.convert_tokens_to_ids(byte_to_char[b]) for b in range(0xF5, 0x100)]
print("reference token ids:", REFERENCE)

W_U_folded = W_U * ln_g                       # fold the ln_f gain into the unembedding
u_ref = W_U_folded[REFERENCE].mean(0)         # the "unused" direction
indicator = F.cosine_similarity(W_U_folded, u_ref[None], dim=-1)

top_values, top_ids = indicator.topk(30)
print(f"\n{'rank':>4}  {'id':>6}  {'cos':>7}  token")
for rank, (value, i) in enumerate(zip(top_values.tolist(), top_ids.tolist())):
    print(f"{rank:>4}  {i:>6}  {value:>7.4f}  {tokenizer.decode([i])!r}")

def bytes_to_unicode(): """GPT-2's byte <-> printable-character map. transformers 5 no longer exports this.""" bs = (list(range(ord("!"), ord("~") + 1)) + list(range(ord("\xa1"), ord("\xac") + 1)) + list(range(ord("\xae"), ord("\xff") + 1))) cs, n = bs[:], 0 for byte in range(256): if byte not in bs: bs.append(byte) cs.append(256 + n) n += 1 return dict(zip(bs, [chr(c) for c in cs])) byte_to_char = bytes_to_unicode() # bytes 0xF5-0xFF are illegal in UTF-8: these 11 rows are provably never a training target REFERENCE = [tokenizer.convert_tokens_to_ids(byte_to_char[b]) for b in range(0xF5, 0x100)] print("reference token ids:", REFERENCE) W_U_folded = W_U * ln_g # fold the ln_f gain into the unembedding u_ref = W_U_folded[REFERENCE].mean(0) # the "unused" direction indicator = F.cosine_similarity(W_U_folded, u_ref[None], dim=-1) top_values, top_ids = indicator.topk(30) print(f"\n{'rank':>4} {'id':>6} {'cos':>7} token") for rank, (value, i) in enumerate(zip(top_values.tolist(), top_ids.tolist())): print(f"{rank:>4} {i:>6} {value:>7.4f} {tokenizer.decode([i])!r}")

reference token ids: [177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187]

rank      id      cos  token
   0     183   0.9999  '�'
   1     182   0.9999  '�'
   2     178   0.9999  '�'
   3     179   0.9999  '�'
   4     187   0.9999  '�'
   5     177   0.9999  '�'
   6     185   0.9999  '�'
   7     184   0.9998  '�'
   8     181   0.9998  '�'
   9     202   0.9998  '\x0e'
  10     197   0.9998  '\t'
  11     213   0.9998  '\x19'
  12     207   0.9998  '\x13'
  13     204   0.9998  '\x10'
  14     125   0.9998  '�'
  15     205   0.9998  '\x11'
  16   39752   0.9998  'quickShip'
  17     180   0.9998  '�'
  18     208   0.9998  '\x14'
  19     200   0.9998  '\x0c'
  20   40240   0.9998  'oreAndOnline'
  21   30905   0.9998  'rawdownload'
  22     188   0.9998  '\x00'
  23     206   0.9998  '\x12'
  24     214   0.9998  '\x1a'
  25     190   0.9998  '\x02'
  26     195   0.9998  '\x07'
  27   45544   0.9998  ' サーティ'
  28   30898   0.9998  'embedreportprint'
  29     124   0.9998  '�'

Two populations sit at the top, under-trained for different reasons. ids 124–221 are the byte-fallback and control-character block — \x00, \x0e, and, surprisingly, \t: GPT-2's WebText was HTML-extracted, which normalized tabs and carriage returns away, so these are untrained despite being common in ordinary text. ids ~23,000–46,000 are junk merges from the Reddit scrape the tokenizer was fitted on — quickShip, oreAndOnline, rawdownload, embedreportprint. That second group is the SolidGoldMagikarp list, recovered from three lines of arithmetic on lm_head.weight.

The metric¶

We score a detector by recall at $k$: of the tokens a paper confirmed as under-trained in GPT-2, how many appear in our top $k$? Two published ground-truth sets, and a random permutation as the floor.

In [6]:

# Land & Bartolo (EMNLP 2024), verified under-trained in openai-community/gpt2: 33 tokens
LB_GPT2 = ['InstoreAndOnline','rawdownload','quickShip','oreAndOnline','embedreportprint',
    ' サーティ',' RandomRedditor',' externalToEVA',' TheNitrome','reportprint',' externalTo',
    'ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ','StreamerBot','ActionCode',
    'ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ','Nitrome','ÃÂÃÂÃÂÃÂ','DeliveryDate','ThumbnailImage','isSpecial',
    'oreAnd','cloneembedreportprint',' subur',' practition','Orderable',' pione','aditional',
    ' unintention',' councill','ortunately',' entreprene',' unbeliev','everal']

# Rumbelow & Watkins (2023), the SolidGoldMagikarp candidate list: 141 tokens
SGM = ['\x00','\x01','\x02','\x03','\x04','\x05','\x06','\x07','\x08','\x0e','\x0f','\x10','\x11',
    '\x12','\x13','\x14','\x15','\x16','\x17','\x18','\x19','\x1a','\x1b','\x7f','.[','ÃÂÃÂ',
    'ÃÂÃÂÃÂÃÂ','wcsstore','\\.',' practition',' Dragonbound',' guiActive',' \u200b',
    '\\\\\\\\\\\\\\\\','ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ',' davidjl','覚醒','"]=>',' --------',
    ' \u200e','ュ','ForgeModLoader','天',' 裏覚醒','PsyNetMessage',' guiActiveUn',' guiName',
    ' externalTo',' unfocusedRange',' guiActiveUnfocused',' guiIcon',' externalToEVA',
    ' externalToEVAOnly','reportprint','embedreportprint','cloneembedreportprint','rawdownload',
    'rawdownloadcloneembedreportprint','SpaceEngineers','externalActionCode','к',
    '?????-?????-','ーン','cffff','MpServer',' gmaxwell','cffffcc',' "$:/',' Smartstocks',
    '":[{"','龍喚士','":"","',' attRot',"''.",' Mechdragon',' PsyNet',' RandomRedditor',
    ' RandomRedditorWithNo','ertodd',' sqor',' istg',' "\\',' petertodd','StreamerBot',
    'TPPStreamerBot','FactoryReloaded',' partName','ヤ','\\">',' Skydragon','iHUD','catentry',
    'ItemThumbnailImage',' UCHIJ',' SetFontSize','DeliveryDate','quickShip','quickShipAvailable',
    'isSpecialOrderable','inventoryQuantity','channelAvailability','soType','soDeliveryDate',
    '龍契士','oreAndOnline','InstoreAndOnline','BuyableInstoreAndOnline','natureconservancy',
    'assetsadobe','\\-','Downloadha','Nitrome',' TheNitrome',' TheNitromeFan','GoldMagikarp',
    'DragonMagazine','TextColor',' srfN',' largeDownload',' srfAttach','EStreamFrame','ゼウス',
    ' SolidGoldMagikarp','ーティ',' サーティ',' サーティワン',' Adinida','":""},{"','ItemTracker',
    ' DevOnline','@#&','EngineDebug',' strutConnector',' Leilan','uyomi','aterasu',
    'ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ','ÃÂ','ÛÛ',' TAMADRA','EStream']

def single_token_ids(strings):
    ids = set()
    for s in strings:
        encoded = tokenizer(s, add_special_tokens=False)["input_ids"]
        if len(encoded) == 1:
            ids.add(encoded[0])
    return ids

GT_LB, GT_SGM = single_token_ids(LB_GPT2), single_token_ids(SGM)
print(f"ground truth: Land & Bartolo {len(GT_LB)}/{len(LB_GPT2)},  "
      f"SolidGoldMagikarp {len(GT_SGM)}/{len(SGM)}  (all map to single gpt2 ids)")

embed_norm = W_E.norm(dim=-1)
random_score = torch.tensor(np.random.default_rng(0).permutation(V).astype(np.float32))
KS = [25, 50, 100, 250, 500, 1000]

def recall_at_k(score, truth, ks=KS):
    order = score.argsort(descending=True)
    return [len(set(order[:k].tolist()) & truth) / len(truth) for k in ks]

DETECTORS = {"L&B indicator": indicator,
             "-embedding norm": -embed_norm,
             "random permutation": random_score}
curves = {}
print(f"\n{'detector':>20} {'truth':>5} " + " ".join(f"@{k:<6}" for k in KS))
for name, score in DETECTORS.items():
    for truth, label in [(GT_LB, "L&B"), (GT_SGM, "SGM")]:
        r = recall_at_k(score, truth)
        curves[(name, label)] = r
        print(f"{name:>20} {label:>5} " + " ".join(f"{v:<7.2f}" for v in r))

# Land & Bartolo (EMNLP 2024), verified under-trained in openai-community/gpt2: 33 tokens LB_GPT2 = ['InstoreAndOnline','rawdownload','quickShip','oreAndOnline','embedreportprint', ' サーティ',' RandomRedditor',' externalToEVA',' TheNitrome','reportprint',' externalTo', 'ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ','StreamerBot','ActionCode', 'ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ','Nitrome','ÃÂÃÂÃÂÃÂ','DeliveryDate','ThumbnailImage','isSpecial', 'oreAnd','cloneembedreportprint',' subur',' practition','Orderable',' pione','aditional', ' unintention',' councill','ortunately',' entreprene',' unbeliev','everal'] # Rumbelow & Watkins (2023), the SolidGoldMagikarp candidate list: 141 tokens SGM = ['\x00','\x01','\x02','\x03','\x04','\x05','\x06','\x07','\x08','\x0e','\x0f','\x10','\x11', '\x12','\x13','\x14','\x15','\x16','\x17','\x18','\x19','\x1a','\x1b','\x7f','.[','ÃÂÃÂ', 'ÃÂÃÂÃÂÃÂ','wcsstore','\\.',' practition',' Dragonbound',' guiActive',' \u200b', '\\\\\\\\\\\\\\\\','ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ',' davidjl','覚醒','"]=>',' --------', ' \u200e','ュ','ForgeModLoader','天',' 裏覚醒','PsyNetMessage',' guiActiveUn',' guiName', ' externalTo',' unfocusedRange',' guiActiveUnfocused',' guiIcon',' externalToEVA', ' externalToEVAOnly','reportprint','embedreportprint','cloneembedreportprint','rawdownload', 'rawdownloadcloneembedreportprint','SpaceEngineers','externalActionCode','к', '?????-?????-','ーン','cffff','MpServer',' gmaxwell','cffffcc',' "$:/',' Smartstocks', '":[{"','龍喚士','":"","',' attRot',"''.",' Mechdragon',' PsyNet',' RandomRedditor', ' RandomRedditorWithNo','ertodd',' sqor',' istg',' "\\',' petertodd','StreamerBot', 'TPPStreamerBot','FactoryReloaded',' partName','ヤ','\\">',' Skydragon','iHUD','catentry', 'ItemThumbnailImage',' UCHIJ',' SetFontSize','DeliveryDate','quickShip','quickShipAvailable', 'isSpecialOrderable','inventoryQuantity','channelAvailability','soType','soDeliveryDate', '龍契士','oreAndOnline','InstoreAndOnline','BuyableInstoreAndOnline','natureconservancy', 'assetsadobe','\\-','Downloadha','Nitrome',' TheNitrome',' TheNitromeFan','GoldMagikarp', 'DragonMagazine','TextColor',' srfN',' largeDownload',' srfAttach','EStreamFrame','ゼウス', ' SolidGoldMagikarp','ーティ',' サーティ',' サーティワン',' Adinida','":""},{"','ItemTracker', ' DevOnline','@#&','EngineDebug',' strutConnector',' Leilan','uyomi','aterasu', 'ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ','ÃÂ','ÛÛ',' TAMADRA','EStream'] def single_token_ids(strings): ids = set() for s in strings: encoded = tokenizer(s, add_special_tokens=False)["input_ids"] if len(encoded) == 1: ids.add(encoded[0]) return ids GT_LB, GT_SGM = single_token_ids(LB_GPT2), single_token_ids(SGM) print(f"ground truth: Land & Bartolo {len(GT_LB)}/{len(LB_GPT2)}, " f"SolidGoldMagikarp {len(GT_SGM)}/{len(SGM)} (all map to single gpt2 ids)") embed_norm = W_E.norm(dim=-1) random_score = torch.tensor(np.random.default_rng(0).permutation(V).astype(np.float32)) KS = [25, 50, 100, 250, 500, 1000] def recall_at_k(score, truth, ks=KS): order = score.argsort(descending=True) return [len(set(order[:k].tolist()) & truth) / len(truth) for k in ks] DETECTORS = {"L&B indicator": indicator, "-embedding norm": -embed_norm, "random permutation": random_score} curves = {} print(f"\n{'detector':>20} {'truth':>5} " + " ".join(f"@{k:<6}" for k in KS)) for name, score in DETECTORS.items(): for truth, label in [(GT_LB, "L&B"), (GT_SGM, "SGM")]: r = recall_at_k(score, truth) curves[(name, label)] = r print(f"{name:>20} {label:>5} " + " ".join(f"{v:<7.2f}" for v in r))

ground truth: Land & Bartolo 33/33,  SolidGoldMagikarp 141/141  (all map to single gpt2 ids)

            detector truth @25     @50     @100    @250    @500    @1000  
       L&B indicator   L&B 0.09    0.24    0.94    0.97    0.97    0.97   
       L&B indicator   SGM 0.09    0.21    0.32    0.33    0.33    0.33   
     -embedding norm   L&B 0.00    0.00    0.00    0.00    0.00    0.06   
     -embedding norm   SGM 0.00    0.00    0.00    0.00    0.00    0.06   
  random permutation   L&B 0.00    0.00    0.00    0.00    0.00    0.00   
  random permutation   SGM 0.00    0.00    0.00    0.01    0.01    0.02   

In [7]:

long = {"k": [], "recall": [], "detector": []}
for name in DETECTORS:
    for k, r in zip(KS, curves[(name, "L&B")]):
        long["k"].append(k); long["recall"].append(r); long["detector"].append(name)

fig = px.line(long, x="k", y="recall", color="detector", markers=True, log_x=True,
              title="Recall of Land & Bartolo's 33 verified GPT-2 tokens",
              labels=dict(k="candidates inspected (k)", recall="recall"), height=420)
fig.show()

long = {"k": [], "recall": [], "detector": []} for name in DETECTORS: for k, r in zip(KS, curves[(name, "L&B")]): long["k"].append(k); long["recall"].append(r); long["detector"].append(name) fig = px.line(long, x="k", y="recall", color="detector", markers=True, log_x=True, title="Recall of Land & Bartolo's 33 verified GPT-2 tokens", labels=dict(k="candidates inspected (k)", recall="recall"), height=420) fig.show()

Recall 0.94 at k = 100, 0.97 by k = 250: inspecting one fifth of one percent of the vocabulary recovers almost the whole published list, from a computation that never ran the model. The random floor is 0.00 throughout, as it should be for 33 targets in 50,257.

Recall against the SolidGoldMagikarp list saturates at 0.33, which is not a failure — that list was compiled against GPT-3 and GPT-J, which share GPT-2's tokenizer but not its training data. More on that in Finding 5.

Finding 2: embedding norm is at chance (a negative result)¶

The heuristic you will most often see repeated is that under-trained tokens have small embedding norms — never updated, so still near their small random initialization. It is intuitive, it is one line, and on GPT-2 it is worthless. Here is its top of list.

In [8]:

low_values, low_ids = embed_norm.topk(20, largest=False)
print("lowest input-embedding norm in GPT-2:")
for rank, (value, i) in enumerate(zip(low_values.tolist(), low_ids.tolist())):
    print(f"{rank:>4}  {i:>6}  {value:>7.4f}  {tokenizer.decode([i])!r}")

low_values, low_ids = embed_norm.topk(20, largest=False) print("lowest input-embedding norm in GPT-2:") for rank, (value, i) in enumerate(zip(low_values.tolist(), low_ids.tolist())): print(f"{rank:>4} {i:>6} {value:>7.4f} {tokenizer.decode([i])!r}")

lowest input-embedding norm in GPT-2:
   0     379   2.4537  ' at'
   1     287   2.4646  ' in'
   2     319   2.4726  ' on'
   3     281   2.4739  ' an'
   4     329   2.4863  ' for'
   5     355   2.4902  ' as'
   6     326   2.5074  ' that'
   7     838   2.5190  ' 10'
   8     284   2.5231  ' to'
   9    1315   2.5302  ' 15'
  10    1105   2.5339  ' 12'
  11     416   2.5449  ' by'
  12     642   2.5452  ' 5'
  13     807   2.5484  ' 8'
  14     554   2.5562  ' In'
  15     351   2.5679  ' with'
  16     257   2.5727  ' a'
  17    1160   2.5730  ' 20'
  18     718   2.5753  ' 6'
  19    1478   2.5757  ' 14'

at, in, on, to, for, a — the most frequent words in English, plus small integers. Its recall above was 0.00 at k = 250, indistinguishable from a random permutation.

The heuristic has the sign backwards, and saying why needs one number the weights cannot give us: how often each token occurs. We count over NeelNanda/pile-10k — tokenization only, no forward pass, about ten seconds. The same counts build the control set in Finding 4.

In [9]:

from datasets import load_dataset

pile = load_dataset("NeelNanda/pile-10k", split="train")
freq = np.zeros(V, dtype=np.int64)
for start in range(0, len(pile), 200):
    for row in tokenizer(pile[start:start + 200]["text"], add_special_tokens=False)["input_ids"]:
        np.add.at(freq, np.array(row), 1)
clear_output()

from scipy.stats import spearmanr
seen = freq > 0
print(f"{freq.sum():,} tokens counted; {(~seen).sum()} of {V} never occur")
print(f"spearman(embedding norm, log frequency) = "
      f"{spearmanr(embed_norm.cpu().numpy()[seen], np.log(freq[seen])).statistic:+.3f}")
print(f"spearman(L&B indicator,  log frequency) = "
      f"{spearmanr(indicator.cpu().numpy()[seen], np.log(freq[seen])).statistic:+.3f}")

from datasets import load_dataset pile = load_dataset("NeelNanda/pile-10k", split="train") freq = np.zeros(V, dtype=np.int64) for start in range(0, len(pile), 200): for row in tokenizer(pile[start:start + 200]["text"], add_special_tokens=False)["input_ids"]: np.add.at(freq, np.array(row), 1) clear_output() from scipy.stats import spearmanr seen = freq > 0 print(f"{freq.sum():,} tokens counted; {(~seen).sum()} of {V} never occur") print(f"spearman(embedding norm, log frequency) = " f"{spearmanr(embed_norm.cpu().numpy()[seen], np.log(freq[seen])).statistic:+.3f}") print(f"spearman(L&B indicator, log frequency) = " f"{spearmanr(indicator.cpu().numpy()[seen], np.log(freq[seen])).statistic:+.3f}")

17,351,935 tokens counted; 908 of 50257 never occur
spearman(embedding norm, log frequency) = -0.506
spearman(L&B indicator,  log frequency) = +0.218

Spearman −0.51: in GPT-2, embedding norm is a frequency meter. The mechanism is the tying we checked in Setup. Because W_E and W_U are one matrix, a never-occurring row still takes the "you are not the answer" gradient every step, which moves it without shrinking it, while frequent rows are pushed around constantly and under weight decay end up small.

📗 The heuristic is not wrong everywhere: Land & Bartolo use exactly this input-embedding L2 norm for GPT-J, and it works there. GPT-J has the same tokenizer and the same junk tokens but untied embeddings, so an input row really is touched only when its token occurs, and really does decay toward initialization. Same tokenizer, opposite verdict — "low embedding norm ⇒ under-trained" is a claim about weight tying, not about under-training, and it has to be rechecked per architecture.

Finding 3: published statistics are computed on folded, centered weights¶

A methodological trap rather than a fact about GPT-2, but it costs an afternoon and does not announce itself.

There is a well-known one-line test for "hard-to-speak" tokens: a token is anomalous if it is not the argmax of its own unembedding direction. In TransformerLens it reads

best = (model.W_U.T @ model.W_U).argmax(-1)
hard = (best != torch.arange(50257)).nonzero()

and returns 63 tokens for GPT-2. nnsight hands back the checkpoint's weights unmodified, which is right — but TransformerLens's W_U is not lm_head.weight. HookedTransformer.from_pretrained defaults to fold_ln=True (fold ln_f's gain into W_U, its bias into a per-token bias) and center_unembed=True (subtract the vocabulary mean row). Run the one-liner at all four stages.

In [10]:

b_U = W_U @ ln_b                                     # ln_f's bias, folded into a per-token bias
W_U_centered = W_U_folded - W_U_folded.mean(0, keepdim=True)
b_U_centered = b_U - b_U.mean()

@torch.no_grad()
def not_own_argmax(U, bias, chunk=512):
    """Tokens that are not the argmax of their own unembedding direction."""
    U, bias = U.cuda(), bias.cuda()
    flagged, ar = [], torch.arange(V, device="cuda")
    for start in range(0, V, chunk):
        best = (U[start:start + chunk] @ U.T + bias[None]).argmax(-1)
        flagged += (start + (best != ar[start:start + chunk]).nonzero().flatten()).tolist()
    del U, bias
    torch.cuda.empty_cache()
    return flagged

zero = torch.zeros(V)
STAGES = [("raw lm_head.weight",         W_U,          zero),
          ("+ ln_f gain",                W_U_folded,   zero),
          ("+ ln_f gain + bias",         W_U_folded,   b_U),
          ("+ gain + bias + centered",   W_U_centered, b_U_centered)]

for name, U, bias in STAGES:
    flagged = not_own_argmax(U, bias)
    outside_byte_block = [i for i in flagged if not 100 <= i <= 260]
    print(f"{name:<28} {len(flagged):>6} flagged  ({len(outside_byte_block):>5} outside the byte block)")

print("\nthe 17 non-byte tokens at the final stage:")
print([tokenizer.decode([i]) for i in outside_byte_block])

b_U = W_U @ ln_b # ln_f's bias, folded into a per-token bias W_U_centered = W_U_folded - W_U_folded.mean(0, keepdim=True) b_U_centered = b_U - b_U.mean() @torch.no_grad() def not_own_argmax(U, bias, chunk=512): """Tokens that are not the argmax of their own unembedding direction.""" U, bias = U.cuda(), bias.cuda() flagged, ar = [], torch.arange(V, device="cuda") for start in range(0, V, chunk): best = (U[start:start + chunk] @ U.T + bias[None]).argmax(-1) flagged += (start + (best != ar[start:start + chunk]).nonzero().flatten()).tolist() del U, bias torch.cuda.empty_cache() return flagged zero = torch.zeros(V) STAGES = [("raw lm_head.weight", W_U, zero), ("+ ln_f gain", W_U_folded, zero), ("+ ln_f gain + bias", W_U_folded, b_U), ("+ gain + bias + centered", W_U_centered, b_U_centered)] for name, U, bias in STAGES: flagged = not_own_argmax(U, bias) outside_byte_block = [i for i in flagged if not 100 <= i <= 260] print(f"{name:<28} {len(flagged):>6} flagged ({len(outside_byte_block):>5} outside the byte block)") print("\nthe 17 non-byte tokens at the final stage:") print([tokenizer.decode([i]) for i in outside_byte_block])

raw lm_head.weight              172 flagged  (  127 outside the byte block)

+ ln_f gain                   36603 flagged  (36512 outside the byte block)

+ ln_f gain + bias            30179 flagged  (30105 outside the byte block)

+ gain + bias + centered         63 flagged  (   17 outside the byte block)

the 17 non-byte tokens at the final stage:
[' an', 'ÃÂÃÂÃÂÃÂ', 'ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ', 'ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ', ' externalTo', ' externalToEVA', 'reportprint', 'embedreportprint', 'rawdownload', ' RandomRedditor', 'StreamerBot', 'quickShip', '龍�', 'oreAndOnline', 'InstoreAndOnline', ' TheNitrome', ' サーティ']

172 → 36,603 → 30,179 → 63. The published answer is the last, and it needs both steps: stopping after the fold flags 73% of the vocabulary, and skipping the fold gives a plausible-looking 172 that is a different set of tokens. Getting this wrong moves an answer by three orders of magnitude without raising an error. The 17 non-byte survivors are the SolidGoldMagikarp roll-call again, plus an — a genuine false positive, and a reminder that the argmax test is a threshold rather than an oracle.

⚠️ The transferable rule: nnsight gives you the checkpoint, and most published interpretability statistics were computed on processed weights. Before porting a formula, ask what W_U meant in the paper.

Finding 4: behavioural confirmation, against a frequency-matched control¶

A weight statistic is a hypothesis. Land & Bartolo confirm it by asking the model to repeat the token: a token is verified under-trained if its probability stays below 1% under every repetition prompt.

The control is the entire experiment. Every candidate is rare, so "candidates fail" on its own establishes only that rare tokens are rare. We build a control set matched on everything except the indicator: frequency-matched (zero occurrences in our corpus, like every candidate), id-matched (BPE merge order tracks frequency in the tokenizer's corpus, so take nearest ids, three per candidate), and untainted (excluded from the top 2,000 of every detector and from every published list). The byte block stays separate from the junk merges, and frequent tokens give an upper reference.

⚠️ Prompts are assembled from token ids, not strings: written into a string and re-tokenized, a token may merge with its neighbours and you score something else. Ids also make every row the same length, so there is no padding to correct for.

In [11]:

top_k_set = lambda score, k: set(score.argsort(descending=True)[:k].tolist())

CANDIDATES = sorted(top_k_set(indicator, 100))
BYTE_BLOCK = [i for i in CANDIDATES if 100 <= i <= 260]
JUNK_MERGE = [i for i in CANDIDATES if i > 260]

published = GT_LB | GT_SGM
tainted = top_k_set(indicator, 2000) | top_k_set(-embed_norm, 2000) | published | set(range(300))
pool = np.array(sorted(i for i in range(V) if freq[i] == 0 and i not in tainted))

CONTROLS, used = [], set()
for candidate in JUNK_MERGE:
    for j in np.argsort(np.abs(pool - candidate)):
        token = int(pool[j])
        if token not in used:
            used.add(token)
            CONTROLS.append(token)
            if len(CONTROLS) % 3 == 0:
                break
CONTROLS = sorted(CONTROLS)
FREQUENT = sorted(np.argsort(-freq)[200:260].tolist())

print(f"candidates: {len(JUNK_MERGE)} junk merges + {len(BYTE_BLOCK)} byte block")
print(f"controls:   {len(CONTROLS)} drawn from a pool of {len(pool)} untainted zero-frequency tokens")
print(f"median |id difference| to the nearest candidate: "
      f"{int(np.median([min(abs(c - g) for g in JUNK_MERGE) for c in CONTROLS]))}")
print(f"max corpus frequency  candidates {freq[JUNK_MERGE].max():>8}   controls {freq[CONTROLS].max():>8}")
print("\nexample controls:", [tokenizer.decode([i]) for i in CONTROLS[:12]])

top_k_set = lambda score, k: set(score.argsort(descending=True)[:k].tolist()) CANDIDATES = sorted(top_k_set(indicator, 100)) BYTE_BLOCK = [i for i in CANDIDATES if 100 <= i <= 260] JUNK_MERGE = [i for i in CANDIDATES if i > 260] published = GT_LB | GT_SGM tainted = top_k_set(indicator, 2000) | top_k_set(-embed_norm, 2000) | published | set(range(300)) pool = np.array(sorted(i for i in range(V) if freq[i] == 0 and i not in tainted)) CONTROLS, used = [], set() for candidate in JUNK_MERGE: for j in np.argsort(np.abs(pool - candidate)): token = int(pool[j]) if token not in used: used.add(token) CONTROLS.append(token) if len(CONTROLS) % 3 == 0: break CONTROLS = sorted(CONTROLS) FREQUENT = sorted(np.argsort(-freq)[200:260].tolist()) print(f"candidates: {len(JUNK_MERGE)} junk merges + {len(BYTE_BLOCK)} byte block") print(f"controls: {len(CONTROLS)} drawn from a pool of {len(pool)} untainted zero-frequency tokens") print(f"median |id difference| to the nearest candidate: " f"{int(np.median([min(abs(c - g) for g in JUNK_MERGE) for c in CONTROLS]))}") print(f"max corpus frequency candidates {freq[JUNK_MERGE].max():>8} controls {freq[CONTROLS].max():>8}") print("\nexample controls:", [tokenizer.decode([i]) for i in CONTROLS[:12]])

candidates: 48 junk merges + 52 byte block
controls:   144 drawn from a pool of 713 untainted zero-frequency tokens
median |id difference| to the nearest candidate: 94
max corpus frequency  candidates   431071   controls        0

example controls: [' charact', 'ustom', ' targ', ' weap', 'vironment', ' agre', ' promot', ' confir', '¯', '¯¯', ' Ukrain', ' negoti']

The controls are as rare as the candidates by construction — zero occurrences in 17 million tokens — and sit next to them in the merge table. (One candidate has a frequency in the hundreds of thousands: the indicator's top-100 does include a couple of very frequent function words. Those are false positives, and this step removes them.)

We score a token by the maximum probability it reaches at the answer slot across three repetition prompts, so a token fails only if no prompt gets it out.

In [12]:

def ids(text):
    return tokenizer(text, add_special_tokens=False)["input_ids"]

FEWSHOT_PREFIX = ids('A machine that repeats its input exactly.\n'
                     'Input: "apple" Output: "apple"\n'
                     'Input: " pizza" Output: " pizza"\n'
                     'Input: "Monday" Output: "Monday"\n'
                     'Input: "')
FEWSHOT_SUFFIX = ids('" Output: "')
TEMPLATES = [
    (FEWSHOT_PREFIX, FEWSHOT_SUFFIX),
    (ids('Question: Please repeat the string "'), ids('" back to me.\nAnswer: The string is "')),
    (None, None),                                   # 20 bare repetitions of the token
]
SLOT = len(FEWSHOT_PREFIX)                          # where the token sits in template 0

@torch.no_grad()
def repeat_probability(token_ids, batch=32):
    """max over templates of P(token) at the answer slot."""
    scores = np.zeros((len(token_ids), len(TEMPLATES)))
    for t, (prefix, suffix) in enumerate(TEMPLATES):
        for start in range(0, len(token_ids), batch):
            chunk = token_ids[start:start + batch]
            rows = [[i] * 20 for i in chunk] if prefix is None else [prefix + [i] + suffix for i in chunk]
            x = torch.tensor(rows, device=model.device)
            with model.trace(x):
                logits = model.lm_head.output[:, -1, :].save()
            probability = logits.float().softmax(-1)
            scores[start:start + batch, t] = probability[
                torch.arange(len(chunk)), torch.tensor(chunk, device=model.device)].cpu().numpy()
    return scores.max(1)

@torch.no_grad()
def greedy_repeat(token_ids, batch=32):
    """Greedily continue template 0; is the first generated token the target?"""
    correct, continuations = np.zeros(len(token_ids), bool), []
    for start in range(0, len(token_ids), batch):
        chunk = token_ids[start:start + batch]
        x = torch.tensor([FEWSHOT_PREFIX + [i] + FEWSHOT_SUFFIX for i in chunk], device=model.device)
        with model.generate(x, max_new_tokens=3, do_sample=False) as tracer:
            out = tracer.result.save()
        new = out[:, x.shape[1]:]
        for k, token in enumerate(chunk):
            correct[start + k] = int(new[k, 0]) == token
            continuations.append(tokenizer.decode(new[k]))
    return correct, continuations

GROUPS = {"candidates (junk merges)": JUNK_MERGE, "candidates (byte block)": BYTE_BLOCK,
          "controls (frequency-matched)": CONTROLS, "frequent tokens": FREQUENT}
results = {}
print(f"{'group':>30} {'n':>4} {'median P':>10} {'fail (<1%)':>13} {'greedy repeat':>14}")
for name, group in GROUPS.items():
    probability = repeat_probability(group)
    correct, continuations = greedy_repeat(group)
    results[name] = (group, probability, correct, continuations)
    print(f"{name:>30} {len(group):>4} {np.median(probability):>10.4f} "
          f"{(probability < 0.01).mean():>8.3f} ({int((probability < 0.01).sum()):>2}/{len(group)}) "
          f"{correct.mean():>13.3f}")

def ids(text): return tokenizer(text, add_special_tokens=False)["input_ids"] FEWSHOT_PREFIX = ids('A machine that repeats its input exactly.\n' 'Input: "apple" Output: "apple"\n' 'Input: " pizza" Output: " pizza"\n' 'Input: "Monday" Output: "Monday"\n' 'Input: "') FEWSHOT_SUFFIX = ids('" Output: "') TEMPLATES = [ (FEWSHOT_PREFIX, FEWSHOT_SUFFIX), (ids('Question: Please repeat the string "'), ids('" back to me.\nAnswer: The string is "')), (None, None), # 20 bare repetitions of the token ] SLOT = len(FEWSHOT_PREFIX) # where the token sits in template 0 @torch.no_grad() def repeat_probability(token_ids, batch=32): """max over templates of P(token) at the answer slot.""" scores = np.zeros((len(token_ids), len(TEMPLATES))) for t, (prefix, suffix) in enumerate(TEMPLATES): for start in range(0, len(token_ids), batch): chunk = token_ids[start:start + batch] rows = [[i] * 20 for i in chunk] if prefix is None else [prefix + [i] + suffix for i in chunk] x = torch.tensor(rows, device=model.device) with model.trace(x): logits = model.lm_head.output[:, -1, :].save() probability = logits.float().softmax(-1) scores[start:start + batch, t] = probability[ torch.arange(len(chunk)), torch.tensor(chunk, device=model.device)].cpu().numpy() return scores.max(1) @torch.no_grad() def greedy_repeat(token_ids, batch=32): """Greedily continue template 0; is the first generated token the target?""" correct, continuations = np.zeros(len(token_ids), bool), [] for start in range(0, len(token_ids), batch): chunk = token_ids[start:start + batch] x = torch.tensor([FEWSHOT_PREFIX + [i] + FEWSHOT_SUFFIX for i in chunk], device=model.device) with model.generate(x, max_new_tokens=3, do_sample=False) as tracer: out = tracer.result.save() new = out[:, x.shape[1]:] for k, token in enumerate(chunk): correct[start + k] = int(new[k, 0]) == token continuations.append(tokenizer.decode(new[k])) return correct, continuations GROUPS = {"candidates (junk merges)": JUNK_MERGE, "candidates (byte block)": BYTE_BLOCK, "controls (frequency-matched)": CONTROLS, "frequent tokens": FREQUENT} results = {} print(f"{'group':>30} {'n':>4} {'median P':>10} {'fail (<1%)':>13} {'greedy repeat':>14}") for name, group in GROUPS.items(): probability = repeat_probability(group) correct, continuations = greedy_repeat(group) results[name] = (group, probability, correct, continuations) print(f"{name:>30} {len(group):>4} {np.median(probability):>10.4f} " f"{(probability < 0.01).mean():>8.3f} ({int((probability < 0.01).sum()):>2}/{len(group)}) " f"{correct.mean():>13.3f}")

                         group    n   median P    fail (<1%)  greedy repeat

      candidates (junk merges)   48     0.0059    0.521 (25/48)         0.292
       candidates (byte block)   52     0.0000    0.885 (46/52)         0.058

  controls (frequency-matched)  144     0.8988    0.000 ( 0/144)         0.889
               frequent tokens   60     0.9477    0.000 ( 0/60)         0.600

In [13]:

long = {"probability": [], "group": []}
for name, (_, probability, _, _) in results.items():
    long["probability"] += np.maximum(probability, 1e-8).tolist()
    long["group"] += [name] * len(probability)

fig = px.strip(long, x="probability", y="group", color="group", log_x=True,
               title="Probability of repeating the token (max over three prompts)",
               labels=dict(probability="max P(token) at the answer slot", group=""), height=420)
fig.add_vline(x=0.01, line_dash="dash", annotation_text="Land & Bartolo threshold")
fig.update_layout(showlegend=False)
fig.show()

from scipy.stats import mannwhitneyu
from collections import Counter
candidate_p = results["candidates (junk merges)"][1]
control_p = results["controls (frequency-matched)"][1]
print("candidates vs controls, Mann-Whitney one-sided p =",
      f"{mannwhitneyu(candidate_p, control_p, alternative='less').pvalue:.2e}")
hard = [c for c, p in zip(results["candidates (junk merges)"][3], candidate_p) if p < 0.01]
print("\nwhat the model says instead, for the confirmed candidates:", Counter(hard).most_common(4))

long = {"probability": [], "group": []} for name, (_, probability, _, _) in results.items(): long["probability"] += np.maximum(probability, 1e-8).tolist() long["group"] += [name] * len(probability) fig = px.strip(long, x="probability", y="group", color="group", log_x=True, title="Probability of repeating the token (max over three prompts)", labels=dict(probability="max P(token) at the answer slot", group=""), height=420) fig.add_vline(x=0.01, line_dash="dash", annotation_text="Land & Bartolo threshold") fig.update_layout(showlegend=False) fig.show() from scipy.stats import mannwhitneyu from collections import Counter candidate_p = results["candidates (junk merges)"][1] control_p = results["controls (frequency-matched)"][1] print("candidates vs controls, Mann-Whitney one-sided p =", f"{mannwhitneyu(candidate_p, control_p, alternative='less').pvalue:.2e}") hard = [c for c, p in zip(results["candidates (junk merges)"][3], candidate_p) if p < 0.01] print("\nwhat the model says instead, for the confirmed candidates:", Counter(hard).most_common(4))

candidates vs controls, Mann-Whitney one-sided p = 1.22e-16

what the model says instead, for the confirmed candidates: [('lunch"', 13), ('Unfortunately"\n', 1), ('¤"', 1), ('kurisu', 1)]

52% of the junk-merge candidates and 88% of the byte-block candidates fail; 0 of 144 controls do. The controls are exactly as rare as the candidates in the evaluation corpus and repeat correctly 89% of the time, so this is not a rarity effect (Mann-Whitney p ≈ 1e-16). Without that column the result would be unfalsifiable.

Two details worth pausing on. The frequency-matched controls beat the frequent tokens on greedy accuracy, 0.89 to 0.60, because frequent tokens are punctuation and function words for which "repeat this exactly" is genuinely ambiguous. And the failures are not independent: thirteen of the confirmed candidates continue with the same string, lunch". That is one shared "there is nothing here" attractor, not twenty-five separate failures — a hint we pick up in Finding 6.

Finding 5: overlap with the published lists¶

Confirmed means in our top 100 by indicator and below the 1% repeat threshold. How does that set compare to the published ones?

In [14]:

SCHERLIS_NONBYTE = [9364, 14827, 23090, 30208, 30212, 30897, 30898, 30905,
                    39752, 39820, 40240, 40241, 42089, 45544]     # Scherlis (2023), 14 tokens

confirmed_junk = [i for i, p in zip(JUNK_MERGE, candidate_p) if p < 0.01]
confirmed_byte = [i for i, p in zip(BYTE_BLOCK, results["candidates (byte block)"][1]) if p < 0.01]
confirmed = set(confirmed_junk) | set(confirmed_byte)
print(f"confirmed: {len(confirmed_junk)} junk merges + {len(confirmed_byte)} byte block "
      f"= {len(confirmed)} tokens = {100 * len(confirmed) / V:.3f}% of the vocabulary\n")

print(f"of our {len(confirmed_junk)} non-byte confirmations, "
      f"{len(set(confirmed_junk) & GT_LB)} are in Land & Bartolo's 33, "
      f"{len(set(confirmed_junk) & GT_SGM)} in the SolidGoldMagikarp 141, "
      f"{len(set(confirmed_junk) & set(SCHERLIS_NONBYTE))} of Scherlis's 14")
print(f"recall of Land & Bartolo's 33: {len(set(confirmed_junk) & GT_LB)}/{len(GT_LB)} "
      f"= {len(set(confirmed_junk) & GT_LB) / len(GT_LB):.2f}")

everything_published = GT_LB | GT_SGM | set(SCHERLIS_NONBYTE)
print("\nnot in any published list:",
      [(i, tokenizer.decode([i])) for i in sorted(set(confirmed_junk) - everything_published)])
print("\nin L&B's list but outside our top 100:",
      [(i, tokenizer.decode([i])) for i in sorted(GT_LB - set(CANDIDATES))])
print("\nin L&B's list, detected, but PASSING our repeat test:")
for i in sorted(GT_LB & set(JUNK_MERGE)):
    p = candidate_p[JUNK_MERGE.index(i)]
    if p >= 0.01:
        print(f"    {tokenizer.decode([i])!r:<22} max P = {p:.3f}")

SCHERLIS_NONBYTE = [9364, 14827, 23090, 30208, 30212, 30897, 30898, 30905, 39752, 39820, 40240, 40241, 42089, 45544] # Scherlis (2023), 14 tokens confirmed_junk = [i for i, p in zip(JUNK_MERGE, candidate_p) if p < 0.01] confirmed_byte = [i for i, p in zip(BYTE_BLOCK, results["candidates (byte block)"][1]) if p < 0.01] confirmed = set(confirmed_junk) | set(confirmed_byte) print(f"confirmed: {len(confirmed_junk)} junk merges + {len(confirmed_byte)} byte block " f"= {len(confirmed)} tokens = {100 * len(confirmed) / V:.3f}% of the vocabulary\n") print(f"of our {len(confirmed_junk)} non-byte confirmations, " f"{len(set(confirmed_junk) & GT_LB)} are in Land & Bartolo's 33, " f"{len(set(confirmed_junk) & GT_SGM)} in the SolidGoldMagikarp 141, " f"{len(set(confirmed_junk) & set(SCHERLIS_NONBYTE))} of Scherlis's 14") print(f"recall of Land & Bartolo's 33: {len(set(confirmed_junk) & GT_LB)}/{len(GT_LB)} " f"= {len(set(confirmed_junk) & GT_LB) / len(GT_LB):.2f}") everything_published = GT_LB | GT_SGM | set(SCHERLIS_NONBYTE) print("\nnot in any published list:", [(i, tokenizer.decode([i])) for i in sorted(set(confirmed_junk) - everything_published)]) print("\nin L&B's list but outside our top 100:", [(i, tokenizer.decode([i])) for i in sorted(GT_LB - set(CANDIDATES))]) print("\nin L&B's list, detected, but PASSING our repeat test:") for i in sorted(GT_LB & set(JUNK_MERGE)): p = candidate_p[JUNK_MERGE.index(i)] if p >= 0.01: print(f" {tokenizer.decode([i])!r:<22} max P = {p:.3f}")

confirmed: 25 junk merges + 46 byte block = 71 tokens = 0.141% of the vocabulary

of our 25 non-byte confirmations, 23 are in Land & Bartolo's 33, 18 in the SolidGoldMagikarp 141, 14 of Scherlis's 14
recall of Land & Bartolo's 33: 23/33 = 0.70

not in any published list: [(33434, '��士')]

in L&B's list but outside our top 100: [(8438, 'everal'), (20554, ' unbeliev')]

in L&B's list, detected, but PASSING our repeat test:
    ' entreprene'          max P = 0.131
    ' councill'            max P = 0.034
    'aditional'            max P = 0.020
    'ThumbnailImage'       max P = 0.015
    'Orderable'            max P = 0.108
    'isSpecial'            max P = 0.158
    'DeliveryDate'         max P = 0.568
    'oreAnd'               max P = 0.015

71 tokens, 0.14% of the vocabulary. 23 of our 25 non-byte confirmations are in Land & Bartolo's list and all 14 of Scherlis's are; recall of L&B's 33 is 0.70. The shortfall splits cleanly. Two we never tested — unbeliev, everal — fell outside our top 100; recall at k = 250 was 0.97, so a larger budget catches them. Eight we detected but that pass our repeat test at 1.5% to 57%: word-fragment and camel-case tokens like ThumbnailImage and DeliveryDate, which our three prompts happen to elicit and the paper's do not. That disagreement is prompt-set choice, not method — "confirmed under-trained" is a threshold on a prompt-dependent quantity, not a property of the token.

One confirmed token is in no published list: id 33434, '�士', the truncated-UTF-8 neighbour of SolidGoldMagikarp's 龍喚士, at a maximum repeat probability of 0.008. Going the other way, SolidGoldMagikarp itself, petertodd, Leilan and davidjl never surface for GPT-2 — they are GPT-J and GPT-3 phenomena, which is why recall against that list capped at 0.33.

Finding 6: what the model does when it reads one¶

So far this is a token list. What makes it interpretability is asking where in the forward pass the failure happens — and here we finally need traces.

We build matched prompt pairs: the 25 confirmed junk-merge tokens against their nearest-id controls, same template, differing in exactly one token id. All 13 residual-stream points are captured in one trace, and at the token's own position we read a logit lens (ln_f then W_U on the intermediate residual — does the stream decode back to the token sitting there?), the entropy of that readout, and the residual norm relative to the sequence median.

⚠️ Two nnsight details. A transformers-5 decoder block's .output is a bare tensor, not a tuple — no [0]. And trace-body locals do not survive the block, so append into a list created outside it.

In [15]:

W_U_cuda, ln_g_cuda, ln_b_cuda = W_U.cuda(), ln_g.cuda(), ln_b.cuda()

def logit_lens(h):
    return F.layer_norm(h.float(), (h.shape[-1],), ln_g_cuda, ln_b_cuda, 1e-5) @ W_U_cuda.T

paired_controls = []
for candidate in confirmed_junk:
    paired_controls.append(min((c for c in CONTROLS if c not in paired_controls),
                               key=lambda c: abs(c - candidate)))

def prompts_for(token_ids):
    return torch.tensor([FEWSHOT_PREFIX + [i] + FEWSHOT_SUFFIX for i in token_ids],
                        device=model.device)

@torch.no_grad()
def read_residual_stream(token_ids):
    x = prompts_for(token_ids)
    residuals = []                                  # created outside the block on purpose
    with model.trace(x):
        embeddings = model.transformer.drop.output.save()      # resid_pre of block 0
        for layer in range(model.config.n_layer):
            residuals.append(model.transformer.h[layer].output.save())
    assert torch.is_tensor(residuals[0]), "a block's .output is a bare tensor in transformers 5"
    H = torch.stack([embeddings] + residuals)       # [n_layer + 1, batch, seq, d_model]

    rows = torch.arange(H.shape[1], device=model.device)
    targets = torch.tensor(token_ids, device=model.device)
    at_slot = logit_lens(H[:, :, SLOT, :]).softmax(-1)
    return dict(
        norm_ratio=(H.norm(dim=-1)[:, :, SLOT] / H.norm(dim=-1).median(dim=-1).values).cpu().numpy(),
        p_self=at_slot[:, rows, targets].cpu().numpy(),
        entropy=-(at_slot * at_slot.clamp_min(1e-30).log()).sum(-1).cpu().numpy(),
        p_answer=logit_lens(H[:, :, -1, :]).softmax(-1)[:, rows, targets].cpu().numpy(),
        top1=at_slot.argmax(-1).cpu().numpy())

glitch = read_residual_stream(confirmed_junk)
control = read_residual_stream(paired_controls)

print(f"{'layer':>5} | {'|resid| / median':>17} | {'lens P(self) @ slot':>21} | "
      f"{'entropy @ slot':>16} | {'lens P(token) @ answer':>23}")
print(f"{'':>5} | {'glitch  control':>17} | {'glitch   control':>21} | "
      f"{'glitch control':>16} | {'glitch    control':>23}")
for layer in range(13):
    print(f"{layer:>5} |  {glitch['norm_ratio'][layer].mean():.3f}   {control['norm_ratio'][layer].mean():.3f}"
          f"   |  {glitch['p_self'][layer].mean():.4f}    {control['p_self'][layer].mean():.4f}"
          f"       |  {glitch['entropy'][layer].mean():5.2f}  {control['entropy'][layer].mean():5.2f}"
          f"    |  {glitch['p_answer'][layer].mean():.5f}   {control['p_answer'][layer].mean():.5f}")

print("\nMann-Whitney, one-sided, n = 25 pairs:")
print(f"  layer 0 P(self)          p = {mannwhitneyu(glitch['p_self'][0], control['p_self'][0], alternative='less').pvalue:.1e}")
print(f"  layer 1 entropy          p = {mannwhitneyu(glitch['entropy'][1], control['entropy'][1], alternative='greater').pvalue:.1e}")
print(f"  layer 11 P(token)@answer p = {mannwhitneyu(glitch['p_answer'][11], control['p_answer'][11], alternative='less').pvalue:.1e}")

W_U_cuda, ln_g_cuda, ln_b_cuda = W_U.cuda(), ln_g.cuda(), ln_b.cuda() def logit_lens(h): return F.layer_norm(h.float(), (h.shape[-1],), ln_g_cuda, ln_b_cuda, 1e-5) @ W_U_cuda.T paired_controls = [] for candidate in confirmed_junk: paired_controls.append(min((c for c in CONTROLS if c not in paired_controls), key=lambda c: abs(c - candidate))) def prompts_for(token_ids): return torch.tensor([FEWSHOT_PREFIX + [i] + FEWSHOT_SUFFIX for i in token_ids], device=model.device) @torch.no_grad() def read_residual_stream(token_ids): x = prompts_for(token_ids) residuals = [] # created outside the block on purpose with model.trace(x): embeddings = model.transformer.drop.output.save() # resid_pre of block 0 for layer in range(model.config.n_layer): residuals.append(model.transformer.h[layer].output.save()) assert torch.is_tensor(residuals[0]), "a block's .output is a bare tensor in transformers 5" H = torch.stack([embeddings] + residuals) # [n_layer + 1, batch, seq, d_model] rows = torch.arange(H.shape[1], device=model.device) targets = torch.tensor(token_ids, device=model.device) at_slot = logit_lens(H[:, :, SLOT, :]).softmax(-1) return dict( norm_ratio=(H.norm(dim=-1)[:, :, SLOT] / H.norm(dim=-1).median(dim=-1).values).cpu().numpy(), p_self=at_slot[:, rows, targets].cpu().numpy(), entropy=-(at_slot * at_slot.clamp_min(1e-30).log()).sum(-1).cpu().numpy(), p_answer=logit_lens(H[:, :, -1, :]).softmax(-1)[:, rows, targets].cpu().numpy(), top1=at_slot.argmax(-1).cpu().numpy()) glitch = read_residual_stream(confirmed_junk) control = read_residual_stream(paired_controls) print(f"{'layer':>5} | {'|resid| / median':>17} | {'lens P(self) @ slot':>21} | " f"{'entropy @ slot':>16} | {'lens P(token) @ answer':>23}") print(f"{'':>5} | {'glitch control':>17} | {'glitch control':>21} | " f"{'glitch control':>16} | {'glitch control':>23}") for layer in range(13): print(f"{layer:>5} | {glitch['norm_ratio'][layer].mean():.3f} {control['norm_ratio'][layer].mean():.3f}" f" | {glitch['p_self'][layer].mean():.4f} {control['p_self'][layer].mean():.4f}" f" | {glitch['entropy'][layer].mean():5.2f} {control['entropy'][layer].mean():5.2f}" f" | {glitch['p_answer'][layer].mean():.5f} {control['p_answer'][layer].mean():.5f}") print("\nMann-Whitney, one-sided, n = 25 pairs:") print(f" layer 0 P(self) p = {mannwhitneyu(glitch['p_self'][0], control['p_self'][0], alternative='less').pvalue:.1e}") print(f" layer 1 entropy p = {mannwhitneyu(glitch['entropy'][1], control['entropy'][1], alternative='greater').pvalue:.1e}") print(f" layer 11 P(token)@answer p = {mannwhitneyu(glitch['p_answer'][11], control['p_answer'][11], alternative='less').pvalue:.1e}")

layer |  |resid| / median |   lens P(self) @ slot |   entropy @ slot |  lens P(token) @ answer
      |   glitch  control |      glitch   control |   glitch control |       glitch    control
    0 |  0.986   1.186   |  0.2484    1.0000       |   0.41   0.00    |  0.00000   0.00000
    1 |  1.184   1.220   |  0.0019    0.8386       |   5.20   0.72    |  0.00000   0.00000
    2 |  1.153   1.236   |  0.0001    0.6149       |   4.90   1.81    |  0.00000   0.00000
    3 |  1.108   1.239   |  0.0001    0.4718       |   4.97   2.43    |  0.00000   0.00000
    4 |  1.153   1.263   |  0.0000    0.3447       |   4.42   2.82    |  0.00000   0.00000
    5 |  1.171   1.257   |  0.0000    0.1713       |   4.18   3.34    |  0.00000   0.00000
    6 |  1.124   1.200   |  0.0000    0.0858       |   3.87   3.58    |  0.00000   0.00000
    7 |  1.108   1.190   |  0.0000    0.0366       |   3.67   3.35    |  0.00000   0.00001
    8 |  1.089   1.181   |  0.0000    0.0124       |   3.40   2.93    |  0.00000   0.00011
    9 |  0.991   1.119   |  0.0000    0.0039       |   3.05   2.58    |  0.00000   0.17102
   10 |  0.900   1.035   |  0.0000    0.0010       |   0.88   1.10    |  0.00009   0.82272
   11 |  0.828   0.925   |  0.0000    0.0005       |   0.81   0.97    |  0.00053   0.91757
   12 |  0.844   0.760   |  0.0000    0.0003       |   3.88   2.67    |  0.00045   0.60355

Mann-Whitney, one-sided, n = 25 pairs:
  layer 0 P(self)          p = 1.8e-10
  layer 1 entropy          p = 2.5e-08
  layer 11 P(token)@answer p = 7.1e-10

In [16]:

long = {"layer": [], "value": [], "group": [], "metric": []}
for name, data in [("glitch", glitch), ("control", control)]:
    for metric, key in [("logit-lens P(self) at the token's own slot", "p_self"),
                        ("readout entropy at the token's own slot (nats)", "entropy"),
                        ("logit-lens P(token) at the answer slot", "p_answer")]:
        for layer in range(13):
            long["layer"].append(layer); long["value"].append(float(data[key][layer].mean()))
            long["group"].append(name); long["metric"].append(metric)

fig = px.line(long, x="layer", y="value", color="group", facet_col="metric", markers=True,
              title="Reading the residual stream at a glitch token vs a matched control",
              labels=dict(layer="residual stream point (0 = embedding)", value=""), height=380)
fig.update_yaxes(matches=None, showticklabels=True)
fig.for_each_annotation(lambda a: a.update(text=a.text.split("=")[-1]))
fig.show()

long = {"layer": [], "value": [], "group": [], "metric": []} for name, data in [("glitch", glitch), ("control", control)]: for metric, key in [("logit-lens P(self) at the token's own slot", "p_self"), ("readout entropy at the token's own slot (nats)", "entropy"), ("logit-lens P(token) at the answer slot", "p_answer")]: for layer in range(13): long["layer"].append(layer); long["value"].append(float(data[key][layer].mean())) long["group"].append(name); long["metric"].append(metric) fig = px.line(long, x="layer", y="value", color="group", facet_col="metric", markers=True, title="Reading the residual stream at a glitch token vs a matched control", labels=dict(layer="residual stream point (0 = embedding)", value=""), height=380) fig.update_yaxes(matches=None, showticklabels=True) fig.for_each_annotation(lambda a: a.update(text=a.text.split("=")[-1])) fig.show()

Three readings, in order of how much they should surprise you.

1. The divergence is complete at layer 0. Because GPT-2 ties its embeddings, the logit lens on the embedding at a token's own position asks "does this row decode to itself" — and the control answer is P = 1.0000 against 0.248 for a glitch token. There is nothing to localize in the layers: under-training is a property of the row, and every later difference is inherited rather than generated.
2. Early readout entropy explodes — 5.20 nats at layer 1 against 0.72 for a control, on a 50,257-way distribution. The model's early "what is sitting here" is near-uniform, and it reconverges by layer 10 onto the wrong thing.
3. There is no massive-activation signature. The residual at a glitch token is smaller than at a control, by about 10% (0.83 vs 0.93 of the sequence median at layer 11) — significant across 25 pairs, but a deficit, not a spike.

The qualitative version below says the same thing. A control still reads as itself two layers in (VIDIA → VIDIA → GeForce); a glitch token either reads as itself once, weakly, and then scatters ( subur → j → py), or never reads as itself at all — ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ decodes to destro.

In [17]:

print("logit-lens top-1 at the token's own position, layers 0 / 2 / 6 / 12")
for label, data, group in [("glitch ", glitch, confirmed_junk), ("control", control, paired_controls)]:
    print(f"\n{label}:")
    for k in range(6):
        reads = [repr(tokenizer.decode([int(data["top1"][layer, k])])) for layer in (0, 2, 6, 12)]
        print(f"   {tokenizer.decode([group[k]])!r:<24} -> " + "  ".join(f"{r:<12}" for r in reads))

print("logit-lens top-1 at the token's own position, layers 0 / 2 / 6 / 12") for label, data, group in [("glitch ", glitch, confirmed_junk), ("control", control, paired_controls)]: print(f"\n{label}:") for k in range(6): reads = [repr(tokenizer.decode([int(data["top1"][layer, k])])) for layer in (0, 2, 6, 12)] print(f" {tokenizer.decode([group[k]])!r:<24} -> " + " ".join(f"{r:<12}" for r in reads))

logit-lens top-1 at the token's own position, layers 0 / 2 / 6 / 12

glitch :
   'ortunately'             -> 'ortunately'  'entimes'     ','           '"'         
   'ÃÂÃÂÃÂÃÂ'               -> ' corrid'     't'           't'           '"'         
   ' subur'                 -> ' subur'      'j'           'py'          'itsu'      
   'ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ'       -> ' destro'     's'           's'           '"'         
   ' pione'                 -> ' pione'      'fort'        'ert'         'ert'       
   ' practition'            -> ' practition'  'eers'        'eering'      '"'         

control:
   ' confir'                -> ' confir'     ' confir'     'ance'        'me'        
   'perty'                  -> 'perty'       'ty'          'ty'          '"'         
   'VIDIA'                  -> 'VIDIA'       'VIDIA'       ' GeForce'    '"'         
   ' awa'                   -> ' awa'        ' awa'        'ose'         '"'         
   'aukee'                  -> 'aukee'       'aukee'       'ite'         '"'         
   'ocumented'              -> 'ocumented'   'ocumented'   ' change'     '"'         

Where does token identity leave the slot?¶

Layer 0 says the failure is inherited; it does not say when it becomes irreversible. So: run the glitch prompts, splice the control run's residual into the token's slot at one layer, and measure how much of the control's answer comes back.

We batch the sweep with tracer.invoke — each invoke adds its prompts to one batched forward pass, so thirteen patch sites cost far fewer passes than thirteen traces. ⚠️ We group them four at a time rather than all thirteen, because the batched logits are batch × seq × 50,257 floats; a full-vocabulary readout is the one place where batching harder costs more memory than it saves.

First, the control every patching experiment owes the reader: patching a run onto itself must be exactly a no-op.

In [18]:

x_glitch, x_control = prompts_for(confirmed_junk), prompts_for(paired_controls)
rows = torch.arange(len(confirmed_junk), device=model.device)
control_tokens = torch.tensor(paired_controls, device=model.device)

@torch.no_grad()
def patch_onto_itself():
    with model.trace(x_glitch):
        donor = model.transformer.h[5].output.save()
        clean = model.lm_head.output.save()
    with model.trace(x_glitch):
        model.transformer.h[5].output[:, SLOT, :] = donor[:, SLOT, :]
        patched = model.lm_head.output.save()
    return torch.equal(patched, clean), (patched - clean).abs().max().item()

identical, max_difference = patch_onto_itself()
print(f"patch onto itself: torch.equal = {identical}, max |delta| = {max_difference}")

x_glitch, x_control = prompts_for(confirmed_junk), prompts_for(paired_controls) rows = torch.arange(len(confirmed_junk), device=model.device) control_tokens = torch.tensor(paired_controls, device=model.device) @torch.no_grad() def patch_onto_itself(): with model.trace(x_glitch): donor = model.transformer.h[5].output.save() clean = model.lm_head.output.save() with model.trace(x_glitch): model.transformer.h[5].output[:, SLOT, :] = donor[:, SLOT, :] patched = model.lm_head.output.save() return torch.equal(patched, clean), (patched - clean).abs().max().item() identical, max_difference = patch_onto_itself() print(f"patch onto itself: torch.equal = {identical}, max |delta| = {max_difference}")

patch onto itself: torch.equal = True, max |delta| = 0.0

In [19]:

@torch.no_grad()
def layer_sweep():
    donors = []
    with model.trace(x_control):
        donors.append(model.transformer.drop.output.save())
        for layer in range(model.config.n_layer):
            donors.append(model.transformer.h[layer].output.save())
        control_logits = model.lm_head.output[:, -1, :].save()
    baseline = control_logits.float().softmax(-1)[rows, control_tokens].mean().item()

    patched_logits = []
    for group_start in range(0, 13, 4):
        with model.trace() as tracer:
            saved = nnsight.save([])
            for site in range(group_start, min(group_start + 4, 13)):
                with tracer.invoke(x_glitch):
                    if site == 0:
                        model.transformer.drop.output[:, SLOT, :] = donors[0][:, SLOT, :]
                    else:
                        model.transformer.h[site - 1].output[:, SLOT, :] = donors[site][:, SLOT, :]
                    saved.append(model.lm_head.output[:, -1, :])
        patched_logits += list(saved)
    restored = [float(l.float().softmax(-1)[rows, control_tokens].mean()) / baseline
                for l in patched_logits]
    return baseline, restored

baseline, restored = layer_sweep()
print(f"unpatched control run: P(control token) = {baseline:.4f}\n")
names = ["resid_pre"] + [f"block {i}" for i in range(12)]
for name, value in zip(names, restored):
    print(f"  patch at {name:<10} {100 * value:>6.1f}% restored")

fig = px.line(x=names, y=[100 * v for v in restored], markers=True,
              title="Splicing the control run's residual into the glitch run, at one site",
              labels=dict(x="patch site", y="% of control behaviour restored"), height=400)
fig.show()

@torch.no_grad() def layer_sweep(): donors = [] with model.trace(x_control): donors.append(model.transformer.drop.output.save()) for layer in range(model.config.n_layer): donors.append(model.transformer.h[layer].output.save()) control_logits = model.lm_head.output[:, -1, :].save() baseline = control_logits.float().softmax(-1)[rows, control_tokens].mean().item() patched_logits = [] for group_start in range(0, 13, 4): with model.trace() as tracer: saved = nnsight.save([]) for site in range(group_start, min(group_start + 4, 13)): with tracer.invoke(x_glitch): if site == 0: model.transformer.drop.output[:, SLOT, :] = donors[0][:, SLOT, :] else: model.transformer.h[site - 1].output[:, SLOT, :] = donors[site][:, SLOT, :] saved.append(model.lm_head.output[:, -1, :]) patched_logits += list(saved) restored = [float(l.float().softmax(-1)[rows, control_tokens].mean()) / baseline for l in patched_logits] return baseline, restored baseline, restored = layer_sweep() print(f"unpatched control run: P(control token) = {baseline:.4f}\n") names = ["resid_pre"] + [f"block {i}" for i in range(12)] for name, value in zip(names, restored): print(f" patch at {name:<10} {100 * value:>6.1f}% restored") fig = px.line(x=names, y=[100 * v for v in restored], markers=True, title="Splicing the control run's residual into the glitch run, at one site", labels=dict(x="patch site", y="% of control behaviour restored"), height=400) fig.show()

unpatched control run: P(control token) = 0.6035

  patch at resid_pre   100.0% restored
  patch at block 0      98.8% restored
  patch at block 1      99.6% restored
  patch at block 2     103.7% restored
  patch at block 3     104.2% restored
  patch at block 4     104.0% restored
  patch at block 5     101.9% restored
  patch at block 6      96.7% restored
  patch at block 7     104.8% restored
  patch at block 8      87.1% restored
  patch at block 9      26.0% restored
  patch at block 10      0.5% restored
  patch at block 11      0.0% restored

Patch-onto-itself is torch.equal, not merely allclose — the intervention machinery is exact, so what follows is the model's behaviour and not the harness's.

The sweep is flat at ~100% through block 7, drops to 87% at block 8, 26% at block 9, 0.5% at block 10, and nothing after. "Which token is sitting in this slot" is read out of the slot between blocks 8 and 10 — exactly where GPT-2-small's copy and induction heads live; once the wrong content has reached the answer position, repairing the source cannot rescue it. The failure originates at layer 0 and is fully determined by block 8. Nothing about a glitch token is decided late.

Caveats¶

• One model, one tokenizer. Every number here is GPT-2-small. The indicator transfers (Land & Bartolo run it over dozens of models); which criterion works does not. We did not run GPT-J, so the other half of Finding 2's tied/untied comparison is the paper's claim, not our measurement.
• "Confirmed under-trained" is prompt-dependent. The threshold is a convention; the ranking is the reproducible part.
• The corpus is not the training corpus. Our counts come from the Pile, GPT-2 was trained on WebText. \t occurs 51,606 times in our corpus and is still untrained — a genuine finding, and a warning about what "frequency-matched" can mean.
• Precision at k = 100 is 52% on the junk merges, not 100%: the indicator's false positives include a few very frequent function words. The behavioural stage is not optional.
• The internals are 25 pairs and one prompt template. The layer-0 result is a statement about the weights, but the sweep's crossover point would move somewhat with the prompt.

Conclusion¶

🎣 Under-trained tokens are findable before you run the model: rank the vocabulary by how closely each unembedding row aligns with rows that provably cannot be training targets, and the top 100 of 50,257 holds 94% of the published list. Then, and only then, spend forward passes — on controls as well as candidates, because 0 of 144 frequency-matched controls failing is what turns "these tokens are weird" into a result.

This is also a look at a face of nnsight most interpretability work never touches: model.transformer.wte.weight is just the parameter, no trace required, and the only thing between you and it is dispatch=True. Two traps come with it — the checkpoint's weights are not the folded, centered ones published formulas assume, and a write to a Parameter inside a trace block is permanent, unlike a write to an activation, which is scoped to the run.

Related: Logit Lens for the readout used in Finding 6, and Activation Patching for the sweep at the end.

References¶

• Land, Bartolo, Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models, EMNLP 2024, pp. 11631–11646. arXiv:2405.05417; code and per-model reports at github.com/cohere-ai/magikarp
• Rumbelow, Watkins, SolidGoldMagikarp (plus, prompt generation), LessWrong, 2023
• Rumbelow, Watkins, SolidGoldMagikarp II: technical details and more recent findings, LessWrong, 2023
• Scherlis, A mechanistic explanation for SolidGoldMagikarp-like tokens in GPT2, LessWrong, 2023
• Corpus: NeelNanda/pile-10k, a 10,000-document sample of The Pile (Gao et al., 2020)