Info

Last Execution: 2026-08-19

Package	Version
nnsight	0.8
Python	3.12
torch	2.13.0+cu126
transformers	5.15.0
datasets	5.0.0

Under-trained Tokens: Fishing for Magikarp in GPT-2¶

Introduction¶

🎣 A tokenizer is fitted before the model is trained, usually on a different corpus. Anything that reached the merge table but not the training data leaves the model with a vocabulary entry it has essentially never seen — famously SolidGoldMagikarp, a Reddit username that survived into GPT-2's vocabulary and that GPT-3 could not repeat, spell, or acknowledge.

These are under-trained tokens, or glitch tokens, and the question we reproduce is Land & Bartolo's: can you find them from the weights alone, before running a single prompt? Their answer is a weight-based indicator computed from the unembedding matrix, followed by a behavioural check on the top candidates only. Six findings follow: the indicator; the popular embedding-norm heuristic being at chance; why formulas copied from interpretability papers do not reproduce against raw HuggingFace weights; behavioural confirmation against a frequency-matched control; overlap with the published lists; and what the residual stream does at a glitch token.

📗 Primary paper: Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models (Land & Bartolo, EMNLP 2024). The phenomenon was first described in SolidGoldMagikarp (plus, prompt generation) (Rumbelow & Watkins, 2023).

⚠️ This mini-paper is unusual for the site in that most of it reads weights, not activations — there is no trace at all until Finding 4. In nnsight a model's parameters are the wrapped module's own nn.Parameters, reached by walking the envoy exactly as you would walk the HuggingFace module: model.transformer.wte.weight is the embedding matrix, full stop.

Setup¶

If using Colab, install NNsight and datasets (we need a small corpus to build the control set):

!pip install -U nnsight datasets

In [1]:

try:
    import google.colab
    is_colab = True
except ImportError:
    is_colab = False

if is_colab:
    !pip install -U nnsight datasets

try: import google.colab is_colab = True except ImportError: is_colab = False if is_colab: !pip install -U nnsight datasets

In [2]:

import warnings
warnings.filterwarnings("ignore", message=".*IProgress.*")

from IPython.display import clear_output
import numpy as np
import torch
import torch.nn.functional as F
import nnsight
from nnsight import TransformersModel

import plotly.express as px
import plotly.io as pio
pio.renderers.default = "colab" if is_colab else "plotly_mimetype+notebook_connected+colab+notebook"

import warnings warnings.filterwarnings("ignore", message=".*IProgress.*") from IPython.display import clear_output import numpy as np import torch import torch.nn.functional as F import nnsight from nnsight import TransformersModel import plotly.express as px import plotly.io as pio pio.renderers.default = "colab" if is_colab else "plotly_mimetype+notebook_connected+colab+notebook"

⚠️ Pass dispatch=True. A TransformersModel is built lazily, so until it dispatches every module lives on the meta device and a parameter read off it has the right shape, the right dtype, and no storage. norm, topk and argsort all succeed on a meta tensor and return another one, so the failure surfaces well below the mistake. An activation workload never notices, because the first trace dispatches; a weight workload hits it immediately.

In [3]:

model = TransformersModel("openai-community/gpt2", device_map="auto", dispatch=True)
clear_output()

tokenizer = model.tokenizer
V = model.config.vocab_size
print(model.transformer)
print(f"\nvocab size: {V}   dispatched: {model.dispatched}")

model = TransformersModel("openai-community/gpt2", device_map="auto", dispatch=True) clear_output() tokenizer = model.tokenizer V = model.config.vocab_size print(model.transformer) print(f"\nvocab size: {V} dispatched: {model.dispatched}")

GPT2Model(
  (wte): Embedding(50257, 768)
  (wpe): Embedding(1024, 768)
  (drop): Dropout(p=0.1, inplace=False)
  (h): ModuleList(
    (0-11): 12 x GPT2Block(
      (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True, bias=True)
      (attn): GPT2Attention(
        (c_attn): Conv1D()
        (c_proj): Conv1D()
        (attn_dropout): Dropout(p=0.1, inplace=False)
        (resid_dropout): Dropout(p=0.1, inplace=False)
      )
      (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True, bias=True)
      (mlp): GPT2MLP(
        (c_fc): Conv1D()
        (c_proj): Conv1D()
        (act): NewGELUActivation()
        (dropout): Dropout(p=0.1, inplace=False)
      )
    )
  )
  (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True, bias=True)
)

vocab size: 50257   dispatched: True

W_E is the input embedding (wte), W_U the unembedding (lm_head), and ln_f the final layer norm between them.

In [4]:

W_E = model.transformer.wte.weight
W_U = model.lm_head.weight
ln_g = model.transformer.ln_f.weight
ln_b = model.transformer.ln_f.bias

print(f"W_E {tuple(W_E.shape)}  {W_E.dtype}  device={W_E.device}  is_meta={W_E.is_meta}")
print(f"W_U {tuple(W_U.shape)}  lm_head bias: {model.lm_head.bias}")
print(f"tied (same storage)? {W_E.data_ptr() == W_U.data_ptr()}")

W_E   = W_E.detach().float()
W_U   = W_U.detach().float()
ln_g  = ln_g.detach().float()
ln_b  = ln_b.detach().float()

W_E = model.transformer.wte.weight W_U = model.lm_head.weight ln_g = model.transformer.ln_f.weight ln_b = model.transformer.ln_f.bias print(f"W_E {tuple(W_E.shape)} {W_E.dtype} device={W_E.device} is_meta={W_E.is_meta}") print(f"W_U {tuple(W_U.shape)} lm_head bias: {model.lm_head.bias}") print(f"tied (same storage)? {W_E.data_ptr() == W_U.data_ptr()}") W_E = W_E.detach().float() W_U = W_U.detach().float() ln_g = ln_g.detach().float() ln_b = ln_b.detach().float()

W_E (50257, 768)  torch.float32  device=cuda:0  is_meta=False
W_U (50257, 768)  lm_head bias: None
tied (same storage)? True

W_E.data_ptr() == W_U.data_ptr() is True: GPT-2 ties its input and output embeddings. They are one matrix, which turns out to be the whole explanation for Finding 2.

Finding 1: an unused token's output row gives it away¶

An input embedding row is touched only when its token appears in a batch. An output (unembedding) row is touched at every step, because the softmax pushes every non-target logit down everywhere. A token that is never a target therefore still gets a gradient — the same "you are not the answer" gradient as every other never-occurring token — so all unused rows drift in the same direction. Under-training shows up as direction, not magnitude.

Hence Land & Bartolo's indicator: name a few rows that are provably never targets, take their mean as the "unused" direction, and score every token by how closely its unembedding row aligns with it. No corpus, no labels, no forward pass. GPT-2 supplies the reference set for free — its byte-level BPE has a token per byte value, and bytes 0xF5–0xFF cannot occur in valid UTF-8, so those 11 rows can never have been a target.

⚠️ Two details decide whether this reproduces: score the unembedding, not the embedding, and fold ln_f's gain into it. The logit for token $i$ is $\langle \widehat{h},\, g \odot u_i \rangle + \langle b,\, u_i \rangle$, so the vector the residual stream sees is $g \odot u_i$, not $u_i$.

In [5]:

def bytes_to_unicode():
    """GPT-2's byte <-> printable-character map. transformers 5 no longer exports this."""
    bs = (list(range(ord("!"), ord("~") + 1))
          + list(range(ord("\xa1"), ord("\xac") + 1))
          + list(range(ord("\xae"), ord("\xff") + 1)))
    cs, n = bs[:], 0
    for byte in range(256):
        if byte not in bs:
            bs.append(byte)
            cs.append(256 + n)
            n += 1
    return dict(zip(bs, [chr(c) for c in cs]))

byte_to_char = bytes_to_unicode()

# bytes 0xF5-0xFF are illegal in UTF-8: these 11 rows are provably never a training target
REFERENCE = [tokenizer.convert_tokens_to_ids(byte_to_char[b]) for b in range(0xF5, 0x100)]
print("reference token ids:", REFERENCE)

W_U_folded = W_U * ln_g                       # fold the ln_f gain into the unembedding
u_ref = W_U_folded[REFERENCE].mean(0)         # the "unused" direction
indicator = F.cosine_similarity(W_U_folded, u_ref[None], dim=-1)

top_values, top_ids = indicator.topk(30)
print(f"\n{'rank':>4}  {'id':>6}  {'cos':>7}  token")
for rank, (value, i) in enumerate(zip(top_values.tolist(), top_ids.tolist())):
    print(f"{rank:>4}  {i:>6}  {value:>7.4f}  {tokenizer.decode([i])!r}")

def bytes_to_unicode(): """GPT-2's byte <-> printable-character map. transformers 5 no longer exports this.""" bs = (list(range(ord("!"), ord("~") + 1)) + list(range(ord("\xa1"), ord("\xac") + 1)) + list(range(ord("\xae"), ord("\xff") + 1))) cs, n = bs[:], 0 for byte in range(256): if byte not in bs: bs.append(byte) cs.append(256 + n) n += 1 return dict(zip(bs, [chr(c) for c in cs])) byte_to_char = bytes_to_unicode() # bytes 0xF5-0xFF are illegal in UTF-8: these 11 rows are provably never a training target REFERENCE = [tokenizer.convert_tokens_to_ids(byte_to_char[b]) for b in range(0xF5, 0x100)] print("reference token ids:", REFERENCE) W_U_folded = W_U * ln_g # fold the ln_f gain into the unembedding u_ref = W_U_folded[REFERENCE].mean(0) # the "unused" direction indicator = F.cosine_similarity(W_U_folded, u_ref[None], dim=-1) top_values, top_ids = indicator.topk(30) print(f"\n{'rank':>4} {'id':>6} {'cos':>7} token") for rank, (value, i) in enumerate(zip(top_values.tolist(), top_ids.tolist())): print(f"{rank:>4} {i:>6} {value:>7.4f} {tokenizer.decode([i])!r}")

reference token ids: [177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187]

rank      id      cos  token
   0     183   0.9999  '�'
   1     182   0.9999  '�'
   2     178   0.9999  '�'
   3     179   0.9999  '�'
   4     187   0.9999  '�'
   5     177   0.9999  '�'
   6     185   0.9999  '�'
   7     184   0.9998  '�'
   8     181   0.9998  '�'
   9     202   0.9998  '\x0e'
  10     197   0.9998  '\t'
  11     213   0.9998  '\x19'
  12     207   0.9998  '\x13'
  13     204   0.9998  '\x10'
  14     125   0.9998  '�'
  15     205   0.9998  '\x11'
  16   39752   0.9998  'quickShip'
  17     180   0.9998  '�'
  18     208   0.9998  '\x14'
  19     200   0.9998  '\x0c'
  20   40240   0.9998  'oreAndOnline'
  21   30905   0.9998  'rawdownload'
  22     188   0.9998  '\x00'
  23     206   0.9998  '\x12'
  24     214   0.9998  '\x1a'
  25     190   0.9998  '\x02'
  26     195   0.9998  '\x07'
  27   45544   0.9998  ' サーティ'
  28   30898   0.9998  'embedreportprint'
  29     124   0.9998  '�'

Two populations sit at the top, under-trained for different reasons. ids 124–221 are the byte-fallback and control-character block — \x00, \x0e, and, surprisingly, \t: GPT-2's WebText was HTML-extracted, which normalized tabs and carriage returns away, so these are untrained despite being common in ordinary text. ids ~23,000–46,000 are junk merges from the Reddit scrape the tokenizer was fitted on — quickShip, oreAndOnline, rawdownload, embedreportprint. That second group is the SolidGoldMagikarp list, recovered from three lines of arithmetic on lm_head.weight.

The metric¶

We score a detector by recall at $k$: of the tokens a paper confirmed as under-trained in GPT-2, how many appear in our top $k$? Two published ground-truth sets, and a random permutation as the floor.

In [6]:

# Land & Bartolo (EMNLP 2024), verified under-trained in openai-community/gpt2: 33 tokens
LB_GPT2 = ['InstoreAndOnline','rawdownload','quickShip','oreAndOnline','embedreportprint',
    ' サーティ',' RandomRedditor',' externalToEVA',' TheNitrome','reportprint',' externalTo',
    'ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ','StreamerBot','ActionCode',
    'ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ','Nitrome','ÃÂÃÂÃÂÃÂ','DeliveryDate','ThumbnailImage','isSpecial',
    'oreAnd','cloneembedreportprint',' subur',' practition','Orderable',' pione','aditional',
    ' unintention',' councill','ortunately',' entreprene',' unbeliev','everal']

# Rumbelow & Watkins (2023), the SolidGoldMagikarp candidate list: 141 tokens
SGM = ['\x00','\x01','\x02','\x03','\x04','\x05','\x06','\x07','\x08','\x0e','\x0f','\x10','\x11',
    '\x12','\x13','\x14','\x15','\x16','\x17','\x18','\x19','\x1a','\x1b','\x7f','.[','ÃÂÃÂ',
    'ÃÂÃÂÃÂÃÂ','wcsstore','\\.',' practition',' Dragonbound',' guiActive',' \u200b',
    '\\\\\\\\\\\\\\\\','ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ',' davidjl','覚醒','"]=>',' --------',
    ' \u200e','ュ','ForgeModLoader','天',' 裏覚醒','PsyNetMessage',' guiActiveUn',' guiName',
    ' externalTo',' unfocusedRange',' guiActiveUnfocused',' guiIcon',' externalToEVA',
    ' externalToEVAOnly','reportprint','embedreportprint','cloneembedreportprint','rawdownload',
    'rawdownloadcloneembedreportprint','SpaceEngineers','externalActionCode','к',
    '?????-?????-','ーン','cffff','MpServer',' gmaxwell','cffffcc',' "$:/',' Smartstocks',
    '":[{"','龍喚士','":"","',' attRot',"''.",' Mechdragon',' PsyNet',' RandomRedditor',
    ' RandomRedditorWithNo','ertodd',' sqor',' istg',' "\\',' petertodd','StreamerBot',
    'TPPStreamerBot','FactoryReloaded',' partName','ヤ','\\">',' Skydragon','iHUD','catentry',
    'ItemThumbnailImage',' UCHIJ',' SetFontSize','DeliveryDate','quickShip','quickShipAvailable',
    'isSpecialOrderable','inventoryQuantity','channelAvailability','soType','soDeliveryDate',
    '龍契士','oreAndOnline','InstoreAndOnline','BuyableInstoreAndOnline','natureconservancy',
    'assetsadobe','\\-','Downloadha','Nitrome',' TheNitrome',' TheNitromeFan','GoldMagikarp',
    'DragonMagazine','TextColor',' srfN',' largeDownload',' srfAttach','EStreamFrame','ゼウス',
    ' SolidGoldMagikarp','ーティ',' サーティ',' サーティワン',' Adinida','":""},{"','ItemTracker',
    ' DevOnline','@#&','EngineDebug',' strutConnector',' Leilan','uyomi','aterasu',
    'ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ','ÃÂ','ÛÛ',' TAMADRA','EStream']

def single_token_ids(strings):
    ids = set()
    for s in strings:
        encoded = tokenizer(s, add_special_tokens=False)["input_ids"]
        if len(encoded) == 1:
            ids.add(encoded[0])
    return ids

GT_LB, GT_SGM = single_token_ids(LB_GPT2), single_token_ids(SGM)
print(f"ground truth: Land & Bartolo {len(GT_LB)}/{len(LB_GPT2)},  "
      f"SolidGoldMagikarp {len(GT_SGM)}/{len(SGM)}  (all map to single gpt2 ids)")

embed_norm = W_E.norm(dim=-1)
random_score = torch.tensor(np.random.default_rng(0).permutation(V).astype(np.float32))
KS = [25, 50, 100, 250, 500, 1000]

def recall_at_k(score, truth, ks=KS):
    order = score.argsort(descending=True)
    return [len(set(order[:k].tolist()) & truth) / len(truth) for k in ks]

DETECTORS = {"L&B indicator": indicator,
             "-embedding norm": -embed_norm,
             "random permutation": random_score}
curves = {}
print(f"\n{'detector':>20} {'truth':>5} " + " ".join(f"@{k:<6}" for k in KS))
for name, score in DETECTORS.items():
    for truth, label in [(GT_LB, "L&B"), (GT_SGM, "SGM")]:
        r = recall_at_k(score, truth)
        curves[(name, label)] = r
        print(f"{name:>20} {label:>5} " + " ".join(f"{v:<7.2f}" for v in r))

# Land & Bartolo (EMNLP 2024), verified under-trained in openai-community/gpt2: 33 tokens LB_GPT2 = ['InstoreAndOnline','rawdownload','quickShip','oreAndOnline','embedreportprint', ' サーティ',' RandomRedditor',' externalToEVA',' TheNitrome','reportprint',' externalTo', 'ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ','StreamerBot','ActionCode', 'ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ','Nitrome','ÃÂÃÂÃÂÃÂ','DeliveryDate','ThumbnailImage','isSpecial', 'oreAnd','cloneembedreportprint',' subur',' practition','Orderable',' pione','aditional', ' unintention',' councill','ortunately',' entreprene',' unbeliev','everal'] # Rumbelow & Watkins (2023), the SolidGoldMagikarp candidate list: 141 tokens SGM = ['\x00','\x01','\x02','\x03','\x04','\x05','\x06','\x07','\x08','\x0e','\x0f','\x10','\x11', '\x12','\x13','\x14','\x15','\x16','\x17','\x18','\x19','\x1a','\x1b','\x7f','.[','ÃÂÃÂ', 'ÃÂÃÂÃÂÃÂ','wcsstore','\\.',' practition',' Dragonbound',' guiActive',' \u200b', '\\\\\\\\\\\\\\\\','ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ',' davidjl','覚醒','"]=>',' --------', ' \u200e','ュ','ForgeModLoader','天',' 裏覚醒','PsyNetMessage',' guiActiveUn',' guiName', ' externalTo',' unfocusedRange',' guiActiveUnfocused',' guiIcon',' externalToEVA', ' externalToEVAOnly','reportprint','embedreportprint','cloneembedreportprint','rawdownload', 'rawdownloadcloneembedreportprint','SpaceEngineers','externalActionCode','к', '?????-?????-','ーン','cffff','MpServer',' gmaxwell','cffffcc',' "$:/',' Smartstocks', '":[{"','龍喚士','":"","',' attRot',"''.",' Mechdragon',' PsyNet',' RandomRedditor', ' RandomRedditorWithNo','ertodd',' sqor',' istg',' "\\',' petertodd','StreamerBot', 'TPPStreamerBot','FactoryReloaded',' partName','ヤ','\\">',' Skydragon','iHUD','catentry', 'ItemThumbnailImage',' UCHIJ',' SetFontSize','DeliveryDate','quickShip','quickShipAvailable', 'isSpecialOrderable','inventoryQuantity','channelAvailability','soType','soDeliveryDate', '龍契士','oreAndOnline','InstoreAndOnline','BuyableInstoreAndOnline','natureconservancy', 'assetsadobe','\\-','Downloadha','Nitrome',' TheNitrome',' TheNitromeFan','GoldMagikarp', 'DragonMagazine','TextColor',' srfN',' largeDownload',' srfAttach','EStreamFrame','ゼウス', ' SolidGoldMagikarp','ーティ',' サーティ',' サーティワン',' Adinida','":""},{"','ItemTracker', ' DevOnline','@#&','EngineDebug',' strutConnector',' Leilan','uyomi','aterasu', 'ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ','ÃÂ','ÛÛ',' TAMADRA','EStream'] def single_token_ids(strings): ids = set() for s in strings: encoded = tokenizer(s, add_special_tokens=False)["input_ids"] if len(encoded) == 1: ids.add(encoded[0]) return ids GT_LB, GT_SGM = single_token_ids(LB_GPT2), single_token_ids(SGM) print(f"ground truth: Land & Bartolo {len(GT_LB)}/{len(LB_GPT2)}, " f"SolidGoldMagikarp {len(GT_SGM)}/{len(SGM)} (all map to single gpt2 ids)") embed_norm = W_E.norm(dim=-1) random_score = torch.tensor(np.random.default_rng(0).permutation(V).astype(np.float32)) KS = [25, 50, 100, 250, 500, 1000] def recall_at_k(score, truth, ks=KS): order = score.argsort(descending=True) return [len(set(order[:k].tolist()) & truth) / len(truth) for k in ks] DETECTORS = {"L&B indicator": indicator, "-embedding norm": -embed_norm, "random permutation": random_score} curves = {} print(f"\n{'detector':>20} {'truth':>5} " + " ".join(f"@{k:<6}" for k in KS)) for name, score in DETECTORS.items(): for truth, label in [(GT_LB, "L&B"), (GT_SGM, "SGM")]: r = recall_at_k(score, truth) curves[(name, label)] = r print(f"{name:>20} {label:>5} " + " ".join(f"{v:<7.2f}" for v in r))

ground truth: Land & Bartolo 33/33,  SolidGoldMagikarp 141/141  (all map to single gpt2 ids)

            detector truth @25     @50     @100    @250    @500    @1000  
       L&B indicator   L&B 0.09    0.24    0.94    0.97    0.97    0.97   
       L&B indicator   SGM 0.09    0.21    0.32    0.33    0.33    0.33   
     -embedding norm   L&B 0.00    0.00    0.00    0.00    0.00    0.06   
     -embedding norm   SGM 0.00    0.00    0.00    0.00    0.00    0.06   
  random permutation   L&B 0.00    0.00    0.00    0.00    0.00    0.00   
  random permutation   SGM 0.00    0.00    0.00    0.01    0.01    0.02   

In [7]:

long = {"k": [], "recall": [], "detector": []}
for name in DETECTORS:
    for k, r in zip(KS, curves[(name, "L&B")]):
        long["k"].append(k); long["recall"].append(r); long["detector"].append(name)

fig = px.line(long, x="k", y="recall", color="detector", markers=True, log_x=True,
              title="Recall of Land & Bartolo's 33 verified GPT-2 tokens",
              labels=dict(k="candidates inspected (k)", recall="recall"), height=420)
fig.show()

long = {"k": [], "recall": [], "detector": []} for name in DETECTORS: for k, r in zip(KS, curves[(name, "L&B")]): long["k"].append(k); long["recall"].append(r); long["detector"].append(name) fig = px.line(long, x="k", y="recall", color="detector", markers=True, log_x=True, title="Recall of Land & Bartolo's 33 verified GPT-2 tokens", labels=dict(k="candidates inspected (k)", recall="recall"), height=420) fig.show()