Three other libraries put interpretability on vLLM: interp-engine
(Neuronpedia / Decode Research), vLLM-Lens
(UK AISI) and TransformerLens 4.0,
whose bridge can boot a vLLM engine. This page compares nnsight's vLLM integration with each,
job by job, and ends with one throughput grid run over all of them on the same machine. Every
other page in this section avoids comparison; this one is nothing else.
interp-engine is the interpretability engine behind Neuronpedia: a
fixed vocabulary of 34 named points (resid_post.10, mlp_act.5, ...) served on a hooked vLLM
backend, a CUDA-graph vllm-static backend, and a HuggingFace eager backend, with a validator
that checks its captures against TransformerLens and nnsight across 50+ architectures. It is
good, careful software, and this section deliberately follows the shape of
its documentation page for page so the two can be read
side by side.
This page is the comparison the others avoid. It was written against interp-engine 1.3.4 and
nnsight 0.8 on the same machine, and the throughput table at the bottom is one grid run over
both.
interp-engine answers a closed question well: give me point P, steered by spec S. nnsight
runs your Python inside the engine's forward: any module, any computation, any write, on the
same request path. Most of what follows is that difference worked through each job.
gate_up_proj.output split in two is mlp_pre and mlp_pre_linear; logits_processor(lm_head, h) is the unembed; the pattern is a recompute here too; expert selection is inside the fused kernel for both
The table on Locations is, in effect, interp-engine's point
list rewritten as nnsight locations on Qwen3-8B.
TransformersModel(id) — the same block, with gradients
warmup() / shutdown()
dispatch=True / process exit
num_gpus=4
tensor_parallel_size=4
configure_static(points) after construction
taps are fixed at construction
Both static/tapped engines sit on the same seam — vLLM's breakable CUDA graphs, a callable
recorded at capture and run on every replay — and both turn torch.compile off to get it.
interp-engine's "auto" static set is resid_post at every layer, read and write;
nnsight's taps are whatever module locations you name, read and write.
interp-engine asks first: model.hooks_available, model.points(), model.grad_support,
CapabilityUnsupported naming the capability and an alternative. nnsight has less to ask
because there is no point table to be absent from — the tree is the capability — and refuses
in the same spirit: a non-tap read, a barrier, a chunked prompt, a misspelled sampling keyword
and a typo'd tap each raise with the location and the fix
(Capabilities and limits).
z[:, h*d:(h+1)*d] @ W_O[:, h*d:(h+1)*d].T in the block
direct logit attribution
client-side, from captured points
every layer and head in one forward, in the worker (Attribution)
SAE features
capture the point, encode on the client
the same, or a feature's live activation in the worker from two rows (SAE features)
gradients
eager only
HuggingFace path only
Where the two differ in kind: interp-engine's cache is assembled on the client from what the
worker shipped; nnsight's block runs on the worker and ships what you saved. Capturing every layer
of a 70B model at every step costs interp-engine 3.5× its single-layer rate and nnsight nothing
measurable (table below), because the clones never leave the worker until the end.
Neither library has the attention pattern as a hook on vLLM — the paged kernel never forms it.
interp-engine rebuilds it inside capture_attention from captured q/k; on nnsight you rebuild it
in the block from self_attn.attn.inputs, in a dozen lines, and both match HuggingFace's eager
attention to bf16 noise. interp-engine additionally ships per_head_value, attn_out_gate and
split_fused_qkv for the family-specific layouts; on nnsight those are slices you write, and the
Attention page gives them for Qwen3. Under tensor parallelism the two part ways:
interp-engine's per-head z and direct attribution are single-GPU only (num_gpus > 1 shards the
heads across ranks and its off-kernel recompute sees one shard), while nnsight gathers o_proj.input
whole before the block reads it, so the per-head material above works at any tensor_parallel_size.
interp-engine's decode_residuals is a method that applies the family's post-unembed arithmetic
and runs on the client from captured residuals (a topk variant runs on the worker). nnsight
calls the model's own norm and logits_processor in the block, on the worker, so the vocab-wide
tensor never travels unless you save it — and it works at every layer in one forward on vLLM,
where interp-engine's layer_logits is eager-only.
AddSpec, OrthogonalDecompSpec, ProjectionCapSpec, in order per layer
any expression
Where
resid_post by default; point="z" and the stream points
any location — a head's slice of o_proj.input, a router logit, the logits, the sampled id
Positions
position_mask, SteerMask.SPECIAL_TOKENS
a boolean mask on the rows
During generation
the spec is applied every step
put the edit under tracer.iter[:N]
Conditional / stateful
no
yes: if step == 3, a running estimate, a probe's output deciding the write (Conditional interventions)
Activation patching
no write of another run's activation
a saved tensor written at a position, per layer or per head, batched as invokes (Activation patching)
Ablate a component
no write off the residual points
mlp.output[:] = 0, a head's slice of o_proj.input, a neuron, a router logit (Ablation)
SAE feature clamp
add a fixed decoder direction
scale the feature's live activation through generation (SAE features)
The ablate and force rows of the grid — zeroing a head's slice of o_proj.input, and
overriding the sampled token — are the two the interp-engine harness could not express on any
backend.
generate_stream yields a GenStep per token with n_logprobs; on vLLM its .logits is
None because the sampler never ships the tensor out of the worker. nnsight's model.logits is
that tensor, on the worker, readable and writable every step, and logprobs=k rides
tracer.result for the portable case. Streaming text is generate_stream on both backends in
interp-engine and mode="async" in nnsight; generate_full and tracer.result are the same
vLLM RequestOutput.
interp-engine's Tokenize helper is the richer one: message_partition, message_spans,
GeneratedTurnSpans, compose_assistant_turns, with the DeepSeek-V4 template quirks handled.
nnsight hands you the HuggingFace tokenizer and the Chat and tokens page shows
offset-mapping spans in a few lines.
interp-engine's validator compares its points against TransformerLens and nnsight/nnterp across
50+ architectures at early, middle and late layers, with the results checked into the
repository; that is a level of cross-engine validation nnsight does not have. nnsight's vLLM path
is covered by ~195 tests on two GPUs, and every value shown in this section was checked against
a HuggingFace forward of the same checkpoint. Its sharded path is checked value by value against
a one-rank engine of the same checkpoint — Qwen2.5 at tp=2, DeepSeek-V2-Lite at tp=2 and at
tp=4, dcp=2 — and the request accounting under preemption, n > 1, aborted streams and foreign
tenants sharing the batch has a test each. Both libraries note the same trap: a value that is the
right shape from the wrong place raises nothing.
See Throughput, measured at the end of the page — one grid over all
three libraries. The interp-engine columns there are vllm (hooked, eager) and vllm-static
(CUDA graphs, declared taps); vllm-generate is left out because it is vanilla vLLM under
another name (its numbers matched the vanilla column to within 1%), and the eager HuggingFace
backends are compared in the text below rather than plotted.
What the grid says about interp-engine:
Under graphs the two are the same engine.nnsight taps and IE vllm-static are within
noise of each other and of vanilla vLLM on plain generation and single-layer capture, at every
size and parallelism — as they should be, sitting on the same vLLM seam.
Both eager engines pay the same tax, and it is the driver's: 86 vs 86 tok/s on one GPU,
~70 tok/s flat as GPUs are added.
Where the computation happens is the difference that scales. Every-layer capture and the
logit lens ship tensors to interp-engine's client per request; nnsight keeps them on the
worker. At 70B: 35 vs 10 tok/s for every-layer capture, 35 vs 22 for the lens.
Gradients are the one row interp-engine leads, on its eager HuggingFace backend: one
forward+backward at 70B takes 531 ms there against 1,216 ms on nnsight's TransformersModel.
vLLM-Lens (UK AISI, MIT) is a vLLM plugin: it
registers through vLLM's general_plugins entry point, so an unmodified vllm serve or offline
LLM gains activation capture, steering vectors and Garçon-style hooks the moment the package is
installed, driven by SamplingParams.extra_args (offline) or vllm_xargs (over the OpenAI API).
It also ships an Inspect AI model provider and a set of examples — causal tracing, logit and
Jacobian lens, a deception probe, an emotion tracker, an activation oracle. This section was
written against vLLM-Lens 1.2.1 on vLLM 0.27.1, the same engine version as the rest of this site.
The Examples group of this section is the vLLM-Lens example set
redone in nnsight, so each can be read against its original.
vLLM-Lens exposes one seam — a decoder layer's residual stream, on the way in (pre-hook) or
out (post-hook) — and lets you capture it, add to it, or run a pickled function on it. nnsight
exposes every module and runs your block interleaved with the forward: the attention
projections, the per-head outputs, the router, the logits, the sampled id, and the residual
stream are all locations, and the same block reads and writes any of them.
Both libraries take the residual stream to be vLLM's (hidden, residual) summed, and both
clone before handing it to user code (vLLM-Lens clones for you; on nnsight a kept reference
must be cloned — Locations).
This is where vLLM-Lens is strongest. It lives insidevllm serve: the OpenAI-compatible
completions and chat endpoints accept vllm_xargs for capture, steering and hooks, the server
gains /v1/hooks/* for persistent hooks and parameter prefetch, activations come back base64-
encoded in the response, and any OpenAI client — or Inspect, through the bundled provider — can
drive it. nnsight's equivalent is nnsight-serve, a single-model
server that runs nnsight traces submitted by GPU-less clients and installs engine-wide edits;
it does not speak the OpenAI API. An edit installed on an nnsight engine does run on every
request the engine serves, whoever sent it, which is the persistent-hook pattern; but the
front door for OpenAI-style traffic is vLLM-Lens's.
The last row matters operationally: vLLM-Lens patches EngineArgs.create_engine_config and
LLM.generate at import, so every vLLM engine in a process that has it installed runs eager
with the worker extension attached — including one you did not mean to instrument.
vLLM-Lens is the fifth series in Throughput, measured; its column ran
in its own environment (the plugin forces eager mode on every engine in a process) on the same
cards, and the grid gained one row for it: a sweep of 1024 short prompts at one token each,
capturing one layer — the activation-extraction workload vLLM-Lens is built for.
Plain generation and single-layer capture are a wash. Both libraries hook the same seam
and both run the engine eagerly by default: 86–87 tok/s plain, 77–79 capturing one layer.
nnsight's taps column is the only one that keeps CUDA graphs, and the only one within a few
percent of vanilla.
vLLM-Lens pays per layer and per hook. Its hooks are installed on every decoder layer and
each does its bookkeeping for every in-flight request on every step, and a steering vector or
hook clones the layer's output: capturing all 32 layers halves throughput (37 vs nnsight's 68
tok/s), and a steering vector, a probe or a lens each cost about 40% (48–49 vs 78). nnsight's
block visits only the locations it names.
The sweep is close. vLLM-Lens's capture rides the request with little per-request setup
(1.32 s over 1024 prompts, 1.8× vanilla). An nnsight trace serializes a block per invoke and
collects per step (1.6 s, 2.1×); the intended shape for a sweep is
model.edit(), which installs the block once — 1.1 s eager,
1.0 s under taps, the fastest capture of the three.
Two rows vLLM-Lens cannot express: an ablation inside the attention block and an override
of the sampled token, because its hook points are layer boundaries and it has no hook on the
sampler.
A trap found while measuring, worth knowing on the nnsight side: writing
model.model.layers[16].outputinside the block references the model, and each of the 1024
invokes then serializes it — 8.6 s for the sweep. Binding the layer envoy before the trace and
using it inside is what the grid shows. Invisible in a single trace, decisive in a sweep
(Performance).
Tensor parallelism. vLLM-Lens installs its hooks on every rank; at tp=4 plain generation
matches the other eager engines (76 tok/s vs 69–72), single-layer capture too (69), but a
steering vector or probe costs a quarter (58, 57) where the eager nnsight engine and
interp-engine lose nothing, and every-layer capture drops to 37 (nnsight eager 58, taps 208).
At 70B/tp=4 the same shape: plain generation and one-layer capture on par or slightly ahead,
steering, probe and lens at 18–19 tok/s against the eager engines' 27–29.
TransformerLens 4.0 adds vLLM as a
source for its bridge: RemoteBridge.boot_vllm(id) returns an object with the familiar
run_with_cache, run_with_hooks and hook_dict, under the canonical TransformerLens hook
names, with a vLLM engine doing the forward. The same driver backs an
Inspect AI provider (tl_bridge_vllm). This section was written
against TransformerLens 4.0.0, which pins vllm>=0.20.2,<0.21; it was run there, and tried on
vLLM 0.27.1, the engine version of the rest of this site: the default compiled path works
unmodified and captures bit-identical activations, and the batched path fails at boot
('GPUModelRunner' object has no attribute 'input_batch'), which is the drift the pin is for.
TransformerLens uses vLLM as a fast prefill for a cache: one forward over a prompt, a fixed
set of module outputs copied into preallocated buffers, and a four-word vocabulary of affine
edits. nnsight runs your Python inside the engine's forward, through generation: any module,
any expression, every decode step, many requests in one batch.
This is the most interesting part of the design, and it is the opposite bet to nnsight's. The
package registers a vllm.general_plugins entry point that patches Worker.load_model, and
installs its forward hooks after the weights load and before vLLM compiles the model, so the
hooks are traced into the torch.compile graph and captured in the CUDA graphs. A hook can
therefore hold no Python: it writes output * scale + bias into a preallocated
(max_num_batched_tokens, width) buffer through a first-write-wins gate made of torch.where,
and an intervention is a swap of the contents of scale and bias between forwards. The
engine keeps compilation and graphs, which neither nnsight's taps nor interp-engine's static
backend do (both turn torch.compile off).
The cost is everything the graph cannot express, which is the rest of this section.
TransformerLens
nnsight
Hooks live
inside the compiled graph, fixed at boot
eager: Python per module call; taps: a callable at declared graph breaks
torch.compile
kept (compile cache disabled, so every boot recompiles)
off under taps; off when eager
Hook set
every hook point, always — 3 × layers + 2 buffers allocated and written on every forward, whatever you asked for
only the locations a block names
Buffer memory
(3L + 2) × max_num_batched_tokens × d_model: 1.6 GB on Llama-8B at the default 2048 tokens, doubled again by enable_position_interventions
none resident
Requests per forward
one (batch_size=1); enable_batching=True lifts it by switching the engine to enforce_eager
many invokes, batched by the scheduler, either mode
Prefix caching
forced off
on by default: a trace asks for a recompute of its own prompt, plain requests still hit the cache; only an engine-wide edit() needs it off
In other engines in the process
the patch is inert unless boot_vllm set its spec channel
reconstructed on the client as ln_final @ W_U in fp32, full sequence — so return_type="loss" works
model.logits on the worker, readable and writable
The sampled id
—
model.samples
Inside a module's forward
—
.source ops
TransformerLens is careful about what it cannot do: every unavailable name is declared up
front (non_fireable_hook_points), asking for one raises this backend cannot fire
'blocks.8.attn.hook_z'; use boot_transformers(), a hook that installed on no rank fails the
boot, and a tensor-parallel boot cross-checks that every rank's copy of a capture agrees. One
architecture overlay covers every decoder-only model by vLLM's conventional module paths
(model.layers.{i}, .self_attn, .mlp), and the models that fall outside it fail at boot
rather than reading zeros: Qwen3.5-0.8B, whose composite multimodal config has no
hidden_size, fails before the engine is built on either path, and Qwen1.5-MoE-A2.7B fails
on the compiled path when torch.compile rejects the graph with the hooks in it
(ConstraintViolationError on the token dimension; plain vLLM compiles the same model, and
the eager batched path runs it).
no (bwd_hooks, incl_bwd raise) — use boot_transformers
HuggingFace path only
Two defaults are worth knowing before timing anything. A run_with_cache with no
names_filter copies every hook point to the client, and every forward rebuilds full-sequence
logits on the CPU unless you pass return_logits=False, a driver argument the bridge passes
through. On Llama-3.2-1B with a 512-token prompt: 1,571 ms by default, 217 ms
for one hook with logits, 32 ms for one hook without. The batched path has a third: its eager
hook copies every hook point of every request to the CPU as it fires, and names_filter
only trims what is concatenated afterwards.
no: the full spec set is pushed, and reset, every forward
model.edit()
An add does what it should — a steering vector at layer 8 of Llama-3.2-1B moves the top
prediction after The Eiffel Tower is in the city of off Paris — and the write is carried
into the next layer by returning (modified − residual, residual), exact at identity and
bf16-rounded under an edit.
On Llama-3.2-1B against a HuggingFace forward of the same checkpoint, TransformerLens's vLLM
captures agree at every hook family (cosine 0.9998–1.0000; block output exact to bf16
rounding), its reconstructed logits pick the same top token at every position, and the
captures are bit-identical between vLLM 0.20.2 and 0.27.1. The library's own end-to-end check
of the compiled mutation path is a notebook run by hand on a GPU; its CI covers the dispatch
protocol.
TransformerLens is the yellow pair in Throughput, measured, on the
panels it was run on, and most of its cells are ✗: the grid is built around generation — a
512-token prompt and 128 new tokens, with something read or written at every step — and the
bridge does one forward per call. The rows it can express are the single forward with a
capture and the 1024-prompt sweep. Its columns ran in their own environment on vLLM 0.20.2,
the version it pins, so each dot is a share of plain vLLM 0.20.2 measured in that same
environment (the vanilla vLLM 0.20.2 column of the tables) rather than of the 0.27.1
vanilla the other libraries are read against.
Two engines, neither of which is both fast and batched. The default path keeps
torch.compile and CUDA graphs but takes one prompt per forward, so the sweep is 1,024
sequential generate calls, each bracketed by two RPCs to push the intervention state and
open the capture gates and one to read the buffer back: 25.9 s on Llama-8B (23.8 s on
Qwen3-8B, 7.7 s on Llama-1B), against 0.76 s for plain vLLM 0.20.2 given the same prompts as
one batch and 1.6 s for an nnsight trace. Most of that is the shape rather than the hook —
plain vLLM driven one prompt per call, nothing attached, takes 15–19 s — and the same path
on vLLM 0.27.1 is 20% slower again (29.8 s). enable_batching=True submits the sweep as one
batch, and gives up compilation and graphs to do it: 7.2 s, 4.5× nnsight's per-request trace
and 6.6× its edit().
The batched hook copies everything. On the eager path every hook point of every request
is sliced and moved to the CPU as it fires — 98 tensors per request on Llama-8B — and
names_filter is applied afterwards, so asking for one layer costs the same as asking for
all of them. It shows most on the small model: a single forward on Llama-1B is 72 ms
batched against 13 ms compiled and 9 ms for plain vLLM.
A single forward is where it is closest, provided the two defaults above are turned
off (names_filter=[...], return_logits=False): 62 ms on Llama-8B against 37 ms for plain
vLLM 0.20.2 and 53 ms for an nnsight trace on 0.27.1; the compiled hooks add an affine and a
gated copy at every hook point whether or not anything was asked for. With the defaults on
it is about 370 ms.
Tensor parallelism makes it slower. From one card to tensor_parallel_size=4, plain
vLLM's single forward drops 37 → 25 → 19 ms and its one-per-call sweep 15–19 → 9 → 6 s;
TransformerLens's forward goes the other way, 62 → 73 → 101 ms, because every capture read
now crosses every rank and is merged on the client, and its sweep comes down only with the
engine (25.9 → 14.1 → 12.6 s). nnsight's trace follows vanilla (53 → 45 → 39 ms;
1.6 → 1.3 → 1.2 s).
The mixture of experts is its best panel, on the path that survives. On
Qwen1.5-MoE-A2.7B the compiled path does not boot (above), but the eager batched path is
closer to nnsight than anywhere else: 2.8 s for the sweep against nnsight's 1.4 s
(0.7 s with edit()), and 62 ms for a single forward against 46 ms. An MoE forward is
expensive relative to the hook's fixed per-request copy, so the copy shows less.
Every other row is ✗, for one of three reasons recorded in the tables' hover text: no
generation through the bridge; no hook point inside the attention block or on the sampler;
no backward pass.
One harness, every library, the same machine: bf16, A100-80GB, vLLM 0.27.1 (TransformerLens
on the 0.20.2 it pins, read against that version's vanilla),
transformers 5.15, 512-token prompt, 128 new tokens, greedy, prefix caching off on every
engine, 3 processes × 3 timed runs per cell (mean; std ≤ 2% except the HuggingFace-eager rows,
which are not plotted). Each dot is a library's throughput on a workload as a share of plain
vLLM doing the same generation with nothing attached — vanilla generate for single-stream
rows, vanilla 8-concurrent generate for the ×8 rows, and vanilla's own sweep time for the
sweep rows (where less time is more). Hollow dots are eager engines, filled dots keep CUDA
graphs; hue is the library. The whisker through each dot is the min–max over its repeated runs
(nine for most cells: three processes × three timed runs; the 70B and tp=8 panels have six).
A cell that is statistically significantly faster than its nnsight counterpart — the eager
engines and vLLM-Lens against nnsight eager, vllm-static against nnsight taps — by an
exact two-sided Mann-Whitney U test at p < 0.05 and at least 3% apart is bold in the tables;
a bold nnsight cell beat every counterpart in its row by the same test. Hover a dot for the
number, the run count and the range.
Every panel was measured in one session on one machine, which is what makes the libraries
comparable; the absolute tok/s of an eager row is not. A graph engine is bound by the GPU and
reproduces: re-run on the same machine while other tenants are running, vanilla vLLM gives 91.8 tok/s
against the 92 below and nnsight taps 88.6 against 89. An eager engine spends a Python round
trip per module call on the driver, so its throughput follows whatever CPU the host has left, and
the whole nnsight eager column on that re-run came back between 44 and 58 tok/s instead of 68 to
86. Plain vLLM with enforce_eager=True and nothing attached moves with it (52.1 tok/s there,
against nnsight's 54.1), which is the reason the eager rows on
Performance are published as a share of that engine rather than as a
rate.
The same grid on more models and two more workload shapes (three processes each; the
DeepSeek panel has no interp-engine column, whose runner was not pointed at it):
Qwen3-8B (36 layers with QK-norm, the model this section's examples use). The eager
engines are all at ~74% of vanilla on plain generation (67–69 tok/s vs 91) where on
Llama-8B they were at 93%: the eager tax scales with the number of module calls per layer, and
Qwen3 has more of them. The graph engines are unaffected (nnsight taps 88, interp-engine
static 88). Everything else has the Llama shape — nnsight leads on every-layer capture and the
lens, interp-engine's hooked engine leads by ~10% on single-layer capture, steering and the
probe, vLLM-Lens pays ~35% for a hook or steering vector.
Qwen1.5-MoE-A2.7B (60 experts, 4 active). The eager engines collapse to 20% of vanilla
(40 tok/s vs 210): an MoE layer is many small kernels, and each is a Python round trip on an
eager path. nnsight's taps keep 98% (205); interp-engine's static engine 86% (181); vLLM-Lens
is an eager engine and sits with the eager nnsight and interp-engine at 40–42. At 8 concurrent
requests nnsight taps (1,091) and interp-engine static (874) both exceed vanilla (671),
which turns out to be vLLM's torch.compile path losing on this model — see the note under
the MoE chart.
DeepSeek-V2-Lite (MLA attention + MoE, 27 layers, 16B). Same picture, stronger:
eager engines at 19% of vanilla (31 vs 164), nnsight taps at 98% (161). The MLA path
(kv_a_proj, kv_b_proj, the absorbed decode kernel) adds module calls that only a
graph engine hides.
Llama-3.2-1B. With a small model the fixed per-step Python cost dominates: the eager
engines are at 37–39% of vanilla, the graph engines at 94–95%, and vLLM-Lens's hook costs
show most clearly (steering 116 vs interp-engine 145, every-layer capture 96 vs nnsight 119).
Long context (2048-token prompt, 512 new tokens, Llama-8B). Longer decode amortizes
per-step overheads: every eager engine moves up a few points relative to the 512/128 grid, the
ordering is unchanged, and vLLM-Lens's per-hook cost persists (steering 50, probe 49 vs the
eager engines' 75–84).
32 concurrent requests (Llama-8B). Plain generation is a wash for every engine
(1,430–1,470 tok/s vs vanilla 1,523). Capturing one layer on all 32 streams costs nnsight
6–10% (1,330 eager, 1,367 taps), interp-engine 4–8% (1,408 static, 1,371 hooked) and
vLLM-Lens 25% (1,100): at this batch size the per-request bookkeeping that all three do on
every step is what shows, and the graph engines' decode advantage is mostly spent.
The numbers — Llama-3.1-8B, one GPU (tok/s; sweeps in seconds)
workload
vanilla vLLM
nnsight eager
nnsight taps (CUDA graphs)
interp-engine vllm
interp-engine vllm-static
vLLM-Lens
vanilla vLLM 0.20.2
TransformerLens batched
TransformerLens (compile + CUDA graphs)
generate
92
86
89
86
90
86
92
✗
✗
generate, 8 concurrent
618
577
597
580
596
578
616
✗
✗
capture 1 layer, every step
·
79
89
84
89
75
·
✗
✗
capture every layer, every step
·
68
88
48
71
37
·
✗
✗
capture 1 layer, 8 concurrent
·
529
577
568
588
470
·
✗
✗
additive steering, 1 layer
·
78
89
85
89
49
·
✗
✗
logit lens every step
·
78
84
63
65
47
·
✗
✗
linear probe every step
·
78
89
84
89
49
·
✗
✗
zero one attention head every step
·
79
89
✗
✗
✗
·
✗
✗
override the sampled token every step
·
79
89
✗
✗
✗
·
✗
✗
one forward over 512 tokens, capture 1 layer
45 ms
53 ms
53 ms
43 ms
44 ms
45 ms
37 ms
98 ms
62 ms
sweep: 1024 × 1 token, capture 1 layer, per request
0.75 s
1.60 s
1.48 s
·
·
1.33 s
0.76 s
7.18 s
25.90 s
sweep: 1024 × 1 token, capture 1 layer, edit() once
·
1.09 s
1.00 s
·
·
✗
·
✗
✗
The numbers — Llama-3.1-8B, tp=2
workload
vanilla vLLM
nnsight eager
nnsight taps (CUDA graphs)
interp-engine vllm
interp-engine vllm-static
vLLM-Lens
vanilla vLLM 0.20.2
TransformerLens (compile + CUDA graphs)
generate
148
72
142
71
142
75
149
✗
generate, 8 concurrent
933
525
898
520
893
537
942
✗
capture 1 layer, every step
·
67
140
71
139
67
·
✗
capture every layer, every step
·
59
138
36
65
35
·
✗
capture 1 layer, 8 concurrent
·
462
851
507
859
424
·
✗
additive steering, 1 layer
·
66
139
72
139
53
·
✗
logit lens every step
·
66
134
43
59
51
·
✗
linear probe every step
·
66
138
72
139
53
·
✗
zero one attention head every step
·
66
✗
✗
✗
✗
·
✗
override the sampled token every step
·
67
140
✗
✗
✗
·
✗
one forward over 512 tokens, capture 1 layer
30 ms
45 ms
46 ms
37 ms
38 ms
36 ms
25 ms
73 ms
sweep: 1024 × 1 token, capture 1 layer, per request
0.49 s
1.33 s
1.42 s
·
·
1.17 s
0.49 s
14.05 s
sweep: 1024 × 1 token, capture 1 layer, edit() once
·
1.05 s
1.05 s
·
·
✗
·
✗
The numbers — Llama-3.1-8B, tp=4
workload
vanilla vLLM
nnsight eager
nnsight taps (CUDA graphs)
interp-engine vllm
interp-engine vllm-static
vLLM-Lens
vanilla vLLM 0.20.2
TransformerLens (compile + CUDA graphs)
generate
228
72
216
69
216
76
231
✗
generate, 8 concurrent
1,431
532
1,355
512
1,345
550
1,460
✗
capture 1 layer, every step
·
66
213
69
206
69
·
✗
capture every layer, every step
·
58
208
29
54
37
·
✗
capture 1 layer, 8 concurrent
·
463
1,256
482
1,163
430
·
✗
additive steering, 1 layer
·
66
210
71
210
58
·
✗
logit lens every step
·
66
205
33
48
55
·
✗
linear probe every step
·
66
209
69
206
57
·
✗
zero one attention head every step
·
66
✗
✗
✗
✗
·
✗
override the sampled token every step
·
67
213
✗
✗
✗
·
✗
one forward over 512 tokens, capture 1 layer
23 ms
39 ms
39 ms
34 ms
34 ms
29 ms
19 ms
101 ms
sweep: 1024 × 1 token, capture 1 layer, per request
0.33 s
1.20 s
1.18 s
·
·
1.00 s
0.33 s
12.59 s
sweep: 1024 × 1 token, capture 1 layer, edit() once
·
1.46 s
1.35 s
·
·
✗
·
✗
The numbers — Llama-3.1-70B, tp=4
workload
vanilla vLLM
nnsight eager
nnsight taps (CUDA graphs)
interp-engine vllm
interp-engine vllm-static
vLLM-Lens
generate
37
30
36
29
36
32
generate, 8 concurrent
235
200
226
203
226
214
capture 1 layer, every step
·
27
35
29
35
28
capture every layer, every step
·
23
35
7
10
10
capture 1 layer, 8 concurrent
·
179
220
187
212
170
additive steering, 1 layer
·
27
35
29
35
19
logit lens every step
·
27
35
19
22
18
linear probe every step
·
27
35
29
35
19
zero one attention head every step
·
27
35
✗
✗
✗
override the sampled token every step
·
27
35
✗
✗
✗
The numbers — Qwen3-8B, one GPU
workload
vanilla vLLM
nnsight eager
nnsight taps (CUDA graphs)
interp-engine vllm
interp-engine vllm-static
vLLM-Lens
vanilla vLLM 0.20.2
TransformerLens batched
TransformerLens (compile + CUDA graphs)
generate
91
67
88
69
88
68
91
✗
✗
generate, 8 concurrent
604
471
583
496
580
490
605
✗
✗
capture 1 layer, every step
·
60
87
67
87
60
·
✗
✗
capture every layer, every step
·
52
86
39
67
32
·
✗
✗
capture 1 layer, 8 concurrent
·
419
562
475
568
387
·
✗
✗
additive steering, 1 layer
·
59
87
67
87
42
·
✗
✗
logit lens every step
·
61
82
51
63
41
·
✗
✗
linear probe every step
·
60
87
68
87
42
·
✗
✗
zero one attention head every step
·
61
87
✗
✗
✗
·
✗
✗
override the sampled token every step
·
60
87
✗
✗
✗
·
✗
✗
one forward over 512 tokens, capture 1 layer
44 ms
54 ms
55 ms
43 ms
46 ms
45 ms
37 ms
101 ms
51 ms
sweep: 1024 × 1 token, capture 1 layer, per request
0.78 s
1.74 s
1.59 s
·
·
1.41 s
0.77 s
6.64 s
23.75 s
sweep: 1024 × 1 token, capture 1 layer, edit() once
·
1.19 s
1.04 s
·
·
✗
·
✗
✗
The vanilla column on this model is vLLM's default engine: CUDA graphs andtorch.compile.
On this MoE the compiled path is the slow one at 8 concurrent requests. Measured directly on
plain vLLM, same prompts and settings: eager 315 tok/s; torch.compile alone 245 (the
compiled forward is slower than eager on this model); compile + CUDA graphs, the default,
669; CUDA graphs with compilation off 1,095 (1,099 in the breakable-graph mode nnsight's taps
use). Enabling vLLM's own custom kernels under compile (custom_ops=all) changes nothing
(668), so it is the inductor-compiled forward itself, not a missing fused op. nnsight taps
(1,091) and interp-engine static (874) are therefore not beating vLLM; they run the un-compiled
graph path that vLLM's default loses to on this model. Single-request generation is unaffected
(210 default vs 205 taps).
The numbers — Qwen1.5-MoE-A2.7B, one GPU
workload
vanilla vLLM
nnsight eager
nnsight taps (CUDA graphs)
interp-engine vllm
interp-engine vllm-static
vLLM-Lens
vanilla vLLM 0.20.2
TransformerLens batched
generate
210
40
205
41
181
42
209
✗
generate, 8 concurrent
671
303
1,091
309
874
312
673
✗
capture 1 layer, every step
·
38
202
41
179
39
·
✗
capture every layer, every step
·
35
200
34
148
33
·
✗
capture 1 layer, 8 concurrent
·
280
1,023
304
863
279
·
✗
additive steering, 1 layer
·
38
201
41
180
36
·
✗
logit lens every step
·
38
188
34
99
35
·
✗
linear probe every step
·
38
200
40
179
36
·
✗
zero one attention head every step
·
38
202
✗
✗
✗
·
✗
override the sampled token every step
·
38
202
✗
✗
✗
·
✗
one forward over 512 tokens, capture 1 layer
31 ms
46 ms
43 ms
36 ms
34 ms
37 ms
29 ms
62 ms
sweep: 1024 × 1 token, capture 1 layer, per request
0.44 s
1.36 s
1.27 s
·
·
0.82 s
0.44 s
2.81 s
sweep: 1024 × 1 token, capture 1 layer, edit() once
·
0.71 s
0.66 s
·
·
✗
·
✗
The numbers — Llama-3.2-1B, one GPU
workload
vanilla vLLM
nnsight eager
nnsight taps (CUDA graphs)
interp-engine vllm
interp-engine vllm-static
vLLM-Lens
vanilla vLLM 0.20.2
TransformerLens batched
TransformerLens (compile + CUDA graphs)
generate
395
145
372
150
376
155
359
✗
✗
generate, 8 concurrent
2,548
1,090
2,421
1,102
2,405
1,148
2,353
✗
✗
capture 1 layer, every step
·
135
366
143
370
135
·
✗
✗
capture every layer, every step
·
119
361
102
295
96
·
✗
✗
capture 1 layer, 8 concurrent
·
940
2,224
1,032
2,339
882
·
✗
✗
additive steering, 1 layer
·
134
365
145
370
116
·
✗
✗
logit lens every step
·
134
325
90
145
109
·
✗
✗
linear probe every step
·
134
364
143
367
112
·
✗
✗
zero one attention head every step
·
135
367
✗
✗
✗
·
✗
✗
override the sampled token every step
·
136
365
✗
✗
✗
·
✗
✗
one forward over 512 tokens, capture 1 layer
11 ms
19 ms
18 ms
14 ms
14 ms
15 ms
9 ms
60 ms
13 ms
sweep: 1024 × 1 token, capture 1 layer, per request
0.22 s
0.90 s
1.07 s
·
·
0.56 s
0.19 s
3.73 s
8.07 s
sweep: 1024 × 1 token, capture 1 layer, edit() once
·
0.54 s
0.48 s
·
·
✗
·
✗
✗
The numbers — Llama-3.1-8B, 2048-token prompt, 512 new tokens
workload
vanilla vLLM
nnsight eager
nnsight taps (CUDA graphs)
interp-engine vllm
interp-engine vllm-static
vLLM-Lens
generate
90
84
87
85
87
85
generate, 8 concurrent
574
539
558
537
553
539
capture 1 layer, every step
·
77
87
84
87
76
capture every layer, every step
·
66
86
48
67
38
capture 1 layer, 8 concurrent
·
530
553
528
544
468
additive steering, 1 layer
·
75
86
84
87
50
logit lens every step
·
77
82
65
64
48
linear probe every step
·
75
86
83
87
49
zero one attention head every step
·
77
87
✗
✗
✗
override the sampled token every step
·
76
87
✗
✗
✗
sweep: 1024 × 1 token, capture 1 layer, per request
0.76 s
1.64 s
1.54 s
·
·
1.35 s
sweep: 1024 × 1 token, capture 1 layer, edit() once
·
1.12 s
1.03 s
·
·
✗
The numbers — Llama-3.1-8B, 32 concurrent
workload
vanilla vLLM
nnsight eager
nnsight taps (CUDA graphs)
interp-engine vllm
interp-engine vllm-static
vLLM-Lens
generate, 32 concurrent
1,523
1,434
1,469
1,430
1,456
1,455
capture 1 layer, 32 concurrent
·
1,325
1,366
1,371
1,408
1,100
The numbers — DeepSeek-V2-Lite, one GPU
workload
vanilla vLLM
nnsight eager
nnsight taps (CUDA graphs)
vLLM-Lens
generate
164
31
161
32
generate, 8 concurrent
873
232
871
237
capture 1 layer, every step
·
29
159
30
capture every layer, every step
·
27
157
26
capture 1 layer, 8 concurrent
·
213
810
215
additive steering, 1 layer
·
29
158
28
logit lens every step
·
29
152
27
linear probe every step
·
29
157
27
zero one attention head every step
·
29
✗
✗
override the sampled token every step
·
29
159
✗
sweep: 1024 × 1 token, capture 1 layer, per request
0.50 s
1.53 s
1.42 s
0.92 s
sweep: 1024 × 1 token, capture 1 layer, edit() once
·
0.94 s
0.71 s
✗
The numbers — Qwen3.5-0.8B, one GPU
workload
vanilla vLLM
nnsight eager
nnsight taps (CUDA graphs)
interp-engine vllm
interp-engine vllm-static
vLLM-Lens
generate
420
47
251
48
241
48
generate, 8 concurrent
2,522
346
1,545
354
1,406
358
capture 1 layer, every step
·
45
245
49
239
35
capture every layer, every step
·
41
239
40
215
32
capture 1 layer, 8 concurrent
·
319
1,374
352
1,370
254
additive steering, 1 layer
·
44
241
49
239
32
logit lens every step
·
44
226
36
89
31
linear probe every step
·
44
238
48
238
32
zero one attention head every step
·
45
✗
✗
✗
✗
override the sampled token every step
·
45
245
✗
✗
✗
sweep: 1024 × 1 token, capture 1 layer, per request
0.45 s
1.30 s
1.28 s
·
·
0.81 s
sweep: 1024 × 1 token, capture 1 layer, edit() once
·
0.91 s
0.82 s
·
·
✗
The numbers — Qwen3.5-4B, one GPU
workload
vanilla vLLM
nnsight eager
nnsight taps (CUDA graphs)
interp-engine vllm
interp-engine vllm-static
vLLM-Lens
generate
140
35
106
37
103
36
generate, 8 concurrent
857
263
648
274
627
270
capture 1 layer, every step
·
33
104
36
102
26
capture every layer, every step
·
31
103
28
87
22
capture 1 layer, 8 concurrent
·
242
604
263
623
193
additive steering, 1 layer
·
33
104
36
102
24
logit lens every step
·
33
97
28
60
23
linear probe every step
·
33
104
36
102
24
zero one attention head every step
·
33
✗
✗
✗
✗
override the sampled token every step
·
33
104
✗
✗
✗
sweep: 1024 × 1 token, capture 1 layer, per request
0.96 s
2.01 s
2.01 s
·
·
1.59 s
sweep: 1024 × 1 token, capture 1 layer, edit() once
·
1.65 s
1.49 s
·
·
✗
The numbers — Qwen3.6-35B-A3B (MoE), tp=2
workload
vanilla vLLM
nnsight eager
nnsight taps (CUDA graphs)
interp-engine vllm
interp-engine vllm-static
vLLM-Lens
generate
182
15
119
16
108
15
generate, 8 concurrent
732
112
545
115
491
117
capture 1 layer, every step
·
14
116
16
107
13
capture every layer, every step
·
14
114
13
74
12
capture 1 layer, 8 concurrent
·
105
487
116
492
95
additive steering, 1 layer
·
14
115
16
107
12
logit lens every step
·
14
111
12
38
12
linear probe every step
·
14
115
15
106
12
zero one attention head every step
·
14
✗
✗
✗
✗
override the sampled token every step
·
14
116
✗
✗
✗
sweep: 1024 × 1 token, capture 1 layer, per request
0.82 s
2.27 s
2.11 s
·
·
1.62 s
sweep: 1024 × 1 token, capture 1 layer, edit() once
·
1.59 s
1.52 s
·
·
✗
The three panels above are hybrid gated-delta-net trunks (Qwen3.5-0.8B: 18 of 24 layers are
recurrent), and they needed a correctness fix before any taps number could be published: a full
CUDA graph captured over such a trunk silently miscomputes prefill (plain vLLM does, with
compilation off — the recurrent layers branch on the batch's prefill/decode composition), so a
tapped nnsight engine now pins cudagraph_mode="FULL_DECODE_ONLY" on any model vLLM reports as
hybrid or attention-free: prefill runs eagerly, decode keeps replay, and tapped generation
matches eager exactly. Two consequences show in the numbers. The taps-to-vanilla gap is wider
here (60–76%) than on standard trunks (93–97%), because vanilla's torch.compile genuinely pays
on these new architectures and taps run without it — yet taps still lead every graph-mode
alternative on these models. And the eager engines collapse hardest of any model measured
(8–25% of vanilla, identically for all three libraries): a recurrent layer is many small kernels,
each a Python round trip, and vanilla is very fast. The head-ablation row is ✗ under taps here
because only layer outputs are tapped on these trunks.
The numbers — Llama-3.1-8B, tp=8
workload
vanilla vLLM
nnsight eager
nnsight taps (CUDA graphs)
interp-engine vllm
interp-engine vllm-static
vLLM-Lens
generate
313
69
286
72
289
74
generate, 8 concurrent
1,689
496
1,568
511
1,540
525
capture 1 layer, every step
·
64
284
68
251
65
capture every layer, every step
·
57
276
22
36
36
capture 1 layer, 8 concurrent
·
442
1,443
440
1,082
415
additive steering, 1 layer
·
64
278
70
251
57
logit lens every step
·
63
275
23
30
52
linear probe every step
·
64
276
68
245
56
zero one attention head every step
·
64
✗
✗
✗
✗
override the sampled token every step
·
65
284
✗
✗
✗
sweep: 1024 × 1 token, capture 1 layer, per request
0.25 s
1.21 s
1.18 s
·
·
1.02 s
sweep: 1024 × 1 token, capture 1 layer, edit() once
·
2.24 s
2.21 s
·
·
✗
The numbers — Llama-3.1-70B, tp=8
workload
vanilla vLLM
nnsight eager
nnsight taps (CUDA graphs)
interp-engine vllm
interp-engine vllm-static
vLLM-Lens
generate
61
30
59
32
59
32
generate, 8 concurrent
351
202
331
218
334
219
capture 1 layer, every step
·
28
58
31
55
29
capture every layer, every step
·
24
57
5
7
10
capture 1 layer, 8 concurrent
·
185
323
189
279
173
additive steering, 1 layer
·
27
58
31
56
21
logit lens every step
·
28
58
16
21
21
linear probe every step
·
28
58
30
56
22
zero one attention head every step
·
28
✗
✗
✗
✗
override the sampled token every step
·
28
58
✗
✗
✗
sweep: 1024 × 1 token, capture 1 layer, per request
3.16 s
5.51 s
5.12 s
·
·
5.70 s
sweep: 1024 × 1 token, capture 1 layer, edit() once
·
6.24 s
5.70 s
·
·
✗
The two panels above extend the tensor-parallel series to all eight cards (three trials for
8B, two for 70B, on a shared node — runs that overlapped another user's job were dropped, and
the trials that remain agree within 3%). The trend from tp=2 and tp=4 simply continues. Plain
vLLM keeps scaling (8B: 92 → 148 → 229 → 313 tok/s from one to eight cards; 70B: 37 → 61 from
four to eight), and the graph engines follow it: capturing a layer every step, nnsight taps
hold 91% of vanilla on 8B and 95% on 70B, interp-engine static 80% and 90%. The eager engines
do not move at all — nnsight eager, interp-engine's hooked engine and vLLM-Lens sit at 64–74
tok/s on 8B at every card count, which is now 20–24% of vanilla, and at 28–32 on 70B (46–53%;
the heavier step hides more of the per-module handoff). Where the libraries differ is what they
serve under graphs: every-layer capture on 70B is 57 tok/s under taps against 5–10 for the
three engines that gather the tensors out of the worker. One number goes the other way: the
edit()-once sweep, the fastest capture on one GPU, gets slower with every rank added while the
per-request trace does not — on 8B, 1.05 s at tp=2, 1.35–1.46 s at tp=4, 2.2 s at tp=8, against
a per-request trace steady at 1.2–1.4 s (70B/tp=8: 5.7 against 5.1). An installed block runs
its saves on every rank; a per-request one collects once. Past four cards, trace the sweep.
The harness (ie-bench/: one runner per library, common.py for the rows, report.py and the
chart script), the raw results-*.jsonl and every log live alongside nnsight's tests; every
number above was produced by it.