GPUModelRunner¶
GPUModelRunner
¶
Run interventions inside vLLM's worker, against the real weights.
This is where a trace written in another process actually happens. The runner
builds its own VLLM over the module vLLM
loaded, so the module tree here has the same paths the client wrote against; a
worker arriving on a request then resolves straight onto the real modules.
Three points on vLLM's own path carry it:
_update_states— the scheduler has just decided what runs this step, so new requests hand over their workers and every worker's token span is recomputed.execute_model— the forward, run with the interleaver open so hooks serve the parked workers;_model_forwardre-points them from tokens to rows as soon as the forward returns, before the logits are computed._sample— logits and sampled ids never pass through a module, so they are offered to workers directly by location.
Workers run as greenlets on this thread. _update_states is called from
execute_model, and hooks fire on whichever thread runs the forward, so the
worker and the model take strict turns on one thread — there is nothing to
synchronize during the forward. Collection (collect_nnsight) runs on that
thread too, between steps, which is what lets it resume a worker to serve
tracer.result or unwind it; the one executor where it would not, Ray's
compiled-graph one, is refused (see NNsightGPUWorker).
Request
¶
What this engine carries for one of its requests.
vLLM names a request "{external}-{8 hex}" on the way in and an n > 1
child "{index}_{parent}", while the engine asks about — and returns —
the external id; both spellings are parsed here once. mediator is the
traced worker, if the request is an nnsight trace; copies are the
registered blocks' per-request workers, moved to harvested once the
request is over; error is a payload that failed to deserialize.
__slots__
class-attribute
instance-attribute
¶
key
¶
(engine_id, sequence index) as the engine that asked about ids knows it.
The child reading is taken only when the parent it names is one the engine asked about, so an id that merely starts with digits and an underscore is not mistaken for somebody's second sequence.
deferred
¶
The request's deferred error, captured for the client, or None.
The request's own error first — a block that failed to deserialize, or an
edits= name nothing is installed under — then the traced block's.
Requests
¶
The workers riding this engine's in-flight requests.
Two kinds of worker run here. A traced one arrives on its request, one per
request, and goes home when that request finishes. A registered one is a
block left on the engine ahead of time (see
nnsight.modeling.vllm.registration):
the template is deserialized once, and every request that arrives afterwards
— whether or not it is an nnsight trace at all — gets a fresh copy with a
scope of its own, whose saves are kept here until they are collected.
| ATTRIBUTE | DESCRIPTION |
|---|---|
requests |
Request id ->
TYPE:
|
templates |
Registration id -> the deserialized block each request's copy is built from, so the source is compiled once rather than per request.
TYPE:
|
names |
Registration id -> the name it was installed under, or
TYPE:
|
register
¶
register(registration_id: str, payload: bytes, persistent_objects: dict, name: str | None = None) -> None
Take a block the engine should run for every request from now on.
unregister
¶
Stop running a block and forget anything it has not handed back.
add
¶
Take the worker off each new request that carries one.
A request with no nnsight payload is another tenant of the same engine and
runs only the registered blocks — it still occupies tokens in the batch,
so scope
counts it. A payload that fails to deserialize is recorded as that
request's error and surfaced at collect, rather than raised inside
execute_model where it would take the engine every tenant shares down.
A request this engine already carries is one vLLM preempted and is
resuming; its workers continue (the engine replays the tokens they already
saw inside one recompute step, so a fresh block would be short by exactly
those steps — scope keeps the steps they sat out off their count).
refuse_chunked
¶
Give a request whose prompt this step only partly prefills its error, not a worker.
Chunked prefill (off by default on an nnsight engine) splits a prompt
across steps: a block would see a slice of its prompt now and the rest
later, and the sample of every chunk but the last is one vLLM discards.
states is the runner's per-request state, which knows how much of the
prompt is computed before this step.
scope
¶
Point every worker at its own tokens within this step's batch.
spans is each scheduled request and its token count, in the order the
forward's tensors follow. A worker's span is only meaningful for the step
it was computed in, so every worker's is recomputed: workers whose request
isn't running now report no group and are dropped from the interleaver, and
a block that already ran to completion is dropped too, since the
interleaver starts anything not alive and a finished block must not run a
second time.
unflatten
¶
Re-point each worker from its tokens to its row.
Logits and sampled ids carry one row per request, not per token, so the spans that scoped the forward would select the wrong thing.
harvest
¶
Shelve finished requests' registered values until they are collected.
Driven by the scheduler's own finished set, and again by a collect that finds a request not yet harvested — the scheduler's pass happens at the top of the next step, which on the async path may come after the collect, or (for the last request in flight) never. Off the scheduler rather than only off a collect so a registration works for requests nobody is tracing — the OpenAI server's, say — where nothing would otherwise come asking. A copy still parked when its request ends was waiting for a location this request never reached, which is ordinary for a registration, so it is unwound quietly and whatever it saved is kept.
serve_result
¶
Hand a finished request's output to a worker parked on tracer.result.
A worker parked anywhere else is left alone — its own location is what it
is waiting for, and finish_dangling reports it.
finish_dangling
¶
Surface a worker still parked when its request has finished.
A worker still alive at the end was waiting on a location the model
never reached — the interleaver's check_dangling_mediators, but for a
single request as it retires here rather than after a whole local run. Which
of those a parked worker is, and what it should be told, is
dangling_unwind's to
decide, so a trace behaves the same on this engine as it does locally: a read
past the model's point is an error, and a tracer.iter loop the request
could not supply — bounded and open alike — is cut short with a warning,
keeping the values from the steps that ran. Surfacing differs, since the
client is in another process: an error becomes the request's deferred error
and is raised there.
taps is the engine's tap set when it replays CUDA graphs: a module
location outside it is never reached by a replayed step, and the error
says so rather than reporting the model ran past it. quiet only
unwinds (a registered copy has no request to report to).
NNsightGPUModelRunner
¶
Bases: GPUModelRunner
A vLLM model runner that interleaves interventions with the forward.
capture_model
¶
Record vLLM's CUDA graphs with the interleaver open.
A module hands off only while interleaving, and recording the graphs runs
the forward outside any step — so open the interleaver (with no workers)
for the recording, which is what lets VLLMInterleaver.handle register
each tap's replay into the graph.
execute_model
¶
execute_model(scheduler_output: 'SchedulerOutput', intermediate_tensors: IntermediateTensors | None = None) -> Any
collect_nnsight
¶
collect_nnsight(request_ids: list[str], finished_request_ids: list[str] | None = None, outputs: dict | None = None) -> bytes | None
Return the saved values and any deferred error of the named requests.
Keyed per request rather than merged, so two traces that happen to name a
variable the same don't overwrite each other on the way home. Each entry is
{"error": ..., "sequences": {index: {"saves": {...}, "registered": {...}}}},
one sequence per sampled n (just 0 unless the request asked for
several), plus flat saves and registered: the primary sequence's.
Every rank ran the block and so has workers to wind up, which is why every
rank is called; an early return on the others once left their workers in
place for the life of the engine. But only the first tensor-parallel rank
replies with values, and the rest return None: every rank runs every
layer and its reads are gathered across the group, so each holds the same
values, and pickling them on every rank would only be thrown away.
A registered block's values are kept apart from the trace's own because
they are not the same kind of thing: a name a trace saved on several
requests is one shared container the invokes were writing into, and
merge_shared_saves
reassembles it on that assumption. A registration saving the same name on
every request it runs on looks identical from the outside and is not — it
is one value per request — so merging them together would quietly fold a
thousand separate activations into one.
| PARAMETER | DESCRIPTION |
|---|---|
request_ids
|
Requests to collect saved values from.
TYPE:
|
finished_request_ids
|
Those that are done, whose workers are wound up and forgotten afterwards.
TYPE:
|
outputs
|
Engine request id -> the
TYPE:
|
nnsight_register
¶
Keep a block and run it for every request from now on.
See nnsight.modeling.vllm.registration.
Runs on all ranks, so every rank builds the same per-request copies and
their reads stay in lockstep — which is what a sharded model's gathers
need. name is what requests may address it by (edits=[...]).
nnsight_clear_registered
¶
Stop running a block and drop what it has not handed back.
nnsight_request_count
¶
How many requests' workers this runner still tracks.
A leak gauge: it should return to zero once every request has finished or
been aborted. A number that only grows across requests means workers are
outliving their requests — a finished one is freed in collect_nnsight,
an aborted one when its stream is closed (see the async and serve backends).