engine¶
engine
¶
Carry saved values back out of the engine.
A worker's saved values live in the worker process, and nothing in vLLM's own output carries them. The engine knows when a request finishes, which is when its worker has nothing left to run, so that is where they are fetched and attached to the output the request produced.
Every finished request is collected for, not only the traced ones: a registered
block (see nnsight.modeling.vllm.registration)
runs on requests nnsight never created, and this is the one place their values
can be handed back on the output they belong to.
NNsightLLMEngine
¶
attach
¶
Put a request's collected values on its output.
saves carries both kinds, since from the caller's side they are simply
what was saved for this request. nnsight_saves keeps the trace's own
apart, because that is what gets pushed back into the trace's variables and a
registered value must not land there — see
collect_nnsight.
A request that asked for several sampled sequences ran the block once per
sequence, so each has values of its own. They go on the completion they
belong to — output.outputs[i].saves, alongside that sequence's text and
token ids — while output.saves stays the primary sequence's, which is all
there is unless n was set.
acollect
async
¶
One finished request's collected values.
The awaitable form, for the engines whose collective_rpc is a coroutine.
output is the finished RequestOutput, which the worker cannot build
and needs in order to serve tracer.result; without one this is simply the
call that winds the request's workers up and frees what they held. Every rank
is called, since each has workers to wind up, but only the first
tensor-parallel rank replies with values; the rest reply None.