server

server

A local HTTP server that runs nnsight traces on a vLLM engine.

The server holds one dispatched async VLLM. A client with only a meta model (no GPU) writes a trace as usual and sends it here with model.trace(..., serve=url); the server deserializes it, builds and submits the request to its engine, collects the saved values, and returns them. The trace is authored on the client and run on the server — the same serialized-trace format the remote (NDIF) path uses, so the client needs no weights.

The request handler builds its workers synchronously (deserialize → prepare → attach), all before the first await, so concurrent requests never interleave while sharing the model's one interleaver.

logger module-attribute

logger = logging.getLogger('nnsight.serve')

app module-attribute

app = FastAPI(title='nnsight-vllm-serve')

set_model

set_model(model: 'VLLM') -> None

Register the dispatched engine the server runs traces on.

health async

health() -> dict

register async

register(registration_id: str, request: Request, name: Optional[str] = None) -> dict

Install a block on this server's engine, to run for every request it handles.

Body: the serialized block model.edit(serve=url) prepared on the client. The client has no engine to collective_rpc into, so the install comes over HTTP and this hands it to every rank — the same call the in-process form makes. What the block saves rides the output of the request it ran on, so a serve client reads it through tracer.result. ?name= is the name requests may address the edit by (trace(..., edits=[name])); without it the edit runs for every request.

clear_registration async

clear_registration(registration_id: str) -> dict

Stop running an installed block. Unknown ids are a no-op, so clearing twice is safe.

generate async

generate(request: Request) -> Response

Run a serialized trace and return its saved values.

Body: RequestModel.serialize(tracer) bytes. Response: torch.save of {"saves": {name: value}, "error": <deferred error or None>} — saved values only, so read generated tokens by saving tracer.result, which is served here like anywhere else. A build error (a corrupt/incompatible trace) or a runtime intervention error both come back as the deferred error so the client re-raises the real exception type with its traceback, rather than an opaque HTTP error.