Info
Last Execution: 2026-09-03
| Package | Version |
|---|---|
| nnsight | 0.8.0 |
| Python | 3.12.13 |
| torch | 2.13.0+cu126 |
| transformers | 5.15.0.dev0 |
Empty Invokers¶
An empty invoker is a tracer.invoke() with no input. It contributes no rows to the batch and gets no row scoping, which is what makes it the place for four things:
- reading the whole combined batch at once,
- one edit that lands on every row,
- reading a module an earlier invoker already passed,
- code that has to run after an unbounded
tracer.iter[:]loop.
It sees every input invoker's rows wherever you write it, first or last, because the batch is assembled before any worker starts. And like every invoke it runs as its own worker, so what it resets is its ordering relative to the other invokers. Inside its own block it still reads modules in forward order.
from nnsight.modeling.transformers import TransformersModel
model = TransformersModel("openai-community/gpt2", device_map="auto", dispatch=True)
Batch-Wide Reads¶
Each input invoker's interventions are scoped to its rows of every activation. An empty invoker has no scope, so it sees the full batch — useful for logic that needs every row at once.
with model.trace() as tracer:
with tracer.invoke("The Eiffel Tower is in the city of"):
pass
with tracer.invoke("The Colosseum is in the city of"):
pass
# Empty invoke — operates on both prompts as a single batch
with tracer.invoke():
all_logits = model.lm_head.output.save()
print(f"Combined batch shape: {all_logits.shape}")
print(f"Prompt 1: {model.tokenizer.decode(all_logits[0, -1].argmax(dim=-1))}")
print(f"Prompt 2: {model.tokenizer.decode(all_logits[1, -1].argmax(dim=-1))}")
Combined batch shape: torch.Size([2, 10, 50257]) Prompt 1: Paris Prompt 2: P
Batch-Wide Interventions¶
Because an empty invoker sees the whole batch, a single edit inside one applies to every row. Here one empty invoker zero-ablates layer 5 for both prompts at once, then reads the resulting predictions.
with model.trace() as tracer:
with tracer.invoke("The Eiffel Tower is in the city of"):
pass
with tracer.invoke("The Colosseum is in the city of"):
pass
# Empty invoke — one edit hits every row of the batch
with tracer.invoke():
model.transformer.h[5].output[0][:] = 0
ablated = model.lm_head.output[:, -1].argmax(dim=-1).save()
print(f"After ablating layer 5 (both prompts): {[model.tokenizer.decode(t) for t in ablated]}")
After ablating layer 5 (both prompts): [' and', ' P']
Accessing Modules Out of Order¶
Within a single invoker you must access modules in forward-pass order. An empty invoker is a fresh worker, so it can reach a module that an earlier invoker already passed. Going backwards inside the empty invoker still raises OutOfOrderError; it is the ordering between invokers that resets.
with model.trace() as tracer:
with tracer.invoke("The Eiffel Tower is in the city of"):
# Must be in order: layer 0 before the last layer
early_hs = model.transformer.h[0].output[0].save()
late_hs = model.transformer.h[-1].output[0].save()
# Empty invoke — new worker, can go back to a middle layer
with tracer.invoke():
mid_hs = model.transformer.h[5].output[0].save()
print(f"Layer 0: {early_hs.shape}")
print(f"Layer 5: {mid_hs.shape}")
print(f"Layer 11: {late_hs.shape}")
Layer 0: torch.Size([10, 768]) Layer 5: torch.Size([10, 768]) Layer 11: torch.Size([10, 768])
Each empty invoker is an independent worker, so you can chain several to touch the same module at different points:
with model.trace() as tracer:
with tracer.invoke("The Eiffel Tower is in the city of"):
pass
with tracer.invoke():
hs_layer0 = model.transformer.h[0].output[0].save()
with tracer.invoke():
hs_layer5 = model.transformer.h[5].output[0].save()
with tracer.invoke():
hs_layer11 = model.transformer.h[11].output[0].save()
print(f"Layer 0: {hs_layer0.shape}")
print(f"Layer 5: {hs_layer5.shape}")
print(f"Layer 11: {hs_layer11.shape}")
Layer 0: torch.Size([10, 768]) Layer 5: torch.Size([10, 768]) Layer 11: torch.Size([10, 768])
Running Code After Unbounded Iteration¶
An unbounded tracer.iter[:] loop runs until generation stops, so any code written after the loop in the same invoker never runs. Putting that code in a separate empty invoker lets it run — a separate worker isn't tangled up in the loop's unwinding.
model.generate(...) returns the generated token ids on tracer.result; the empty invoker is a natural place to capture them once the loop is done. (The warning below is expected: an open loop ends by asking for a step the run does not make.)
import nnsight
with model.generate(max_new_tokens=3, do_sample=False) as tracer:
with tracer.invoke("The Eiffel Tower is in the city of"):
tokens = nnsight.save([])
for step in tracer.iter[:]:
tokens.append(model.lm_head.output[0, -1].argmax(dim=-1))
# Code here would NEVER run — the loop unwinds past it.
# Empty invoker — a separate worker, so this runs after generation completes
with tracer.invoke():
result_ids = tracer.result.save()
for i, t in enumerate(tokens):
print(f"Step {i}: {model.tokenizer.decode(t)}")
print(f"Generated: {model.tokenizer.decode(result_ids[0])}")
Step 0: Paris Step 1: , Step 2: and Generated: The Eiffel Tower is in the city of Paris, and
/home/localjadenfk/wd/nnsight/src/nnsight/intervention/interleaver.py:859: UserWarning: 'model.lm_head.output.i3' was never reached: the loop asked for a step the run did not make, so it was cut short — values saved inside the loop are kept, and the statements after it did not run. An open `tracer.iter[:]` / `tracer.all()` loop ends this way by design. To hold a generation to a bounded loop's count, pass `min_new_tokens=` on transformers or `min_tokens=` / `ignore_eos=True` on vLLM; put what follows the loop in a separate `tracer.invoke()`. warnings.warn(expected)
Carrying a Value Into an Empty Invoker¶
Invokers of one trace share the scope they were written in, so a name an input invoker binds is readable in a later empty invoker. Whether it is bound yet depends on where each worker has parked: a worker runs until it asks for a value the model has not produced, and only resumes once the model reaches that location. A name is readable once the reader has parked past the location where it was bound.
The empty invoker below reads acts before it touches any module, so it has parked nowhere. A barrier gives it the ordering instead, exactly as for two input invokers.
with model.trace() as tracer:
barrier = tracer.barrier(2)
with tracer.invoke("The Eiffel Tower is in the city of"):
acts = model.transformer.h[5].output[0].detach()
barrier() # signal: acts is bound
with tracer.invoke(): # whole batch
barrier() # wait for acts
logits = model.lm_head.output.save() # a later module — safe to read here
captured_norm = acts.norm().save()
print(f"Whole-batch logits: {logits.shape}")
print(f"Norm of captured layer-5 activation: {float(captured_norm):.2f}")
Whole-batch logits: torch.Size([1, 10, 50257]) Norm of captured layer-5 activation: 3000.30
What an empty invoker needs
It contributes no rows, so a trace needs at least one input invoker beside it, or a direct input to trace()/generate(). A trace whose only invoke is an empty one has nothing to run and fails inside tokenization.
It never calls the model's batching methods, which is why it works on a base NNsight model where two input invokers would raise NotImplementedError.