Info

Last Execution: 2026-09-02

Package	Version
nnsight	0.8.0
Python	3.12.13
torch	2.13.0+cu126
transformers	5.15.0.dev0

Scan¶

model.scan() runs the forward pass under PyTorch's FakeTensorMode: tensors carry real shapes and dtypes, but no data, no kernels run, and — crucially — the model is never dispatched. That means you can inspect activation shapes or debug interventions on an undispatched (meta) model, before any weights are loaded into memory.

What a scan checks is shapes. Devices and values are outside what it can see, and this page covers that boundary as well as the feature.

Setup¶

We construct a TransformersModel. In nnsight 0.8 the model is built lazily on the meta device — its architecture is known, but no real weights are loaded until the first real forward. model.dispatched tells us whether weights have been loaded yet.

In [1]:

import nnsight
import torch
from nnsight.modeling.transformers import TransformersModel

model = TransformersModel("openai-community/gpt2", device_map="auto")

print(f"Dispatched (weights loaded)? {model.dispatched}")

import nnsight import torch from nnsight.modeling.transformers import TransformersModel model = TransformersModel("openai-community/gpt2", device_map="auto") print(f"Dispatched (weights loaded)? {model.dispatched}")

Dispatched (weights loaded)? False

Getting Shape Information¶

Use model.scan() like model.trace(), but no real computation happens. You can inspect .shape on any module's output to learn its dimensions. Because scan never dispatches, this runs without loading GPT-2's weights.

In [2]:

with model.scan("The Eiffel Tower is in the city of"):
    hidden_dim = nnsight.save(model.transformer.h[0].output.shape[-1])
    seq_len = nnsight.save(model.transformer.h[0].output.shape[1])
    vocab_size = nnsight.save(model.lm_head.output.shape[-1])

print(f"Hidden dim: {hidden_dim}")
print(f"Sequence length: {seq_len}")
print(f"Vocab size: {vocab_size}")

print(f"\nStill undispatched after scan? {not model.dispatched}")

with model.scan("The Eiffel Tower is in the city of"): hidden_dim = nnsight.save(model.transformer.h[0].output.shape[-1]) seq_len = nnsight.save(model.transformer.h[0].output.shape[1]) vocab_size = nnsight.save(model.lm_head.output.shape[-1]) print(f"Hidden dim: {hidden_dim}") print(f"Sequence length: {seq_len}") print(f"Vocab size: {vocab_size}") print(f"\nStill undispatched after scan? {not model.dispatched}")

Hidden dim: 768
Sequence length: 10
Vocab size: 50257

Still undispatched after scan? True

You must save values to access them outside scan

model.scan() is a tracing context just like model.trace(). Values defined inside it are only valid within the block. Use nnsight.save() for non-tensor values like shape integers, or .save() for tensors. Note that saved tensors are FakeTensors — you can read their .shape/.dtype, but they hold no data.

Values Inside Scan Are Fake Tensors¶

Inside a scan, module outputs are FakeTensors. They know their shape and dtype but carry no real data — so read metadata, not values.

In [3]:

with model.scan("The Eiffel Tower is in the city of"):
    hs = model.transformer.h[-1].output.save()

print(f"Type:  {type(hs).__name__}")
print(f"Shape: {tuple(hs.shape)}")
print(f"Dtype: {hs.dtype}")

with model.scan("The Eiffel Tower is in the city of"): hs = model.transformer.h[-1].output.save() print(f"Type: {type(hs).__name__}") print(f"Shape: {tuple(hs.shape)}") print(f"Dtype: {hs.dtype}")

Type:  FakeTensor
Shape: (1, 10, 768)
Dtype: torch.float32

Inspecting Shapes with Print¶

You can print() inside a scan context to inspect shapes interactively — useful for exploring an unfamiliar model. Access modules in forward-pass order within an invoke, just like in model.trace().

In [4]:

with model.scan("The Eiffel Tower is in the city of"):
    print(f"Embedding output: {model.transformer.wte.output.shape}")
    print(f"Layer 0 output:   {model.transformer.h[0].output.shape}")
    print(f"Layer 11 output:  {model.transformer.h[11].output.shape}")
    print(f"LM head output:   {model.lm_head.output.shape}")

with model.scan("The Eiffel Tower is in the city of"): print(f"Embedding output: {model.transformer.wte.output.shape}") print(f"Layer 0 output: {model.transformer.h[0].output.shape}") print(f"Layer 11 output: {model.transformer.h[11].output.shape}") print(f"LM head output: {model.lm_head.output.shape}")

Embedding output: torch.Size([1, 10, 768])
Layer 0 output:   torch.Size([1, 10, 768])

Layer 11 output:  torch.Size([1, 10, 768])
LM head output:   torch.Size([1, 10, 50257])

Debugging Interventions¶

Shape errors surface inside a scan exactly as they would in a real forward, so a broken index or a mis-sized vector is caught before you pay for the real run. On a large model that is before you have even downloaded the weights.

In [5]:

import logging

# Torch dumps its own traceback to stderr before raising; quiet it for this demo.
logging.getLogger("torch._subclasses.fake_tensor").setLevel(logging.CRITICAL)

input_text = "The Eiffel Tower is in the city of"

# Bug: GPT-2's hidden dimension is 768, but this steering vector is size 1024
wrong_vector = torch.randn(1024)

try:
    with model.scan(input_text):
        model.transformer.h[5].output[:, -1, :] += wrong_vector
except RuntimeError as e:
    print(f"Caught error: {e}")

import logging # Torch dumps its own traceback to stderr before raising; quiet it for this demo. logging.getLogger("torch._subclasses.fake_tensor").setLevel(logging.CRITICAL) input_text = "The Eiffel Tower is in the city of" # Bug: GPT-2's hidden dimension is 768, but this steering vector is size 1024 wrong_vector = torch.randn(1024) try: with model.scan(input_text): model.transformer.h[5].output[:, -1, :] += wrong_vector except RuntimeError as e: print(f"Caught error: {e}")

Caught error: Attempting to broadcast a dimension of length 1024 at -1! Mismatching argument at index 1 had torch.Size([1024]); but expected shape should be broadcastable to [1, 768]

In [6]:

# Fixed version — a correctly-sized vector passes the shape check
right_vector = torch.randn(768)

with model.scan(input_text):
    model.transformer.h[5].output[:, -1, :] += right_vector
    print("Intervention shape check passed!")

# Fixed version — a correctly-sized vector passes the shape check right_vector = torch.randn(768) with model.scan(input_text): model.transformer.h[5].output[:, -1, :] += right_vector print("Intervention shape check passed!")

Intervention shape check passed!

What a Scan Cannot Catch¶

right_vector above is on the CPU, and with device_map="auto" on a CUDA machine the model is not. Those same two lines in a real trace would raise. The scan said nothing, because on an undispatched model there is no GPU yet: with no weights loaded every activation lives on meta, and FakeTensorMode is constructed with allow_non_fake_inputs=True so that a real tensor may combine with a fake one at all.

A device mismatch is a common intervention bug, and it is the kind of thing you would most want a scan to catch.

In [7]:

# A second model so the one above stays undispatched.
probe = TransformersModel("openai-community/gpt2", device_map="auto")
cpu_vector = torch.randn(768)                     # on the CPU

with probe.scan(input_text):
    device = nnsight.save(str(probe.transformer.h[5].output.device))
    probe.transformer.h[5].output[:, -1, :] += cpu_vector

print(f"Activation device under scan: {device}")
assert device == "meta"

try:
    with probe.trace(input_text):                 # the same two lines, for real
        probe.transformer.h[5].output[:, -1, :] += cpu_vector
except RuntimeError as error:
    print(f"Under trace: {error}")

# A second model so the one above stays undispatched. probe = TransformersModel("openai-community/gpt2", device_map="auto") cpu_vector = torch.randn(768) # on the CPU with probe.scan(input_text): device = nnsight.save(str(probe.transformer.h[5].output.device)) probe.transformer.h[5].output[:, -1, :] += cpu_vector print(f"Activation device under scan: {device}") assert device == "meta" try: with probe.trace(input_text): # the same two lines, for real probe.transformer.h[5].output[:, -1, :] += cpu_vector except RuntimeError as error: print(f"Under trace: {error}")

Activation device under scan: meta

Under trace: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Scanning a model that has already been dispatched does catch it: the activations then carry the real device, and you get a FakeTensorDeviceMismatchError. But a scan is usually reached for precisely because nothing has been loaded yet.

So build every tensor from the activation you are combining it with, and there is nothing left for a scan to catch:

with model.trace(input_text):
    resid = model.transformer.h[5].output
    steering = torch.randn(resid.shape[-1], device=resid.device, dtype=resid.dtype)
    resid[:, -1, :] += steering

Values are outside a scan's reach too. A fake tensor has no data, so .item() cannot return a number. It returns a symbol, and arithmetic on it keeps composing symbols rather than failing.

In [8]:

with model.scan(input_text):
    total = nnsight.save(model.transformer.h[5].output.sum().item())

print(f"sum().item() ->  {total!r}")
print(f"     total + 1 ->  {total + 1!r}")
assert not isinstance(total, float)

with model.scan(input_text): total = nnsight.save(model.transformer.h[5].output.sum().item()) print(f"sum().item() -> {total!r}") print(f" total + 1 -> {total + 1!r}") assert not isinstance(total, float)

sum().item() ->  zuf0
     total + 1 ->  zuf0 + 1.0

Only the conversion to a real number raises (float(total) → GuardOnDataDependentSymNode), which can be a long way from the line that produced the symbol. Anything numeric belongs in a trace.

Using Scan for Dynamic Dimensions¶

Scan is useful when you need a model's dimensions to construct tensors for interventions — steering vectors, probes, noise. Here we read the hidden dimension under fake tensors (no weights loaded), then run a real trace, which dispatches the model on demand. Note the .to(device): the scan told us the size and nothing about where the tensor has to live.

In [9]:

with model.scan("test"):
    dim = nnsight.save(model.transformer.h[0].output.shape[-1])

print(f"Hidden dimension: {dim}")
print(f"Dispatched after scan? {model.dispatched}")

# Now use the dimension in a real trace. A random direction of the right size is
# still a random direction, so seed it — otherwise this cell prints a different
# answer every time it runs.
torch.manual_seed(0)
steering_vector = torch.randn(dim)

input_text = "The Eiffel Tower is in the city of"

with model.trace(input_text):
    baseline = model.lm_head.output.save()

with model.trace(input_text):
    device = model.transformer.h[5].output.device
    model.transformer.h[5].output[:, -1, :] += steering_vector.to(device)
    logits = model.lm_head.output.save()

print(f"Baseline prediction: {model.tokenizer.decode(baseline[0, -1].argmax(dim=-1))}")
print(f"Steered prediction:  {model.tokenizer.decode(logits[0, -1].argmax(dim=-1))}")
print(f"Dispatched after trace? {model.dispatched}")

with model.scan("test"): dim = nnsight.save(model.transformer.h[0].output.shape[-1]) print(f"Hidden dimension: {dim}") print(f"Dispatched after scan? {model.dispatched}") # Now use the dimension in a real trace. A random direction of the right size is # still a random direction, so seed it — otherwise this cell prints a different # answer every time it runs. torch.manual_seed(0) steering_vector = torch.randn(dim) input_text = "The Eiffel Tower is in the city of" with model.trace(input_text): baseline = model.lm_head.output.save() with model.trace(input_text): device = model.transformer.h[5].output.device model.transformer.h[5].output[:, -1, :] += steering_vector.to(device) logits = model.lm_head.output.save() print(f"Baseline prediction: {model.tokenizer.decode(baseline[0, -1].argmax(dim=-1))}") print(f"Steered prediction: {model.tokenizer.decode(logits[0, -1].argmax(dim=-1))}") print(f"Dispatched after trace? {model.dispatched}")

Hidden dimension: 768
Dispatched after scan? False

Baseline prediction:  Paris
Steered prediction:   London
Dispatched after trace? True

When to use scan

• Exploring unfamiliar models — quickly check shapes at every layer without loading weights or running the full model
• Debugging interventions — catch shape mismatches and index errors before a costly real run
• Dynamic tensor construction — learn hidden dimensions to build correctly-sized steering vectors, probes, or adapters
• Remote execution — validate the shapes in an intervention locally before sending it to NDIF

And what it will not tell you:

• Devices, on an undispatched model — everything is on meta
• Values — .item() gives a symbol; branching on tensor content raises
• Anything a real kernel would decide — an op with no fake/meta implementation raises inside a scan even though it works in a real forward

Gotchas

• Outputs are FakeTensors — read .shape/.dtype/.device, not data.
• Save the shape, not the fake tensor. A FakeTensor saved out of a scan does not become invalid when the block exits: .sum(), .mean() + 1, .cpu(), .tolist() all keep working and keep handing back fake tensors. What raises is float(hs.sum()), .numpy(), and any op mixing it with a real tensor.
• .save() is required just like in model.trace(). Use nnsight.save() for non-tensor values (ints, shapes, lists).
• Access modules in forward-pass order within an invoke (same rule as trace).