Skip to content

Release

  • in Release
  • 17 min read

NNsight 0.8

We spent the summer rebuilding NNsight from the inside, and it's finally ready. v0.8 is the biggest release the library has had: the execution engine is rewritten from scratch, and a lot of what was hard or impossible in v0.7 falls out of that. The way you use NNsight hasn't changed at all. You still open a with model.trace(...) block and read, edit, and .save() a model's internals as ordinary Python.

The parts we're most excited about:

  • One class for everything in transformers. TransformersModel is backed by a transformers.pipeline, so any task the pipeline factory can build is now traceable with the same code you write for GPT-2. Whisper and the audio models, LLaVA and Qwen3-VL, BERT and the encoder tasks, classifiers, and PEFT adapters applied at load. Interpretability on speech and vision models stops being a porting exercise.

  • Speed was a big focus. Your code and the forward pass now interleave as greenlets rather than OS threads. NNsight's own overhead is cheaper and, more to the point, stays flat as you touch more of the model: per read it is 3x flatter than v0.7, per batched prompt 6x. Caching every layer, sweeping every head, and running big batches stop being the slow path.

  • NNsight 🤝 vLLM is now the fastest and most complete way to do interpretability at scale. Interventions run inside the engine worker and survive CUDA graph replay, so you keep 91% to 96% of vanilla vLLM's throughput while tracing it. You can install a block on the engine itself and serve it over HTTP to clients that have no GPU of their own.

  • And plenty more, like tensor parallelism for models too big for one card, quantization by naming it where you would name a dtype, and .source, which opens up the operations inside a forward pass that never had a module to attach to.

This is a pre-release. You can install it and use it locally today. The full release lands next month, when v0.8 goes live on NDIF and remote execution comes with it.

  • in Release
  • 13 min read

Introducing NNsight 0.6

By Jaden Fiotto-Kaufman

NNsight is releasing its sixth major version, focused on addressing user feedback about common hurdles with the library.

Wait, What is NNsight Again?

If you are a new user or you haven't used NNsight in a while, here's a small refresher! NNsight is a Python library for interpreting and intervening on the internals of PyTorch models. You wrap a model, open a tracing context, and read or write activations at any layer. While the tool supports any Pytorch model, we also provide first-class support for popular architectures such as 🤗 transformers and diffusers.

from nnsight import LanguageModel

model = LanguageModel("openai-community/gpt2", device_map="auto", dispatch=True)

with model.trace("The Eiffel Tower is in"):
    # Read the hidden states at layer 5
    hidden = model.transformer.h[5].output[0].save()

    # Zero out the MLP output at layer 0
    model.transformer.h[0].mlp.output[:] = 0

Under the hood, NNsight uses deferred execution. When you enter with model.trace(...), your code AST is extracted, compiled into a function, and run in a worker thread. When that thread accesses .output, it waits until the model's forward pass reaches the selected module, extracting the desired output tensor through a PyTorch hook. This means your intervention code is fully aligned with the forward pass—no proxies, no fake tensors. You're working with real PyTorch values!