Features

Explore nnsight's capabilities in depth through interactive notebooks.

Core Features

• Getting Activations - Access hidden states and activations from any layer
• Setting Activations - Modify activations to study causal effects
• Gradients - Compute gradients with respect to intermediate values
• Multiple Token Generation - Work with autoregressive generation
• Loading a Model - Constructor options, meta-model dispatch, and device placement
• Module Access - Navigate and access model components
• Model Editing - Create persistent model modifications
• Batching - Combine multiple prompts in one forward pass and move values between them
• Empty Invokers - Run interventions without input data
• Cache - Bulk-collect activations from many modules in one pass
• Intermediate Operations - Access intermediate operations inside modules
• Early Stopping - Stop computation early for efficiency
• Skip Execution - Skip parts of model execution
• Scan - Infer shapes without full execution
• Remote Execution - Run on NDIF's remote infrastructure
• vLLM Support - High-performance inference with vLLM (the vLLM section has one page per job)
• Tensor Parallelism - Trace a model sharded across several GPUs with transformers tensor parallelism
• Quantization - Hold a model in 4 or 8 bits: the names, what a trace sees, and what memory, accuracy and speed cost