transformers

transformers

HuggingFace models, whatever the task, without knowing the task.

Reading a model means getting an input into it. A prompt has to be tokenized, an image featurized, chat messages templated; a batch of them has to be padded to a common length; and each of those is different per task, per checkpoint, and per release of transformers.

A transformers.pipeline already knows all of it — which preprocessors a task loads, how to turn its inputs into model inputs, and how to collate them — so this module leans on the pipeline rather than re-deriving any of it:

• Loading: pipeline(model=repo_id, ...) infers the preprocessors the task needs; the task's pipeline class says which those are through its _load_* flags. The lazy meta build is the exception — pipeline() can't from_config a model, so the meta model is built here and handed to it.
• Input: each invoke goes through the task's own preprocess (with its own _sanitize_parameters splitting preprocess from forward kwargs), and the per-invoke encodings are padded together by the pipeline's pad_collate_fn.
• Padding: which side to pad is the model's business, not the task's, so it follows TransformersModel._is_causal — decoders left-pad and get mask-derived position_ids; encoders keep right padding.

Three ways in, and the difference matters:

• trace runs one forward. Its input is assembled here, so it accepts what the model accepts: text, token ids, a tensor, or an encoding.
• generate generates through the model and returns token ids. It takes the same inputs a forward does (assembled here) and generates with the checkpoint's own settings, not the task_specific_params a pipeline folds in.
• pipe runs the whole pipeline, which preprocesses and collates its own text — so it takes what that pipeline takes — and returns what the pipeline postprocesses to (decoded text, labels, ...).

Some inputs can't be padded into a batch at all — a raw feature tensor, or a multimodal encoding — so a lone invoke carries them straight to the model, and asking to batch several of them is refused rather than silently mangled.

A chunked task splits one input into several encodings, each forwarded on its own: token-classification past the model's length limit, one entailment pair per candidate label in zero-shot-classification, a long recording's windows in automatic-speech-recognition. Those become rows of the trace's one forward — which is what the pipeline does at a batch_size of its chunk count — so a read inside the block sees one row per chunk, in the order the task yields them. A chunked invoke is the whole batch: the row count is the task's to decide, and the trace counts one row per invoke, so batching it against another invoke is refused rather than served the wrong rows.

WrapperModule

Bases: Module

Identity module: returns its input unchanged.

Lets nnsight expose a value that isn't produced by a real submodule — the value is passed through this module so it is served at the module's .output.

forward

forward(*args: Any, **kwargs: Any) -> Any

Generator

Generator()

Bases: WrapperModule

Passthrough for the generation output.

Generation output is passed through this module so it is readable/editable at model.generator.output inside a trace. Its Streamer submodule receives tokens as they are decoded (HuggingFace's streamer protocol), so model.generator.streamer.output gives per-step token access.

Reading the finished ids through model.generator.output is deprecated — generate returns them, so use tracer.result instead. The module stays for the per-step streamer access, which tracer.result has no equivalent for.

streamer instance-attribute

streamer = Generator.Streamer()

Streamer

Bases: WrapperModule

Receives generated tokens during decoding via put / end.

put

put(value: Any) -> Any

end

end() -> None

GeneratorEnvoy

GeneratorEnvoy(module: Module, path: str = 'model', interleaver: Interleaver | None = None, rename: dict[str, str | list[str]] | None = None, envoys: dict | None = None)

Bases: Envoy

The envoy for Generator, whose .output is deprecated.

model.generator.output is the only served value in nnsight that is deprecated rather than removed, so the warning lives on the envoy of the one module that has it — the rest of the tree keeps the plain Envoy.

output property writable

output: Any

Deprecated: the finished generated ids — read tracer.result.

A plain property wrapping Envoy.output, not an eproperty of its own: the warning has to reach the user before the read parks the worker, and an eproperty's preprocess runs only once the value has been served.

TransformersModel

TransformersModel(repo_id: Any, *args: Any, task: Optional[str] = None, tokenizer: Optional[Any] = None, processor: Optional[Any] = None, image_processor: Optional[Any] = None, feature_extractor: Optional[Any] = None, peft: Optional[str] = None, **kwargs: Any)

Bases: HuggingFaceModel

A model backed by a transformers.pipeline, for any of its tasks.

See the module docstring for what the pipeline is leaned on for. task picks the pipeline (inferred from the checkpoint when unset). There are three ways to run it: trace runs one forward, generate generates through the model and returns token ids, and pipe runs the whole pipeline and returns what it postprocesses to (decoded text, labels, ...).

The pipeline and its preprocessors are exposed as attributes, so the tokenizer that will actually be used is model.tokenizer. Which of them a task loads varies — a text task has a tokenizer and no image_processor, a multimodal one has a processor — so any of them may be None. Passing one in adopts it instead of loading it.

ATTRIBUTE	DESCRIPTION
pipeline	The task's pipeline. Owns the model and its preprocessors. TYPE: Optional['Pipeline']
tokenizer	The tokenizer, for a task that has one. TYPE: Optional['PreTrainedTokenizerBase']
processor	The processor, for a multimodal task. TYPE: Optional['ProcessorMixin']
image_processor	The image processor, for a vision task. TYPE: Optional['BaseImageProcessor']
feature_extractor	The feature extractor, for an audio task. TYPE: Optional['FeatureExtractionMixin']
generator	The module generated ids are passed through. Reading them at model.generator.output is deprecated (use tracer.result); it remains for per-step access at .streamer.output.

task instance-attribute

task = task

pipeline instance-attribute

pipeline: Optional['Pipeline'] = None

tokenizer instance-attribute

tokenizer: Optional['PreTrainedTokenizerBase'] = tokenizer

processor instance-attribute

processor: Optional['ProcessorMixin'] = processor

image_processor instance-attribute

image_processor: Optional['BaseImageProcessor'] = image_processor

feature_extractor instance-attribute

feature_extractor: Optional['FeatureExtractionMixin'] = feature_extractor

peft instance-attribute

peft = peft

generator instance-attribute

generator = GeneratorEnvoy(Generator(), path=f'{self.path}.generator', interleaver=self.interleaver)

trace

trace(*inputs: Any, fn: Any = None, **kwargs: Any)

scan

scan(*inputs: Any, fn: Any = None, **kwargs: Any)

generate

generate(*inputs: Any, **kwargs: Any) -> Any

Generate through the model, returning the generated token ids.

with model.generate(...): traces the generation, so the block's interventions run against every forward the decode loop makes — use tracer.iter to target a particular step. Calling it directly just generates. The output is the whole prompt plus completion as token ids.

Generating goes through the model, not the task's pipeline (see pipe for that): the model takes the same inputs a forward does — text, token ids, a tensor, or an encoding — and generates the way calling it would, with the checkpoint's own settings rather than the task_specific_params a pipeline would fold in. Read the ids off tracer.result; they also pass through generator, whose model.generator.streamer.output gives per-step access (reading the finished ids at model.generator.output is deprecated in favor of tracer.result).

Examples:

>>> model = TransformersModel("openai-community/gpt2", dispatch=True)
>>> with model.generate("The Eiffel Tower is in", max_new_tokens=3) as tracer:
...     ids = tracer.result.save()
>>> print(model.tokenizer.batch_decode(ids))

PARAMETER	DESCRIPTION
*inputs	What to generate from — the same forms trace takes. TYPE: Any DEFAULT: ()
**kwargs	Passed to the model's generate, e.g. max_new_tokens. streamer defaults to this model's (except under beam search, which transformers refuses a streamer for); pass it to override. TYPE: Any DEFAULT: {}

RETURNS	DESCRIPTION
Any	The generated token ids, as a [batch, seq] tensor.

pipe

pipe(*inputs: Any, **kwargs: Any) -> Any

Run the task's pipeline end to end, returning what it postprocesses to.

Where generate goes through the model and returns token ids, this runs the whole pipeline — decoded-text records for text-generation, labels for a classifier, and so on — the pipeline tokenizing and collating its own input. Traced like the others: the block sees every forward the pipeline makes.

Examples:

>>> model = TransformersModel("openai-community/gpt2", dispatch=True)
>>> with model.pipe("The Eiffel Tower is in", max_new_tokens=3) as tracer:
...     out = tracer.result.save()
>>> print(out[0]["generated_text"])

PARAMETER	DESCRIPTION
*inputs	Inputs for the task's pipeline — text, chat messages, images. TYPE: Any DEFAULT: ()
**kwargs	Passed to the pipeline, e.g. max_new_tokens. TYPE: Any DEFAULT: {}

RETURNS	DESCRIPTION
Any	The task pipeline's postprocessed output.

__getstate__

__getstate__() -> dict