language¶

language ¶

LanguageModel ¶

LanguageModel(*args: Any, tokenizer: Optional[PreTrainedTokenizer] = None, automodel: Type[AutoModel] = AutoModelForCausalLM, **kwargs: Any)

Bases: TransformersModel

LanguageModels are NNsight wrappers around transformers language models.

Inputs can be in the form of

Prompt: (str) Prompts: (List[str]) Batched prompts: (List[List[str]]) Tokenized prompt: (Union[List[int], torch.Tensor]) Tokenized prompts: (Union[List[List[int]], torch.Tensor]) Direct input: (Dict[str,Any])

If using a custom model, you also need to provide the tokenizer like LanguageModel(custom_model, tokenizer=tokenizer)

Calls to generate pass arguments downstream to :func:GenerationMixin.generate

ATTRIBUTE	DESCRIPTION
`config`	Huggingface config file loaded from repository or checkpoint. TYPE: `PretrainedConfig`
`tokenizer`	Tokenizer for LMs. TYPE: `PreTrainedTokenizer`
`automodel`	AutoModel type from transformer auto models. TYPE: `Type`
`model`	Meta version of underlying auto model. TYPE: `PreTrainedModel`

tokenizer `instance-attribute` ¶

tokenizer: PreTrainedTokenizer = tokenizer

generator `instance-attribute` ¶

generator: Envoy = Generator()

Generator ¶

Generator()

Bases: WrapperModule

Wrapper module that captures the final generation output.

Contains a :class:Streamer submodule that receives tokens during generation. The generator output can be accessed via model.generator.output inside a trace, though tracer.result is preferred for new code.

streamer `instance-attribute` ¶

streamer: Streamer = Streamer()

Streamer ¶

Bases: WrapperModule

Streamer that receives tokens during generation and passes them through as a module call.

put ¶

put(*args: Any) -> Any

end ¶

end() -> None

generate ¶

generate(*args: Any, **kwargs: Any) -> Union[InterleavingTracer, Any]

__nnsight_generate__ ¶

__nnsight_generate__(*args, **kwargs)

Custom generation entry point used when .generate() is called as a tracing context.

Sets up iteration tracking via max_new_tokens, injects a streamer for token-by-token access, and wraps the final output through the :attr:generator module.

getstate ¶

__getstate__()

setstate ¶

__setstate__(state)