Getting Values#

Hidden states are exposed by accessing the desired module and calling its .input or .output attributes.

Once accessed, you call .save() on it so it’s value is populated and not deleted after.

[1]:

from nnsight import LanguageModel

model = LanguageModel('openai-community/gpt2', device_map='cuda')

with model.trace("The Eiffel Tower is in the city of") as tracer:

    hidden_states = model.transformer.h[-1].output[0].save()

You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.

After exiting the tracing context, the .value attribute of the hidden_states object will be populated.

[2]:

print(hidden_states)

tensor([[[ 0.0505, -0.1728, -0.1690,  ..., -1.0096,  0.1280, -1.0687],
         [ 8.7494,  2.9057,  5.3024,  ..., -8.0418,  1.2964, -2.8677],
         [ 0.2960,  4.6686, -3.6642,  ...,  0.2391, -2.6064,  3.2263],
         ...,
         [ 2.1537,  6.8917,  3.8651,  ...,  0.0588, -1.9866,  5.9188],
         [-0.4460,  7.4285, -9.3065,  ...,  2.0528, -2.7946,  0.5556],
         [ 6.6286,  1.7258,  4.7969,  ...,  7.6714,  3.0683,  2.0481]]],
       device='cuda:0', grad_fn=<AddBackward0>)