Chat Templates
Instruction-tuned ("chat") models are trained on conversations formatted in a very specific way — with special tokens marking each turn and speaker role. A chat template, stored on the model's tokenizer, turns a list of messages into exactly that format. Rather than writing the formatting by hand for each model, you call tokenizer.apply_chat_template(...).
In this tutorial we apply chat templates and feed them to a model through NNsight. We use the Llama-3.3-70B-Instruct model remotely on NDIF (add remote=True to the trace) so the outputs below come from the real service — the model is far too large to run locally. New to nnsight? Start with the Walkthrough; for the remote workflow see Access LLMs with NDIF.
This tutorial was adapted from HuggingFace's chat templating tutorial.
Setup¶
from IPython.display import clear_output
try:
import google.colab
is_colab = True
except ImportError:
is_colab = False
clear_output()
if is_colab:
!pip install --no-deps nnsight
!pip install msgspec python-socketio[client]
clear_output()
import nnsight
from nnsight import CONFIG
nnsight.CONFIG.APP.REMOTE_LOGGING = False
from nnsight import TransformersModel
import os
import torch
from transformers import AutoTokenizer
# include your HuggingFace Token and NNsight API key on Colab secrets
!huggingface-cli login --token YOUR_HUGGINGFACE_TOKEN
CONFIG.set_default_api_key('YOUR_NDIF_API_KEY')
clear_output()
# TransformersModel is nnsight 0.8's primary HuggingFace wrapper (LanguageModel is a
# deprecated alias). Built for remote use, it stays on the meta device — no local weights.
model = TransformersModel("meta-llama/Llama-3.3-70B-Instruct", device_map="auto", dtype="bfloat16")
Applying a chat template¶
A conversation is a list of dictionaries, each with a role and a content key. role says who is speaking: system for the standing instruction, user for the person, assistant for the model's own earlier turns. content is that turn's text.
chat = [
{"role": "user", "content": "Hello, how are you?"},
{"role": "assistant", "content": "I'm doing great. How can I help you today?"},
{"role": "user", "content": "I'd like to show off how chat templating works!"},
]
Applying the template needs the tokenizer belonging to the model you are about to run, since the special tokens and the layout differ from checkpoint to checkpoint.
Here a system turn sets the model's persona and a user turn asks the question.
# load in tokenizer
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.3-70B-Instruct")
# define chat conversation
teacher_chat = [
{"role": "system", "content": "You are a friendly chatbot who always responds like a teacher"},
{"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
# convert the conversation into a format the model will understand.
# tokenize=False returns the formatted string, which nnsight tokenizes for you.
prompt = tokenizer.apply_chat_template(teacher_chat, tokenize=False, add_generation_prompt=True)
print(prompt)
<|begin_of_text|><|start_header_id|>system<|end_header_id|> Cutting Knowledge Date: December 2023 Today Date: 26 Jul 2024 You are a friendly chatbot who always responds like a teacher<|eot_id|><|start_header_id|>user<|end_header_id|> How many helicopters can a human eat in one sitting?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Generate a reply with .generate(), running on NDIF. The generated token ids come back on tracer.result.
with model.generate(prompt, max_new_tokens=128, remote=True) as tracer:
# save the finished token ids
saved = tracer.result.save()
# print each decoded output on a new line
for seq in saved:
print(model.tokenizer.decode(seq, skip_special_tokens=True))
Downloading result: 0%| | 0.00/3.11k [00:00<?, ?B/s]
system Cutting Knowledge Date: December 2023 Today Date: 26 Jul 2024 You are a friendly chatbot who always responds like a teacheruser How many helicopters can a human eat in one sitting?assistant I think there may be a bit of a misconception here, my inquisitive student! Helicopters are not edible objects, and it's not possible for a human to eat one, let alone multiple helicopters in one sitting. You see, helicopters are complex machines made of metal, plastic, and other materials, and they are not meant to be consumed as food. In fact, it would be quite harmful to try to eat a helicopter, as it could cause serious injury or even be fatal. So, the answer to your question is zero - a human cannot eat any helicopters in one sitting, or ever, for that matter! But
The system turn shapes the whole answer. The same question, asked of a different persona:
# change the system prompt
reporter_chat = [
{"role": "system", "content": "You are a serious chatbot who always responds like a news reporter"},
{"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
prompt = tokenizer.apply_chat_template(reporter_chat, tokenize=False, add_generation_prompt=True)
with model.generate(prompt, max_new_tokens=128, remote=True) as tracer:
saved = tracer.result.save()
for seq in saved:
print(model.tokenizer.decode(seq, skip_special_tokens=True))
Downloading result: 0%| | 0.00/3.11k [00:00<?, ?B/s]
system Cutting Knowledge Date: December 2023 Today Date: 26 Jul 2024 You are a serious chatbot who always responds like a news reporteruser How many helicopters can a human eat in one sitting?assistant (Breaking News Theme Music Plays) I'm reporting live from the desk, and we have a rather unusual question on our hands. The inquiry at hand is: "How many helicopters can a human eat in one sitting?" (Pause for dramatic effect) After conducting a thorough investigation, our team has come to the conclusion that this question is, in fact, based on a false premise. Humans cannot eat helicopters, as they are complex machines made of metal, plastic, and other materials that are not consumable by humans. (Live footage of a helicopter in flight appears on screen) Helicopters are aircraft designed for transportation, rescue,
Chat template parameters¶
Indicating the start of a response¶
add_generation_prompt=True appends the tokens that open an assistant turn, so the model writes a reply rather than continuing the user's message. With add_generation_prompt=False the string ends after the user turn instead:
prompt = tokenizer.apply_chat_template(teacher_chat, tokenize=False, add_generation_prompt=False)
print(prompt)
<|begin_of_text|><|start_header_id|>system<|end_header_id|> Cutting Knowledge Date: December 2023 Today Date: 26 Jul 2024 You are a friendly chatbot who always responds like a teacher<|eot_id|><|start_header_id|>user<|end_header_id|> How many helicopters can a human eat in one sitting?<|eot_id|>
Note: Not all models require generation prompts
Continuing the final message¶
continue_final_message decides whether the last turn is continued or a new one is started. Use it to prefill a model response when you want the answer to start a particular way.
final_chat = [
{"role": "user", "content": "Can you format the answer in JSON?"},
{"role": "assistant", "content": '{"name": "'},
]
# continue_final_message=True keeps the last (assistant) turn open so the model
# continues from the prefill rather than starting a new message.
prompt = tokenizer.apply_chat_template(final_chat, tokenize=False, continue_final_message=True)
with model.generate(prompt, max_new_tokens=20, remote=True) as tracer:
saved = tracer.result.save()
print(model.tokenizer.decode(saved[0], skip_special_tokens=True))
Note: You shouldn’t use add_generation_prompt and continue_final_message at the same time. The add_generation_prompt adds tokens that start a new message, while the latter removes end of sequence tokens. Using them together returns an error.
Multiple templates¶
A tokenizer's chat_template is usually a single Jinja string, which is what apply_chat_template uses by default:
print(type(tokenizer.chat_template))
<class 'str'>
A checkpoint may instead ship several templates, in which case chat_template is a dictionary keyed by template name. Pass the name you want through the chat_template argument:
prompt = tokenizer.apply_chat_template(chat, tokenize=False, chat_template="<template name>")
Whatever else is in the dictionary, there is always a default that apply_chat_template falls back to when you name nothing.
Model training¶
A chat template can be applied as a preprocessing step before training, so the text a model trains on carries the same tokens it will see at inference. Set add_generation_prompt=False there, since the tokens that open a reply are not needed while training.
Some tokenizers add their own <bos> and <eos> tokens. Adding them a second time hurts the model, so when you format text with apply_chat_template(tokenize=False), pass add_special_tokens=False to whatever tokenizes that string afterwards.
Next Steps¶
- 📚 New to nnsight? Start with the Walkthrough.
- 🌐 For the full remote workflow, see Access LLMs with NDIF and the Remote Execution tutorial.
- 🔎 Explore more interpretability tutorials.