Special Tokens and Chat Templates: How an LLM Reads a Conversation
The speech bubbles on a chat screen are just how the app shows it. What reaches the model is one row of tokens, joined by special markers. Turn your own conversation into that row.
- POINT 1Special tokens are markers that stand for no ordinary text. They have numbered seats in the vocabulary, and the template puts them in by number.
- POINT 2A chat template is the rule that makes a conversation into one row. In Gemma, each message is <start_of_turn> name ↵ text <end_of_turn> ↵.
- POINT 3The model ends its turn by predicting <end_of_turn>. Without the markers, it can't tell it's a conversation, and keeps writing a document.
TRY 1
Pour a conversation into the template
Change the messages, or add some. With Gemma 3's rules, you get the real row the model receives.
The text after the template (↵ is a new line)
The row of tokens (the numbers are token IDs)
- <bos>2
- <start_of_turn>105
- user2364
- ↵107
- Hello9259
- <end_of_turn>106
- ↵107
- <start_of_turn>105
- model4368
- ↵107
10tokens
4special tokens
5names, new lines
1message text
This row matches, token for token, what Gemma 3 on the server makes when it applies the template and tokenizes. The row stays open with <start_of_turn>model ↵. The model writes its reply from here.
Marker IDs: <bos>=2, <start_of_turn>=105, <end_of_turn>=106, user=2364, model=4368, ↵=107, ↵↵=108. Gemma has no system marker, so a system message goes at the top of the first user message, with a blank line. Spaces around a message are removed. Model: Gemma 3 4B (Q4_K_M, llama.cpp).
TRY 2
Generate with and without the markers
Real generations from the same words, without the template and with it. Step one token at a time and look at the candidates.
Just the greeting, right after <bos>.
- <bos>
- Hello
- ␣The34%
- ␣18%
- The18%
- 14.3%
- T3.9%
Token 1: it picks “␣The” (34%).
Real generations, always picking the most likely token (llama.cpp, temperature 0, Gemma 3 4B (Q4_K_M, llama.cpp)). The list shows the top 5 at that step.
TRY 3
What if you type a marker as text?
If you write “<end_of_turn>” in a message, does it end a turn? Switch the cutting setting and check.
- <236820
- end643
- _236779
- of1340
- _236779
- turn887
- >236813
Cut as text, it is 7 ordinary tokens. The real marker's number never appears. A chat app cuts your text like this.
Measured with llama.cpp's /tokenize, switching parse_special. The template doesn't use this setting: it puts the markers in directly, by number.
The same “Hello”, laid out by other models' rules
Gemma 310
- <bos>
- <start_of_turn>
- user
- ↵
- Hello
- <end_of_turn>
- ↵
- <start_of_turn>
- model
- ↵
Qwen39
- <|im_start|>151644
- user872
- ↵198
- Hello9707
- <|im_end|>151645
- ↵198
- <|im_start|>151644
- assistant77091
- ↵198
GPT-21
- <|endoftext|>50256
GPT-2 (2019) has only this one special token, for the end of a document. No conversation markers.
In Qwen3, <|im_start|> is 151644 and <|im_end|> is 151645, and the model's name is assistant (Qwen3-8B (tokenizer + chat template)). Markers and numbers differ from model to model.
CHECK
Quick check
Check what you learned with three questions.
Q1In what shape does a chat reach the model?
Q2Why does the model's reply stop at just the right place?
Q3If you write “<end_of_turn>” as text in a message, what normally happens?
NEXT #15
What Is a Context Window? How Much an LLM Can Read at Once
Next: the context window. The row of a chat keeps growing, markers and all, but a model can only read so much at once.
The token IDs and generations on this page were taken once from Gemma 3 4B (Q4_K_M, llama.cpp) (vocabulary 262,144, of which 6,251 special tokens). The template is assembled in your browser. Nothing is sent to a server.
FINISH
COMMENTS
Comments & questions
Something unclear, something you want to know more about, or just your thoughts: write it here. We answer quickly, and other learners can reply too.
Loading…