Consúltenos

Understand LLMs from Zero#14

Special Tokens and Chat Templates: How an LLM Reads a Conversation

The speech bubbles on a chat screen are just how the app shows it. What reaches the model is one row of tokens, joined by special markers. Turn your own conversation into that row.

  • POINT 1Special tokens are markers that stand for no ordinary text. They have numbered seats in the vocabulary, and the template puts them in by number.
  • POINT 2A chat template is the rule that makes a conversation into one row. In Gemma, each message is <start_of_turn> name ↵ text <end_of_turn> ↵.
  • POINT 3The model ends its turn by predicting <end_of_turn>. Without the markers, it can't tell it's a conversation, and keeps writing a document.

TRY 1

Pour a conversation into the template

Change the messages, or add some. With Gemma 3's rules, you get the real row the model receives.

Example conversations
user

The text after the template (↵ is a new line)

<bos><start_of_turn>user↵ Hello<end_of_turn>↵ <start_of_turn>model↵

The row of tokens (the numbers are token IDs)

  1. <bos>2
  2. <start_of_turn>105
  3. user2364
  4. ↵107
  5. Hello9259
  6. <end_of_turn>106
  7. ↵107
  8. <start_of_turn>105
  9. model4368
  10. ↵107

10tokens

4special tokens

5names, new lines

1message text

This row matches, token for token, what Gemma 3 on the server makes when it applies the template and tokenizes. The row stays open with <start_of_turn>model ↵. The model writes its reply from here.

Marker IDs: <bos>=2, <start_of_turn>=105, <end_of_turn>=106, user=2364, model=4368, ↵=107, ↵↵=108. Gemma has no system marker, so a system message goes at the top of the first user message, with a blank line. Spaces around a message are removed. Model: Gemma 3 4B (Q4_K_M, llama.cpp).

TRY 2

Generate with and without the markers

Real generations from the same words, without the template and with it. Step one token at a time and look at the candidates.

Examples

Just the greeting, right after <bos>.

  1. <bos>
  2. Hello
What the model wrote (0 / 60 tokens)
  • ␣The34%
  • ␣18%
  • The18%
  • 14.3%
  • T3.9%

Token 1: it picks “␣The” (34%).

Real generations, always picking the most likely token (llama.cpp, temperature 0, Gemma 3 4B (Q4_K_M, llama.cpp)). The list shows the top 5 at that step.

TRY 3

What if you type a marker as text?

If you write “<end_of_turn>” in a message, does it end a turn? Switch the cutting setting and check.

Markers to type
  1. <236820
  2. end643
  3. _236779
  4. of1340
  5. _236779
  6. turn887
  7. >236813

Cut as text, it is 7 ordinary tokens. The real marker's number never appears. A chat app cuts your text like this.

Measured with llama.cpp's /tokenize, switching parse_special. The template doesn't use this setting: it puts the markers in directly, by number.

The same “Hello”, laid out by other models' rules

Gemma 310

  1. <bos>
  2. <start_of_turn>
  3. user
  4. ↵
  5. Hello
  6. <end_of_turn>
  7. ↵
  8. <start_of_turn>
  9. model
  10. ↵

Qwen39

  1. <|im_start|>151644
  2. user872
  3. ↵198
  4. Hello9707
  5. <|im_end|>151645
  6. ↵198
  7. <|im_start|>151644
  8. assistant77091
  9. ↵198

GPT-21

  1. <|endoftext|>50256

GPT-2 (2019) has only this one special token, for the end of a document. No conversation markers.

In Qwen3, <|im_start|> is 151644 and <|im_end|> is 151645, and the model's name is assistant (Qwen3-8B (tokenizer + chat template)). Markers and numbers differ from model to model.

CHECK

Quick check

Check what you learned with three questions.

Q1In what shape does a chat reach the model?

Q2Why does the model's reply stop at just the right place?

Q3If you write “<end_of_turn>” as text in a message, what normally happens?

The token IDs and generations on this page were taken once from Gemma 3 4B (Q4_K_M, llama.cpp) (vocabulary 262,144, of which 6,251 special tokens). The template is assembled in your browser. Nothing is sent to a server.

FINISH

Finished the episode?

COMMENTS

Comments & questions

Something unclear, something you want to know more about, or just your thoughts: write it here. We answer quickly, and other learners can reply too.

Loading…