상담하기

Understand LLMs from Zero#12

Tokenizer Encode and Decode: From Text to Token IDs and Back

In the last episodes we counted text and built merge rules and a vocabulary. Now we use them. The sentence you type is cut into tokens on the spot, turned into token IDs, and turned back into the original sentence.

  • POINT 1Encoding: split into letters, apply the merge rules in the order they were learned, and look the pieces up in the vocabulary table. It does not take the longest piece first.
  • POINT 2Decoding: look each ID up, take out its piece, and join. No rules needed.
  • POINT 3A real tokenizer splits a character that is not in its table into bytes. So any text always comes back exactly.

TRY 1

Tokens as you type

The small tokenizer from the video runs inside this page. Type a sentence: the rules are applied on the spot, and it is cut into coloured tokens. The small number under each piece is its token ID.

Examples
  1. the22
  2. f19
  3. a5
  4. the22
  5. r11
  6. likes29
  7. the22
  8. cat26

20letters

8tokens

0unknown (ID 0)

The row of token IDs22, 19, 5, 22, 11, 29, 22, 26
Decodedthe father likes the cat

20 letters became 8 tokens, and decoding gives back exactly the same text.

Inside this tokenizer (you can retrain it)

Current tokenizer: 9 merge rules, 30 vocabulary rows (ID 0 + 20 letters + the pieces made by the rules).

  1. 1t + h → th4× in training
  2. 2th + e → the4× in training
  3. 3a + t → at3× in training
  4. 4k + e → ke3× in training
  5. 5ke + s → kes3× in training
  6. 6c + at → cat2× in training
  7. 7i + n → in2× in training
  8. 8l + i → li2× in training
  9. 9li + kes → likes2× in training

Training works as last time: merge the most frequent pair of neighbours, again and again, until no pair appears twice (ties: the pair seen first). For English, pairs are counted inside words only; capitals and punctuation are removed.

This tokenizer only knows the letters of its training text. Try a letter it has never seen.

TRY 2

Apply the rules one at a time, in order

A slow look at encoding. After splitting into letters, the rules are applied from rule 1 onwards, one at a time. If a rule finds nothing, we simply move on.

Examples
  1. t
  2. h
  3. e
  4. f
  5. a
  6. t
  7. h
  8. e
  9. r
  10. l
  11. i
  12. k
  13. e
  14. s
  15. t
  16. h
  17. e
  18. c
  19. a
  20. t

Split into 20 letters. Now the 9 rules are applied in order.

  1. 1t + h → th
  2. 2th + e → the
  3. 3a + t → at
  4. 4k + e → ke
  5. 5ke + s → kes
  6. 6c + at → cat
  7. 7i + n → in
  8. 8l + i → li
  9. 9li + kes → likes

Now compare with the other idea on the same table: always take the longest piece from the left.

Rules in order8

  1. the
  2. f
  3. a
  4. the
  5. r
  6. likes
  7. the
  8. cat

Longest from the left9

  1. the
  2. f
  3. at
  4. h
  5. e
  6. r
  7. likes
  8. the
  9. cat

Same table, different cut. The model was trained on tokens cut by applying the rules in order, so encoding must follow the same procedure.

The order of the rules is exactly the order in which training merged the pairs. Change the order, and the same sentence can give different tokens.

TRY 3

The real Gemma tokenizer

Now the same thing with the tokenizer of the language model used in this series. Among the neighbours it could join, Gemma first joins the piece with the smallest number (token ID). The earlier a piece was made, the smaller its number, so the order of the numbers is the order of the rules.

Put into the real model (Gemma 3 4B (Q4_K_M, llama.cpp)), this gives 7 token IDs.

  1. Token7211
  2. izers19804
  3. ▁encode41193
  4. ▁and532
  5. ▁decode39961
  6. ▁text1816
  7. .236761

Watch “Tokenizers” being joined, one merge at a time.

  1. T
  2. o
  3. k
  4. e
  5. n
  6. i
  7. z
  8. e
  9. r
  10. s
  • To 2021
  • ok 680
  • ke 1725
  • en 501
  • ni 3598
  • iz 685
  • ze 1934
  • er 497
  • rs 5532

Merge 1: 9 neighbouring pairs could be joined. The one with the smallest number, “er” (497), is joined.

It does not take the longest piece from the left. It joins in order of the numbers (the learned order), so the result can differ.

The space mark: Gemma keeps the space as part of a piece (the ▁ mark). Decoding turns the mark back into a space before joining.

  1. encode18016
  2. ▁decode39961

“unbelievably”4

  1. un602
  2. bel8145
  3. iev20873
  4. ably2579

“▁unbelievably”1

  1. ▁unbelievably170977

Whether a space comes in front makes it a different token.

  1. F0<0xF0>ID 478
  2. 9F<0x9F>ID 397
  3. A6<0xA6>ID 404
  4. A9<0xA9>ID 407
Decoded (4 of 4 bytes)🦩

All 4 bytes are there: “🦩” is back.

Gemma's table has a seat for every one of the 256 byte values (token IDs 238 to 493). A character that is not in the table is split into these byte tokens. The video's “🦩” became 478, 397, 404, 407 in Gemma. The numbers here are the byte tokens used when a character is not in the table (common characters usually have a token of their own).

Encode, then decode (checked on the real model; the number on the left is the token count)

  • 7Tokenizers encode and decode text.✓
  • 12I saw a 🦩 at the zoo.✓
  • 11It costs $42.50, right?✓
  • 7unbelievably untokenizable✓
  • 7Mačka means cat in Croatian.✓

With numbers, symbols and emoji mixed in, every sentence came back without a single character changed. Always coming back exactly is a key property of a tokenizer.

CHECK

Quick check

Three questions on what we learned this time.

Q1When a finished tokenizer encodes a new sentence, how are the merge rules used?

Q2What does decoding (IDs back to text) need?

Q3What happens to a character that is not in Gemma's table (a rare character or emoji)?

Training, encoding and decoding with the small tokenizer are computed by your browser on the spot. The real tokens and token IDs were taken in advance from the tokenizer of a language model (Gemma 3 4B (Q4_K_M, llama.cpp), vocabulary of 262,144 rows).

FINISH

Finished the episode?

COMMENTS

Comments & questions

Something unclear, something you want to know more about, or just your thoughts: write it here. We answer quickly, and other learners can reply too.

Loading…