상담하기

Understand LLMs from Zero#9

What Is Tokenization? Characters vs Words vs Subword Tokens

Before an LLM reads a sentence, it cuts it into tokens. There are three ways to choose the unit: characters, words and subwords. On this page you cut the same sentence in all three ways, compare the counts, and see for yourself what happens when a word table is too small. The subword cuts are those of a real model.

  • POINT 1Cut by characters: the table is small, but the sequence is long, and one character means almost nothing.
  • POINT 2Cut by words: the sequence is short, but the table never stops growing, and a missing word becomes an “unknown word”.
  • POINT 3Subwords keep common words whole and split rare words into pieces. There are no unknown words.

TRY 1

One sentence, cut three ways

Pick a sentence and switch between the three ways of cutting. How does the number of tokens change with characters, words and subwords? The subword cut is the one a real model (Gemma 3 4B) actually uses. You can also type your own sentence and cut it by characters and by words.

Pick a sentence:
  1. T
  2. o
  3. k
  4. e
  5. n
  6. i
  7. z
  8. a
  9. t
  10. i
  11. o
  12. n
  13. ␣
  14. c
  15. u
  16. t
  17. s
  18. ␣
  19. t
  20. e
  21. x
  22. t
  23. ␣
  24. i
  25. n
  26. t
  27. o
  28. ␣
  29. s
  30. m
  31. a
  32. l
  33. l
  34. ␣
  35. p
  36. i
  37. e
  38. c
  39. e
  40. s
  41. .

Cut by characters: 41 tokens. One character is one token. Spaces and punctuation marks count, too.

The words of the prepared sentences are cut the same way as in the video (at spaces and punctuation marks; for Japanese, with a dictionary). Your own sentence is cut into words by your browser's built-in word segmenter, so the result can differ a little. What you type is not sent anywhere.

TRY 2

Change the size of the word table

We made a word table from one book: all its words, the most frequent first. The slider decides how many rows of the table are used, counted from the top. Every word that is not in the table becomes [UNK], an unknown word. Notice that for some sentences the [UNK]s never go away, however large you make the table.

Pick a sentence:
2,900 rows
  1. I#14
  2. watched#1,563
  3. a#8
  4. cat#361
  5. [UNK]video
  6. on#27
  7. my#86
  8. [UNK]smartphone
  9. .#5

Even with every word of “Alice's Adventures in Wonderland” in the table (2,900 rows), 2 words stay [UNK]. They never appear in that book.

Words 2,900Characters 69words read (33,941 in all) →

The number of different characters and words while reading “Alice's Adventures in Wonderland” by Lewis Carroll. The characters settle at 69, but the words keep growing up to 2,900, and even the last tenth of the book brings 166 new words. Of all those words, 1,284 (44%) appear only once in the whole book. The dashed orange line is the table size you chose.

The row number (#) says how frequent the word is in the book: #1 is the most frequent. Punctuation marks count as words, too.

TRY 3

Subwords: whole words, pieces, new words

Pick a word and see how the real model cuts it. A common word is one token. A rare word, a word that was born recently, or a nonsense word is split into pieces that are in the table. The number under each piece is its token ID.

Not in the table:
Common words:
Words that look alike:
New words and a made-up word:
  1. Token#7,211
  2. ization#1,854

“Tokenization” is split into 2 pieces. Every piece is in the table, so it never becomes an “unknown word”.

This model's table has 262,144 rows. 19,225 of them are tokens of a single character, and 236,249 are pieces or words of two or more characters. That is why any word at all can be written: in the end, it can always be split into single characters.

Extra: “how many of this letter?” asked for 20 words (11 of 20 correct)

We asked the model questions of the form “How many times does the letter r appear in the word strawberry?” for twenty words. What the model sees is not letters but token numbers, so it cannot count the spelling directly. That is one reason for the mistakes.

wordlettercorrectmodel's answer
strawberryr33 ✓
bananaa33 ✓
mississippis44 ✓
committeee23 ✗
bookkeepere33 ✓
raspberryr33 ✓
parallell33 ✓
occurrencec33 ✓
assessments43 ✗
blueberryr22 ✓
coffeef21 ✗
lettert22 ✓
giraffef21 ✗
balloonl22 ✓
necessarys23 ✗
embarrassr23 ✗
tomorrowr22 ✓
hippopotamusp32 ✗
refrigeratorr43 ✗
successfulc23 ✗

The cuts do not have to follow meaning or grammar. They are decided by which sequences of letters appear often in a huge amount of text. How that is decided (BPE) is next time's topic.

CHECK

Quick check

Three questions on what you learned today.

Q1What is the weak point of cutting by characters?

Q2What happens to a word that is not in a word table?

Q3Which describes how subwords cut text?

The subword cuts and token IDs, the size of the table and the answers of the letter-counting test all come from a real language model (Gemma 3 4B (Q4_K_M, llama.cpp)). The word table was made by counting the words of “Alice's Adventures in Wonderland” by Lewis Carroll. Everything was computed in advance, so nothing is sent to a server.