Parlons-en

Understand LLMs from Zero#13

Why Token Counts Differ by Language: LLM Tokenization Compared

People say Japanese needs more tokens than English. Is that true? On this page you cut the same text in different languages and with different tokenizers, count the tokens, and find out where the difference comes from.

  • POINT 1The same content can take very different numbers of tokens, depending on the language and the tokenizer.
  • POINT 2A vocabulary depends on the text it was counted on: the more a language was counted, the longer its pieces.
  • POINT 3The token count sets the cost, how much fits at once, and how fast text is written.
The video is coming soonOnce it is published, you can watch it here.

TRY 1

Cut the same text in two languages

The same text in two languages, side by side. Pick the languages with the buttons above each text and the tokenizer with the switch: the coloured tokens and their number change. The small grey boxes are pieces of bytes (in hex) that are not whole characters.

Text:

English174 characters · 174 bytes37

  1. Yesterday
  2. ␣I
  3. ␣had
  4. ␣lunch
  5. ␣with
  6. ␣a
  7. ␣friend
  8. ␣at
  9. ␣a
  10. ␣small
  11. ␣restaurant
  12. ␣near
  13. ␣the
  14. ␣station
  15. .
  16. ␣The
  17. ␣food
  18. ␣was
  19. ␣very
  20. ␣good
  21. ,
  22. ␣and
  23. ␣the
  24. ␣prices
  25. ␣were
  26. ␣low
  27. .
  28. ␣We
  29. ␣promised
  30. ␣to
  31. ␣go
  32. ␣there
  33. ␣together
  34. ␣again
  35. ␣next
  36. ␣week
  37. .

Japanese67 characters · 201 bytes38

  1. 昨日
  2. 、
  3. 友
  4. だ
  5. ち
  6. と
  7. 駅
  8. の
  9. 近く
  10. の
  11. 小さな
  12. 店
  13. で
  14. 昼
  15. ごはん
  16. を食べ
  17. ました
  18. 。
  19. 料理
  20. は
  21. とても
  22. おい
  23. しくて
  24. 、
  25. 値段
  26. も
  27. 安
  28. かったです
  29. 。
  30. 来
  31. 週
  32. また
  33. 一緒
  34. に行く
  35. 約束
  36. を
  37. しました
  38. 。

With Gemma 3 (2025), English takes 37 tokens and Japanese takes 38. English needs 0.97 times as many as Japanese.

The tokens were taken once from the real tokenizers (Gemma 3 4B and GPT-2) and stored in this page. The translations are by the production team; every language says the same thing.

TRY 2

Compare 12 languages

The same text in 12 languages, from the fewest tokens to the most. Switch between GPT-2 and Gemma and see how much each language changes.

Text:
  • English37
  • Spanish60
  • French70
  • Croatian77
  • German81
  • Japanese95
  • Chinese97
  • Arabic139
  • Korean159
  • Russian186
  • Hindi240
  • Thai252

With GPT-2 (2019), Thai needs the most: 252 tokens, 6.8 times English (37 tokens).

GPT-2's vocabulary has 50,257 entries, and only 127 contain kana. It was made from English web pages, so other languages break into small pieces or bytes. Gemma's vocabulary has 262,144 entries, and 4,886 contain kana.

TRY 3

Change the mix of the counted text and train a BPE

The 40 sentences used since episode 4 (Japanese, English, Croatian) are put together, and a vocabulary is trained with BPE starting from characters. With the sliders you decide how many times each language is counted, and how many merges are made. The training runs in your browser.

How many times each language is counted (0 = not counted)

210← 339Japanese, 40 sentences (tokens ← characters)

372← 986English, 40 sentences (tokens ← characters)

343← 725Croatian, 40 sentences (tokens ← characters)

30/ 150merges that joined Japanese characters

The 40 sentences now take 210 tokens in Japanese, 372 in English and 343 in Croatian. Of the 150 merges, 30 joined Japanese characters.

From the 40 sentences:
  1. the
  2. cat
  3. wa
  4. k
  5. es
  6. u
  7. p
  8. early
  9. in
  10. the
  11. morning

11 tokens

This is the experiment from the video. With English counted four times, the Japanese sentences take 276 tokens; with Japanese counted four times, 136 (150 merges). On a tie, the pair found first wins. English and Croatian are lower-cased, and pairs are made only inside a word.

CHECK

Quick check

Three questions on what we found out.

Q1Why did Japanese need more than twice as many tokens as English with GPT-2?

Q2You train with a larger share of Japanese in the counted text. What happens?

Q3The model reads the same number of tokens at once. For a language that needs more tokens…

The tokens on this page were taken once from the tokenizers of Gemma 3 4B (Q4_K_M, llama.cpp) and GPT-2 (tiktoken gpt2, 50,257). The BPE training is computed in your browser. Nothing is sent to a server.