Why LLMs Fail at Spelling and Arithmetic: Tokenization Pitfalls
What reaches the model is token numbers, not letters. Look at how words are split, predict which questions the model gets wrong, and compare with its real answers.
- POINT 1What reaches the model is just numbers: token IDs. The letters inside a token are hidden from it.
- POINT 2So letter counts, letter positions and spelling backwards are hard for it. Writing the letters out one by one can help (not always).
- POINT 3Sums with many digits are hard too: it writes the answer one token at a time from the top digit, so it must foresee the carries. Writing the steps can make it right.
TRY 1
Will the model get it right?
These are questions we really asked Gemma 3. Look at how the word is split, predict whether the model gets it right, then open the answer.
1 / 9 · How many “r”?
How many r's are in the word "strawberry"?
How it is split (numbers are IDs)
- strawberry11513710
Each answer was made by always picking the most likely token, and every question was asked twice with the same answer both times. The truth is counted from the word itself.
TRY 2
The letters inside a token are hidden
Pick a word and compare the letters you see with the numbers that reach the model.
- strawberry11513710
10 letters, inside one token
What if it writes the letters out?
The real answers to “write it one letter per line, numbered, then count”. Each written letter becomes its own token.
What we asked
Spell "blueberry" one letter per line, numbering each line, then count the r's.
TRY 3
A long multiplication, digit by digit
How numbers are split depends on the tokenizer. Gemma 3 uses one digit per token. Then open, digit by digit, its real answer to a multiplication asked for the number only.
How numbers are split
48214821396739671912490719124907123456789123456789202620262984229842Asked for the number only
What is 4821 × 3967? Answer with the number only.
Model — Press a digit to see the candidates for it.
Correct
Digit 4: wrote “3” with 86%
- 386%
- 14.8%
- 73.8%
- 52.5%
- 62.0%
With the steps written out
See the steps the model wrote
To calculate 347 × 86, we can use the distributive property:
347 × 86 = 347 × (80 + 6)
= 347 × 80 + 347 × 6
First, let's calculate 347 × 80:
347 × 80 = 347 × 8 × 10
347 × 8 = (300 + 40 + 7) × 8
= 300 × 8 + 40 × 8 + 7 × 8
= 2400 + 320 + 56
= 2400 + 376
= 2776
So, 347 × 80 = 2776 × 10 = 27760
Next, let's calculate 347 × 6:
347 × 6 = (300 + 40 + 7) × 6
= 300 × 6 + 40 × 6 + 7 × 6
= 1800 + 240 + 42
= 1800 + 282
= 2082
Now, we add the two results:
347 × 86 = 27760 + 2082
= 29842
Alternatively, we can multiply directly:
347 × 86
347
× 86
-----
2082 (347 × 6)
27760 (347 × 80)
-----
29842
Final Answer: The final answer is $\boxed{29842}$4821 × 3967, with the steps
To calculate 4821 × 3967, we can use the distributive property:
4821 × 3967 = 4821 × (3000 + 900 + 60 + 7)
= 4821 × 3000 + 4821 × 900 + 4821 × 60 + 4821 × 7
Let's calculate each term:
1. 4821 × 3000 = 4821 × 3 × 1000 = 14463 × 1000 = 14463000
2. 4821 × 900 = 4821 × 9 × 100 = (4821 × 9) × 100
4821 × 9 = (4800 + 21) × 9 = 4800 × 9 + 21 × 9 = 43200 + 189 = 43389
So, 4821 × 900 = 43389 × 100 = 4338900
3. 4821 × 60 = 4821 × 6 × 10 = (4821 × 6) × 10
4821 × 6 = (4800 + 21) × 6 = 4800 × 6 + 21 × 6 = 28800 + 126 = 28926
So, 4821 × 60 = 28926 × 10 = 289260
4. 4821 × 7 = (4000 + 800 + 20 + 1) × 7 = 4000 × 7 + 800 × 7 + 20 × 7 + 1 × 7
= 28000 + 5600 + 140 + 7 = 28000 + 5600 + 147 = 28000 + 5747 = 33747
Now, add the results:
14463000 + 4338900 + 289260 + 33747 = 14463000 + 4338900 + 322999 + 100 = 14463000 + 4661899 = 19124899
Alternatively, we can use the standard multiplication method:
4821
× 3967
-------
33747 (4821 × 7)
289260 (4821 × 60)
2892600 (4821 × 600)
14463000 (4821 × 3000)
-------
19124897
Let's check the calculation:
4821 × 3967 = 4821 × (4000 - 33) = 4821 × 4000 - 4821 × 33
4821 × 4000 = 19284000
4821 × 33 = 4821 × (30 + 3) = 4821 × 30 + 4821 × 3 = 144630 + 14463 = 15Real answers of Gemma 3 4B (Q4_K_M, llama.cpp), with the probabilities while it wrote them. The correct answers are ordinary arithmetic.
CHECK
Quick check
Check what you learned with three questions.
Q1What reaches the model for “strawberry”?
Q2What makes letter-count questions easier for it?
Q3One reason long multiplications go wrong when asked for the number only?
NEXT #17
What Is a Vector? How LLMs Represent Words as Numbers
Part 3 starts next. A token ID is just a number. Next, we write a word's meaning as a list of numbers: a vector.
The answers, tokens and probabilities on this page were taken once from Gemma 3 4B (Q4_K_M, llama.cpp) (always the most likely token; every question asked twice, with the same answer). The other tokenizers are tiktoken's cl100k_base (GPT-4) and gpt2. Nothing is sent to a server.
FINISH
COMMENTS
Comments & questions
Something unclear, something you want to know more about, or just your thoughts: write it here. We answer quickly, and other learners can reply too.
Loading…