The Limits of N-gram Language Models: Why LLMs Need Neural Networks
Take the bigram, which looks at one previous word, and widen it to two, three, four words: that is an n-gram model. Here you run a real one, counted from a whole book, and find out where it gets stuck.
- POINT 1An n-gram model looks at the last few words and counts what came next.
- POINT 2Look at more words and the table explodes, and the text becomes a copy of the book.
- POINT 3The table has no answer for sequences it has never seen. That is why we need neural networks.
TRY 1
Change how many words it sees, and let it write
Move the slider to change n. With n = 1 the model looks at nothing and just lines up words, n = 2 is last episode's bigram, n = 5 looks at the four previous words. Each press of the button draws the next word by lot, in proportion to the counts in the table.
Alice was
Counting the book…
- 3-word sequences in this book25,638
- possible 3-word sequences31,794,757,632
- rows with only one candidate78%
The book that was counted: Lewis Carroll, Alice's Adventures in Wonderland. Words and punctuation marks together: 33,941, of 3,168 different kinds. The tables are counted in your browser when you open this page.
TRY 2
Compare the size of the tables
Every extra word multiplies the number of possible sequences by the size of the vocabulary. But the sequences that really occur in the book hardly grow at all. Almost all of the table stays empty, and more and more of the rows that exist have just one candidate.
At n = 5, 96% of the rows have only one candidate for the next word. There is no lot to draw: the model just reads the book back.
| n | in the book | possible | one candidate | rest found |
|---|---|---|---|---|
| n = 1 | 3,168 | 3,168 | — | 93% |
| n = 2 | 15,528 | 10,036,224 | 51% | 60% |
| n = 3 | 25,638 | 31,794,757,632 | 78% | 29% |
| n = 4 | 30,395 | 100,725,792,178,176 | 90% | 13% |
| n = 5 | 32,293 | 319,099,309,620,461,568 | 96% | 6.0% |
number of sequences of 2 tokens (11 digits)68719476736
atoms in the universe, roughly (10 to the power of 80, 81 digits)100000000000000000000000000000000000000000000000000000000000000000000000000000000
Move the slider to the right. At 15 tokens the count passes the number of atoms in the universe.
Every number in the table above was really counted from Lewis Carroll, Alice's Adventures in Wonderland.
TRY 3
Meeting a sequence it has never seen
We look up two phrases in the table: one from the book, and the same phrase with a single word swapped for a similar one. Move the slider and see how few words it takes before a row is found. At the end you can give the same phrase to a real LLM.
the Queen was?
- the Queen wasin the table4 time(s). What followed: to 1 · in 1 · close 1
- Queen wasin the table4 time(s). What followed: to 1 · in 1 · close 1
- wasin the table355 time(s). What followed: a 31 · the 19 · , 17
This phrase is in the book, so even the long sequence is found. Now choose the phrase with the swapped word.
- to25%
- in25%
- close25%
- silent25%
The LLM result is a real probability distribution from Gemma 3 4B (google/gemma-3-4b-it, GGUF Q4_K_M) / llama.cpp. That model holds 3,880,099,328 numbers: huge, but far smaller than a table of every possible sequence.
CHECK
Quick check
Three questions about what you learned today.
Q1What does an n-gram model with n = 3 (a trigram) look at to guess the next word?
Q2Why does the text turn into a copy of the book when the model looks at more words?
Q3What happens when you look up a sequence that never occurs in the book?
NEXT #6
How LLMs Turn Text into Numbers: Token IDs Explained
A neural network is a calculating mechanism, so it can only handle numbers. Next time: how text is turned into numbers, the token IDs.
Every n-gram number on this page was really counted from Lewis Carroll, Alice's Adventures in Wonderland. The LLM probabilities are the output of a real language model (Gemma 3 4B (google/gemma-3-4b-it, GGUF Q4_K_M) / llama.cpp). Everything is calculated in your browser; nothing is sent to a server.