Görüşelim

Understand LLMs from Zero#5

The Limits of N-gram Language Models: Why LLMs Need Neural Networks

Take the bigram, which looks at one previous word, and widen it to two, three, four words: that is an n-gram model. Here you run a real one, counted from a whole book, and find out where it gets stuck.

  • POINT 1An n-gram model looks at the last few words and counts what came next.
  • POINT 2Look at more words and the table explodes, and the text becomes a copy of the book.
  • POINT 3The table has no answer for sequences it has never seen. That is why we need neural networks.

TRY 1

Change how many words it sees, and let it write

Move the slider to change n. With n = 1 the model looks at nothing and just lines up words, n = 2 is last episode's bigram, n = 5 looks at the four previous words. Each press of the button draws the next word by lot, in proportion to the counts in the table.

Alice was

Counting the book…

  • 3-word sequences in this book25,638
  • possible 3-word sequences31,794,757,632
  • rows with only one candidate78%

The book that was counted: Lewis Carroll, Alice's Adventures in Wonderland. Words and punctuation marks together: 33,941, of 3,168 different kinds. The tables are counted in your browser when you open this page.

TRY 2

Compare the size of the tables

Every extra word multiplies the number of possible sequences by the size of the vocabulary. But the sequences that really occur in the book hardly grow at all. Almost all of the table stays empty, and more and more of the rows that exist have just one candidate.

What the bars show

At n = 5, 96% of the rows have only one candidate for the next word. There is no lot to draw: the model just reads the book back.

nin the bookpossibleone candidaterest found
n = 13,1683,168—93%
n = 215,52810,036,22451%60%
n = 325,63831,794,757,63278%29%
n = 430,395100,725,792,178,17690%13%
n = 532,293319,099,309,620,461,56896%6.0%

number of sequences of 2 tokens (11 digits)68719476736

atoms in the universe, roughly (10 to the power of 80, 81 digits)100000000000000000000000000000000000000000000000000000000000000000000000000000000

Move the slider to the right. At 15 tokens the count passes the number of atoms in the universe.

Every number in the table above was really counted from Lewis Carroll, Alice's Adventures in Wonderland.

TRY 3

Meeting a sequence it has never seen

We look up two phrases in the table: one from the book, and the same phrase with a single word swapped for a similar one. Move the slider and see how few words it takes before a row is found. At the end you can give the same phrase to a real LLM.

Choose the phrase

the Queen was?

  • the Queen wasin the table4 time(s). What followed: to 1 · in 1 · close 1
  • Queen wasin the table4 time(s). What followed: to 1 · in 1 · close 1
  • wasin the table355 time(s). What followed: a 31 · the 19 · , 17

This phrase is in the book, so even the long sequence is found. Now choose the phrase with the swapped word.

  • to25%
  • in25%
  • close25%
  • silent25%

The LLM result is a real probability distribution from Gemma 3 4B (google/gemma-3-4b-it, GGUF Q4_K_M) / llama.cpp. That model holds 3,880,099,328 numbers: huge, but far smaller than a table of every possible sequence.

CHECK

Quick check

Three questions about what you learned today.

Q1What does an n-gram model with n = 3 (a trigram) look at to guess the next word?

Q2Why does the text turn into a copy of the book when the model looks at more words?

Q3What happens when you look up a sequence that never occurs in the book?

Every n-gram number on this page was really counted from Lewis Carroll, Alice's Adventures in Wonderland. The LLM probabilities are the output of a real language model (Gemma 3 4B (google/gemma-3-4b-it, GGUF Q4_K_M) / llama.cpp). Everything is calculated in your browser; nothing is sent to a server.