Contact

Understand LLMs from Zero#4

Bigram Language Model: The Simplest Language Model, Built by Counting

An LLM is a machine that gives the probability of the next word. Those probabilities can be made by counting alone: count each pair of neighbouring words into a table and divide by the total. That is already a language model that writes text. Try it with your own writing.

  • POINT 1A bigram is a pair of neighbouring words. Counting “what came next” gives a table.
  • POINT 2Divide each count by the total and you get the probability of the next word.
  • POINT 3Look up the table, pick, append: that writes text — but it only ever sees the one word before.

TRY 1

Count four sentences into a table

We count the same four sentences as in the video, one sentence at a time. On the left of the table is the previous word; on the right, the words that came next and how often. Watch how the counts and the probabilities change with every sentence you add. Press a word in the table to see its row in 3D.

  1. startcatseatthefish.
  2. startdogseatthemeat.
  3. startcatsdrinkthewater.
  4. startbirdswatchthefish.

1 sentence(s) counted. “the” appeared 1 time(s); below is what came after it.

  • fish×1
Previous wordWhat came next, and how often
cats×1
eat×1
the×1
fish×1
.×1

“start” is a mark for the beginning of a sentence. It sits in front of every sentence so that the first words can be counted too.

TRY 2

Build a bigram table from your own text

Now try your own text. Put any text into the box (it starts with the same 40 sentences as the video) and press the button: the page counts it on the spot and builds the table. Your text is not sent anywhere.

  • 40sentences
  • 59different words
  • 3,481cells in the table (59 × 59)
  • 135cells with a count (3.9%)

“the” appeared 63 time(s) and was followed by 16 different word(s). The most frequent is “cat” (19%).

  • cat19%
  • dog14%
  • child13%
  • bird9.5%
  • garden7.9%
  • morning6.3%
Previous wordWhat came next, and how often
the×28mother×4father×3there×3at×1in×1
cat×12dog×9child×8bird×6garden×5morning×4house×3…
the×18
watches×3likes×2waits×2wakes×1drinks×1sleeps×1.×1…
likes×2in×2plays×1goes×1drinks×1sleeps×1eats×1
.×2plays×1goes×1likes×1sleeps×1gives×1sings×1
sings×2flies×1from×1in×1eats×1
.×4at×2
big×3small×2cat×1

The table has “different words × different words” cells, but only a small part of them ever gets a count. Words are lower-cased, and the full stop counts as a word.

TRY 3

Write text from the table

Now we write text with the table you built above. Only the row of the last word is looked at. Roll the dice and the next word is picked with a probability proportional to its count. When a full stop (the end of a sentence) comes up, we are done.

?

We are looking at the “start” row. There are 6 candidates. Roll the dice to decide the first word.

  • the70%
  • mother10%
  • father7.5%
  • there7.5%
  • at2.5%

Sentences with a strange meaning appear because only the one word before is seen. Make the text longer or rewrite it, and see how the sentences change.

CHECK

Quick quiz

Three questions to check what you learned.

Q1What is a bigram?

Q2“the” appears 4 times, and 2 of those times it is followed by “fish”. In the bigram model, what is the probability that “fish” comes after “the”?

Q3What is the main reason a bigram model writes sentences with a strange meaning?

Every count and probability on this page is really counted, in your browser, from the text on the page. The text is not sent to a server.