Consúltenos

Understand LLMs from Zero#3

LLM Output Is a Probability Distribution: Next-Token Prediction

An LLM never picks a word. What it puts out is a probability for every token in its vocabulary: a probability distribution. Picking one of them is the job of the dice. Here you can move the distribution and roll the dice from the video with your own hands.

  • POINT 1The output of an LLM is a probability for every token. They add up to 100%.
  • POINT 2The shape of the distribution changes with the sentence: sometimes peaked, sometimes spread out flat.
  • POINT 3One token is picked with dice weighted by the probabilities. That is sampling.

TRY 1

Every token has a probability

The bar chart only shows the top few. Move the slider and see how much of the total you get when you add up more and more tokens from the top. Switch between the two sentences and you will see how different the shape of the distribution can be.

My favorite food is?

Choose a sentence
  • ·pizza46%
  • ·sushi9.8%
  • ·pasta8.1%
  • ·tacos4.8%
  • ·chocolate2.5%
  • This model's vocabulary has 262,144 tokens. Every single one gets a probability, and together they make 100%.
  • For this sentence, the top 2 reach 50%, 25 reach 90%, and 385 reach 99%.
  • The least likely token has a probability with 15 zeros after the decimal point. Even so, it is not zero.

TRY 2

Roll the dice and draw a word

Picking one item from the list is done with dice whose faces are not equally likely: the higher a token's probability, the easier it is to hit. Roll once at a time first, then roll 100 or 1,000 times in one go.

My favorite food is?

Choose a sentence

Press the button and roll the dice. Nobody knows which token comes out until you roll.

  • Tokenprobabilityshare drawn (count)
  • ·pizza
    46%
    — (0)
  • ·sushi
    9.8%
    — (0)
  • ·pasta
    8.1%
    — (0)
  • ·tacos
    4.8%
    — (0)
  • ·chocolate
    2.5%
    — (0)
  • ·ramen
    2.1%
    — (0)
  • ·a
    1.9%
    — (0)
  • ·steak
    1.7%
    — (0)
  • others
    23%
    — (0)

"Others" is all 262,114 tokens below rank 30 together. Each is tiny, but together they add up to something you cannot ignore. These dice are rolled with your browser's random numbers.

TRY 3

Always first place, or the dice?

If the top token is always chosen, the text is the same every time. With dice, one different token at the start changes everything that follows. Press both buttons a few times and compare.

My favorite food is?

Press a button to see the continuation the model really wrote.

    The dice results are 100 real samples from the model, shown in order (random seeds 1 to 100). Watch for the texts that start with a low-probability token.

    CHECK

    Quick check

    Three questions on what this episode showed.

    Q1What does an LLM output?

    Q2One token has a probability of 33%, another 1%. What happens when the dice choose?

    Q3Is the most probable token "the correct answer"?

    Every probability and text on this page is the output of a real language model (Gemma 3 4B). The results were computed in advance, so nothing is sent to a server. Only the dice are rolled, inside your browser.