Görüşelim

Understand LLMs from Zero#8

The Full LLM Pipeline: Tokenization, Transformer, Generation

Inside the giant function called an LLM, five steps stand in a row: cut the text, turn it into lists of numbers, process them block after block, make probabilities, pick one. On this page you click the steps of the map and look at the real numbers inside each part.

  • POINT 1Inside an LLM there are five steps: tokenization, embedding, Transformer blocks, the output layer, picking.
  • POINT 2A Transformer block gathers with attention and processes with the feed-forward layer. This model has 34 blocks.
  • POINT 3For every token it writes, the model runs the whole map once. The map is also the contents of all 100 episodes.

TRY 1

Click through the map and look inside each part

Inside an LLM, five steps stand in a row. Click a step of the map to see the numbers that a real model (Gemma 3 4B) actually handles in that part.

↩ the token picked in step 5 is added to the sentence and goes back to step 1

1. Tokenizationcut

Cut the way the model cuts it, the sentence “The highest mountain in Japan is” becomes 6 tokens. Then every token is replaced by its fixed number, the token ID.

  • The818
  • highest7310
  • mountain10565
  • in528
  • Japan6056
  • is563

The number above each bar is the token ID. An ID is like a name tag: its size means nothing (all bars have the same height). In reality a start marker <bos> (ID 2) is added in front as well.

Covered in detail in: Part 2 (episodes 9–16): Tokenization

Every number was measured on a real model or read from the model file (3,880,099,328 parameters in all).

TRY 2

One lap of the map for every token

Each press goes through the five steps in order and decides one token. It is added to the sentence, and the next lap begins. Count the laps, and how many blocks have been passed.

  1. 1Tokenization
  2. 2Embedding
  3. 3Transformer blocks
  4. 4Output layer
  5. 5Pick

↩ the token picked in step 5 is added to the sentence and goes back to step 1

The text so farThe highest mountain in Japan is

Laps: 0Blocks passed: 34 × 0 = 0

  • Mount92%
  • Mt1.4%
  • ␣1.3%
  • located1.1%
  • the0.8%

No laps yet. Press the button: it goes from step 1 to step 5, and the top candidate “Mount” (92%) is picked.

These are the real model's results when the top candidate is picked every time (eight laps were measured). While it writes, not a single parameter changes.

TRY 3

The map is the contents of all 100 episodes

Each of the twelve parts of the series belongs somewhere on the map. Click a part to see which step of the map it teaches.

  1. 1Tokenization
  2. 2Embedding
  3. 3Transformer blocks
  4. 4Output layer
  5. 5Pick

↩ the token picked in step 5 is added to the sentence and goes back to step 1

Part 2 Tokenization — Covers “1 Tokenization” on the map in detail (episodes 9–16).

The numbers in circles are the steps of the map. The strip on top is all 100 episodes; the white frame is this one (episode 8).

CHECK

Check yourself

Three questions on what we learned today.

Q1Which list has the steps inside an LLM in the right order?

Q2What does “attention” do inside a Transformer block?

Q3A model with 34 blocks writes an answer of 100 tokens. How many blocks are passed in total?

The tokens, IDs, embedding values, probabilities and continuations were all measured on a real language model (Gemma 3 4B, 3,880,099,328 parameters) or read from its model file. The results were computed in advance, so nothing is sent to a server.