Parlons-en

Understand LLMs from Zero#20

What Is an Embedding? The Token Embedding Table Explained

What reaches an LLM is a token's ID, a number. Getting a vector of 2560 numbers from it is very simple: pull that ID's row out of a big table. Pull rows out of Gemma 3's real embedding table yourself, and rewrite them.

  • POINT 1The token embedding table has one row per token (262,144 rows) and 2560 numbers in each row. The row number is the token ID, and the row is that token's vector (its embedding).
  • POINT 2Taking an embedding is just pulling out the ID's row. Multiplying a one-hot vector by the table gives exactly the same row (checked with a one-hot 262,144 long: 671,088,640 multiplications, all 2560 numbers identical).
  • POINT 3The table's numbers are parameters set by training: in Gemma 3 4B about 17% of all of them (671,088,640). The same token gets the same row wherever it appears in a sentence.

TRY 1

Pull a row out of the table

The 3D picture is the embedding table: the rows this page holds, in ID order (each “⋮” stands for thousands of other rows). Click a row, or press a button below, and that row is pulled out.

Pulled out “cat” = row 5,866

−0.00210.02220.01790.0243−0.00110.02640.00630.0032… (2,560)

The strip shows the 2560 numbers left to right, one colour each: cyan = plus, pink = minus, dark = 0.

row number (ID)5,866
numbers per row2,560
numbers that are not 02,487
length1.031
length × √2560 ≈ 50.652.2
one row in this file2,100 BQ6_K format (as 4-byte numbers: 10,240 B)

The whole table is 262,144 rows × 2,560 columns = 671,088,640 numbers, about 17.3% of Gemma 3 4B's 3,880,099,328 parameters. Gemma multiplies the row by √2560 before the first layer (checked in the llama.cpp source).

TRY 2

Rewrite the numbers

Rewrite the first 8 numbers of the row you pulled out. Its length and its cosine similarity (episode 18) with the other words are recomputed from all 2560 numbers.

Rewriting “cat” (row 5,866)

length: was 1.031 → now 1.031

Cosine similarity of “cat” with the other rows

dog0.37
car0.21
truck0.08
bird0.21
fish0.14
The0.05
saw0.06
a0.10

Move a number a lot, and the closeness to other words changes.

TRY 3

A sentence through the table

Press tokens to line them up: each is replaced by its ID's row, and the rows stack into a block of numbers. Try the same token twice.

Press tokens to build a sentence (up to 8)

Thecatsawacat

A block of 5 rows × 2,560 columns → into the model's first layer

The818
cat5866
saw5004
a496
cat5866

The same token gives the same row (all 2560 numbers identical). The look-up sees neither the words around it nor its position.

A row you rewrote in TRY 2 is used here as rewritten.

TRY 4

One-hot × table = one row

With a small table (the first 4 numbers of 4 real rows), work out one-hot × table one row at a time. Choose where the 1 goes, then press “Next row”.

Put the 1 at
one-hotno. 1no. 2no. 3no. 4
cat0×0.00000.00000.00000.0000
dog1×−0.0287−0.00690.00570.0355
car0×0.00000.00000.00000.0000
truck0×0.00000.00000.00000.0000
sum=−0.0287−0.00690.00570.0355

The sum is the “dog” row itself. Only the row times 1 is left; the rows times 0 vanished.

We did it at the real size too: a one-hot 262,144 long times the whole 262,144 × 2,560 table (671,088,640 multiplications) gave exactly the “cat” row taken straight out of the table, all 2,560 numbers. That's why the model skips the multiplication and pulls the row directly.

CHECK

Quick check

Three questions on what we learned.

Q1Where in Gemma 3's embedding table is the embedding of “cat” (ID 5866)?

Q2In “The cat saw a cat”, what about the embeddings of the two “cat”s?

Q3One-hot vector (0, 1, 0, 0) times a 4-row table gives…

The rows come from the embedding table (token_embd.weight, Q6_K) of the Gemma 3 4B (Q4_K_M) file (GGUF), read with llama.cpp's gguf-py (stored × 10,000, rounded to integers). Lengths and cosine similarities are computed in your browser. Nothing is sent to a server.

FINISH

Finished the episode?

COMMENTS

Comments & questions

Something unclear, something you want to know more about, or just your thoughts: write it here. We answer quickly, and other learners can reply too.

Loading…