What Is an Embedding? The Token Embedding Table Explained
What reaches an LLM is a token's ID, a number. Getting a vector of 2560 numbers from it is very simple: pull that ID's row out of a big table. Pull rows out of Gemma 3's real embedding table yourself, and rewrite them.
- POINT 1The token embedding table has one row per token (262,144 rows) and 2560 numbers in each row. The row number is the token ID, and the row is that token's vector (its embedding).
- POINT 2Taking an embedding is just pulling out the ID's row. Multiplying a one-hot vector by the table gives exactly the same row (checked with a one-hot 262,144 long: 671,088,640 multiplications, all 2560 numbers identical).
- POINT 3The table's numbers are parameters set by training: in Gemma 3 4B about 17% of all of them (671,088,640). The same token gets the same row wherever it appears in a sentence.
TRY 1
Pull a row out of the table
The 3D picture is the embedding table: the rows this page holds, in ID order (each “⋮” stands for thousands of other rows). Click a row, or press a button below, and that row is pulled out.
Pulled out “cat” = row 5,866
The strip shows the 2560 numbers left to right, one colour each: cyan = plus, pink = minus, dark = 0.
The whole table is 262,144 rows × 2,560 columns = 671,088,640 numbers, about 17.3% of Gemma 3 4B's 3,880,099,328 parameters. Gemma multiplies the row by √2560 before the first layer (checked in the llama.cpp source).
TRY 2
Rewrite the numbers
Rewrite the first 8 numbers of the row you pulled out. Its length and its cosine similarity (episode 18) with the other words are recomputed from all 2560 numbers.
Rewriting “cat” (row 5,866)
Cosine similarity of “cat” with the other rows
Move a number a lot, and the closeness to other words changes.
TRY 3
A sentence through the table
Press tokens to line them up: each is replaced by its ID's row, and the rows stack into a block of numbers. Try the same token twice.
Press tokens to build a sentence (up to 8)
A block of 5 rows × 2,560 columns → into the model's first layer
The same token gives the same row (all 2560 numbers identical). The look-up sees neither the words around it nor its position.
A row you rewrote in TRY 2 is used here as rewritten.
TRY 4
One-hot × table = one row
With a small table (the first 4 numbers of 4 real rows), work out one-hot × table one row at a time. Choose where the 1 goes, then press “Next row”.
| one-hot | no. 1 | no. 2 | no. 3 | no. 4 | ||
|---|---|---|---|---|---|---|
| cat | 0 | × | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| dog | 1 | × | −0.0287 | −0.0069 | 0.0057 | 0.0355 |
| car | 0 | × | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| truck | 0 | × | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| sum | = | −0.0287 | −0.0069 | 0.0057 | 0.0355 |
The sum is the “dog” row itself. Only the row times 1 is left; the rows times 0 vanished.
We did it at the real size too: a one-hot 262,144 long times the whole 262,144 × 2,560 table (671,088,640 multiplications) gave exactly the “cat” row taken straight out of the table, all 2,560 numbers. That's why the model skips the multiplication and pulls the row directly.
CHECK
Quick check
Three questions on what we learned.
Q1Where in Gemma 3's embedding table is the embedding of “cat” (ID 5866)?
Q2In “The cat saw a cat”, what about the embeddings of the two “cat”s?
Q3One-hot vector (0, 1, 0, 0) times a 4-row table gives…
NEXT #21
Word Embedding Space: Why Similar Words Sit Close Together
Next: word embedding space. Why similar words gather close together, drawn as a map.
The rows come from the embedding table (token_embd.weight, Q6_K) of the Gemma 3 4B (Q4_K_M) file (GGUF), read with llama.cpp's gguf-py (stored × 10,000, rounded to integers). Lengths and cosine similarities are computed in your browser. Nothing is sent to a server.
FINISH
COMMENTS
Comments & questions
Something unclear, something you want to know more about, or just your thoughts: write it here. We answer quickly, and other learners can reply too.
Loading…