What Is a Vector? How LLMs Represent Words as Numbers
A token ID is just a number. To represent meaning, we line up many numbers: a vector. Move the arrows and check adding, subtracting and length with your own hands.
- POINT 1A vector is numbers lined up in a fixed order. How many there are is the dimension. (3, 2) and (2, 3) are different vectors.
- POINT 2Two numbers can be drawn as an arrow, with a direction and a length. The length: square each number, add, take the square root (Pythagoras).
- POINT 3Adding and subtracting work position by position. Adding puts the arrows end to end; subtracting is the arrow from one tip to the other.
TRY 1
Drag the arrows: add and subtract
Drag the tips (the balls) of the cyan arrow a and the yellow arrow b. The answer arrow and the calculation change at once. You can also type the numbers.
1 + 2 = 3Length of the answer (square, add, square root)√(4² + 3²) = √25 ≈ 5
Adding: put the start of b at the tip of a; the answer goes from the start to where you end up. Starting with b gets you to the same place (the faint parallelogram).
TRY 2
Three numbers: three dimensions
Three numbers are three directions: across, up and depth. Change them with the sliders; the arrow is the diagonal of a box. The length works as in 2D.
The length of (2, 3, 6) is 7. With four or more numbers you can't draw it, but the same calculation gives the length.
(2, 3, 6) has a length of exactly 7 (4 + 9 + 36 = 49).
TRY 3
A real model's vectors
In Gemma 3 4B, every token has its own list of 2560 numbers. These are the real numbers, taken from the model file used in this series.
2560 numbers (cyan positive, pink negative)
“cat” is a 2560-dimensional vector. Its length is about 1.02.
There is one such list for each of the 262,144 tokens: 671,088,640 numbers in all (262,144 × 2560). All of them are parameters set by training. How a row is taken out for an ID: episode 20.
By the way, an ID is a filing number. “cat” is 9307; the next one, 9308, is “ perman”, and the one before, 9306, is “ therap”. Close numbers do not mean close meanings.
CHECK
Quick check
Check what you learned with three questions.
Q1(3, 2) and (2, 3) are…
Q2The length of (3, 4) is…
Q3(4, 3) − (1, 2) is…
NEXT #18
Dot Product and Cosine Similarity: How LLMs Measure Meaning
Next: the dot product and cosine similarity, one number that tells how much two arrows point the same way.
The numbers and IDs in TRY 3 come from the embedding table (token_embd.weight) in the Gemma 3 4B (Q4_K_M) model file (GGUF), read with llama.cpp's gguf-py (shown to 4 decimal places). The arrows in TRY 1 and 2 are numbers you choose. Nothing is sent to a server.
FINISH
COMMENTS
Comments & questions
Something unclear, something you want to know more about, or just your thoughts: write it here. We answer quickly, and other learners can reply too.
Loading…