상담하기

Understand LLMs from Zero#17

What Is a Vector? How LLMs Represent Words as Numbers

A token ID is just a number. To represent meaning, we line up many numbers: a vector. Move the arrows and check adding, subtracting and length with your own hands.

  • POINT 1A vector is numbers lined up in a fixed order. How many there are is the dimension. (3, 2) and (2, 3) are different vectors.
  • POINT 2Two numbers can be drawn as an arrow, with a direction and a length. The length: square each number, add, take the square root (Pythagoras).
  • POINT 3Adding and subtracting work position by position. Adding puts the arrows end to end; subtracting is the arrow from one tip to the other.

TRY 1

Drag the arrows: add and subtract

Drag the tips (the balls) of the cyan arrow a and the yellow arrow b. The answer arrow and the calculation change at once. You can also type the numbers.

Operation
ab
a(3, 1)length 3.16
b(1, 2)length 2.24
a + b(4, 3)length 5
Position by position3 + 1 = 4
1 + 2 = 3Length of the answer (square, add, square root)√(4² + 3²) = √25 ≈ 5

Adding: put the start of b at the tip of a; the answer goes from the start to where you end up. Starting with b gets you to the same place (the faint parallelogram).

TRY 2

Three numbers: three dimensions

Three numbers are three directions: across, up and depth. Change them with the sliders; the arrow is the diagonal of a box. The length works as in 2D.

across2
up3
depth6
Length = square each, add, square root√(2² + 3² + 6²) = √(4 + 9 + 36) = √49 ≈ 7

The length of (2, 3, 6) is 7. With four or more numbers you can't draw it, but the same calculation gives the length.

(2, 3, 6) has a length of exactly 7 (4 + 9 + 36 = 49).

TRY 3

A real model's vectors

In Gemma 3 4B, every token has its own list of 2560 numbers. These are the real numbers, taken from the model file used in this series.

Token

2560 numbers (cyan positive, pink negative)

12/ 2560
−0.00170.02260.01560.0226−0.00350.02000.02000.0000−0.02780.00870.00780.0122…
Length = square all 2560 numbers, add, square root (computed in this page)√( (−0.0017)² + 0.0226² + … ) = √1.03083 ≈ 1.015

“cat” is a 2560-dimensional vector. Its length is about 1.02.

There is one such list for each of the 262,144 tokens: 671,088,640 numbers in all (262,144 × 2560). All of them are parameters set by training. How a row is taken out for an ID: episode 20.

By the way, an ID is a filing number. “cat” is 9307; the next one, 9308, is “ perman”, and the one before, 9306, is “ therap”. Close numbers do not mean close meanings.

CHECK

Quick check

Check what you learned with three questions.

Q1(3, 2) and (2, 3) are…

Q2The length of (3, 4) is…

Q3(4, 3) − (1, 2) is…

The numbers and IDs in TRY 3 come from the embedding table (token_embd.weight) in the Gemma 3 4B (Q4_K_M) model file (GGUF), read with llama.cpp's gguf-py (shown to 4 decimal places). The arrows in TRY 1 and 2 are numbers you choose. Nothing is sent to a server.

FINISH

Finished the episode?

COMMENTS

Comments & questions

Something unclear, something you want to know more about, or just your thoughts: write it here. We answer quickly, and other learners can reply too.

Loading…