Contact

Understand LLMs from Zero#19

One-Hot Vectors and Their Limits: Why LLMs Need Embeddings

Giving words numbers (IDs) brings in a meaningless “closeness” between the numbers. The one-hot vector avoids that. Build your own vocabulary, turn it into one-hot vectors and measure the distances. You will also see why LLMs need embeddings.

  • POINT 1A one-hot vector lines up one box per word of the vocabulary, sets the box of its word to 1 and all the others to 0. With four words, “cat” = (1, 0, 0, 0). The vector is as long as the vocabulary is big.
  • POINT 2Any two different words have dot product 0 (cosine similarity 0) and distance √2 ≈ 1.41. All equally far, so it can't say that “cat” and “dog” are close.
  • POINT 3Gemma's vocabulary has 262,144 tokens, so a one-hot vector is 262,144 long (102.4 times Gemma's word vectors of 2560). That's why we need short embeddings packed with numbers.

TRY 1

Build a vocabulary, make it one-hot

Tap the words below, or type your own, to put them in the vocabulary (up to 8). Every word you add adds a box, and every vector still has a single 1. The 3D picture shows the tips of the first four words' arrows and the distances between them.

4 words → 4-dimensional vectors

box 1box 2box 3box 4
cat1000
dog0100
car0010
truck0001

Add a word

4 words: joining the tips gives a regular tetrahedron (all six edges √2). The tips live in 4D; this is them drawn in 3D.

TRY 2

Measure the distance between two words

Pick two words of the vocabulary: you see the subtraction of their one-hot vectors, the sum of squares, the distance and the dot product. Check that every pair gives the same. Next to it are the values measured with Gemma 3's real vectors.

Picking
Subtraction (the arrow from tip to tip)(1, 0, 0, 0) − (0, 1, 0, 0) = (1, −1, 0, 0)Square and add1 + 1 + 0 + 0 = 2Distance = its square root√2 ≈ 1.41

one-hot

distance1.41
dot product0
cosine similarity0

Gemma's real vectors

distance1.16
cosine similarity0.37

With one-hot, “cat” and “dog” are √2 ≈ 1.41 apart. With Gemma's vectors: 1.16.

Distance from “cat” to the other words of the vocabulary

dog1.41
1.16
car1.41
1.30
truck1.41
1.40
one-hotGemma's real vectors√2

Gemma's vectors all have a length of about 1, so words with little to do with each other come out near √2 ≈ 1.41. The more similar, the closer than that.

TRY 3

At the size of a real vocabulary

Change the size of the vocabulary. Watch the length of a one-hot vector, the number of zeros, its size at 4 bytes a number, and how it compares with Gemma's word vectors (2560 numbers).

vector length21
zeros20
size of one token (4 bytes × length)84 B≈ 84 B
compared with 2560shorter than 2560
pairs at distance √2210
one-hot: 21
Gemma: 2,560

In Gemma's real vector for “cat”, 2,487 of the 2,560 numbers are not 0. In a one-hot vector, just one is not 0.

CHECK

Quick check

Three questions on what we found.

Q1With the vocabulary cat, dog, car, truck, what is the one-hot vector of “car”?

Q2With one-hot, how do the distances cat–dog and cat–car compare?

Q3One-hot vectors for Gemma's vocabulary (262,144 tokens): how many numbers in each?

The Gemma values in TRY 2 were computed from the embedding table (token_embd.weight) of the Gemma 3 4B (Q4_K_M) file (GGUF), read with llama.cpp's gguf-py (the episode 18 measurements). The one-hot values are computed on the spot. Nothing is sent to a server.

FINISH

Finished the episode?

COMMENTS

Comments & questions

Something unclear, something you want to know more about, or just your thoughts: write it here. We answer quickly, and other learners can reply too.

Loading…