One-Hot Vectors and Their Limits: Why LLMs Need Embeddings
Giving words numbers (IDs) brings in a meaningless “closeness” between the numbers. The one-hot vector avoids that. Build your own vocabulary, turn it into one-hot vectors and measure the distances. You will also see why LLMs need embeddings.
- POINT 1A one-hot vector lines up one box per word of the vocabulary, sets the box of its word to 1 and all the others to 0. With four words, “cat” = (1, 0, 0, 0). The vector is as long as the vocabulary is big.
- POINT 2Any two different words have dot product 0 (cosine similarity 0) and distance √2 ≈ 1.41. All equally far, so it can't say that “cat” and “dog” are close.
- POINT 3Gemma's vocabulary has 262,144 tokens, so a one-hot vector is 262,144 long (102.4 times Gemma's word vectors of 2560). That's why we need short embeddings packed with numbers.
TRY 1
Build a vocabulary, make it one-hot
Tap the words below, or type your own, to put them in the vocabulary (up to 8). Every word you add adds a box, and every vector still has a single 1. The 3D picture shows the tips of the first four words' arrows and the distances between them.
4 words → 4-dimensional vectors
| box 1 | box 2 | box 3 | box 4 | ||
|---|---|---|---|---|---|
| cat | 1 | 0 | 0 | 0 | |
| dog | 0 | 1 | 0 | 0 | |
| car | 0 | 0 | 1 | 0 | |
| truck | 0 | 0 | 0 | 1 |
Add a word
4 words: joining the tips gives a regular tetrahedron (all six edges √2). The tips live in 4D; this is them drawn in 3D.
TRY 2
Measure the distance between two words
Pick two words of the vocabulary: you see the subtraction of their one-hot vectors, the sum of squares, the distance and the dot product. Check that every pair gives the same. Next to it are the values measured with Gemma 3's real vectors.
one-hot
Gemma's real vectors
With one-hot, “cat” and “dog” are √2 ≈ 1.41 apart. With Gemma's vectors: 1.16.
Distance from “cat” to the other words of the vocabulary
Gemma's vectors all have a length of about 1, so words with little to do with each other come out near √2 ≈ 1.41. The more similar, the closer than that.
TRY 3
At the size of a real vocabulary
Change the size of the vocabulary. Watch the length of a one-hot vector, the number of zeros, its size at 4 bytes a number, and how it compares with Gemma's word vectors (2560 numbers).
In Gemma's real vector for “cat”, 2,487 of the 2,560 numbers are not 0. In a one-hot vector, just one is not 0.
CHECK
Quick check
Three questions on what we found.
Q1With the vocabulary cat, dog, car, truck, what is the one-hot vector of “car”?
Q2With one-hot, how do the distances cat–dog and cat–car compare?
Q3One-hot vectors for Gemma's vocabulary (262,144 tokens): how many numbers in each?
NEXT #20
What Is an Embedding? The Token Embedding Table Explained
Next: the token embedding table. A table with one row of 2560 numbers per token, and how an ID picks out its row.
The Gemma values in TRY 2 were computed from the embedding table (token_embd.weight) of the Gemma 3 4B (Q4_K_M) file (GGUF), read with llama.cpp's gguf-py (the episode 18 measurements). The one-hot values are computed on the spot. Nothing is sent to a server.
FINISH
COMMENTS
Comments & questions
Something unclear, something you want to know more about, or just your thoughts: write it here. We answer quickly, and other learners can reply too.
Loading…