High Dimensions: Why LLM Embeddings Have Hundreds of Dimensions
In Gemma 3 4B, the word “cat” is 2,560 numbers: 2,560 dimensions. A dimension is one number you can move freely. With many of them, things happen that our three-dimensional intuition does not expect: two arrows drawn at random are almost always nearly perpendicular. Raise the number of dimensions and see for yourself.
- POINT 1A dimension is one number you can move freely. Exactly perpendicular directions (dot product 0): only as many as there are dimensions.
- POINT 2But in high dimensions, random directions are almost perpendicular to each other. If “almost” is good enough, the number of directions that fit explodes with the dimensions. In 2,560 dimensions we placed 10,000 without a single collision.
- POINT 3That is why many properties can be stacked without blurring together. One dimension is not one meaning: it is directions that carry meaning. Take dimensions away, and the neighbourhoods break.
TRY 1
Add dimensions and measure the angle
Move the slider and the page draws pairs of arrows in random directions, right now, and measures the angle of each pair. A dot on the globe is the angle of the second arrow when the first one points at the north pole (the longitude means nothing). The green belt is 80° to 100°.
Computed beforehand with 100,000 pairs: 17% between 80° and 100°; angles from 0.5° to 179.6°.
In two dimensions every angle is equally likely. In three, there is little room near the pole and a lot around the equator, so angles near 90° are more common. With every added dimension this “equator” gets wider, until almost every dot sits on it. The random numbers are drawn in your browser.
TRY 2
Fit as many “almost perpendicular” directions as you can
Random directions are drawn one at a time. A new one is kept if its angle to every direction already kept is between 80° and 100°; otherwise it is thrown away. Each press draws 2,000 more.
3 dimensions: drawn 0, kept 0, thrown away 0.
Computed beforehand (a million draws, or until 10,000 were kept)
| Dimensions | Kept | Thrown away |
|---|---|---|
| 2 | 2 | 999,998 |
| 3 | 3 | 999,997 |
| 4 | 4 | 999,996 |
| 5 | 5 | 999,995 |
| 6 | 6 | 999,994 |
| 8 | 8 | 999,992 |
| 10 | 10 | 999,990 |
| 15 | 14 | 999,986 |
| 20 | 17 | 999,983 |
| 30 | 26 | 999,974 |
| 50 | 41 | 999,959 |
| 100 | 119 | 999,881 |
| 200 | 670 | 999,330 |
| 300 | 3,093 | 996,907 |
| 500 | 10,000 * | 6,538 |
| 1,000 | 10,000 * | 2 |
| 2,560 | 10,000 * | 0 |
* means the run stopped because 10,000 had been kept. In 2,560 dimensions not a single arrow was thrown away on the way to 10,000. Placed exactly at right angles, as many directions as dimensions always fit. Drawing at random falls a little short of that between about 15 and 50 dimensions, and beyond 100 dimensions it leaves the number of dimensions far behind.
TRY 3
Stack properties and read them back
Each property gets one random direction. We prepare 1,500 directions, pick a few, add them up, and use dot products to tell which ones were added. In few dimensions they blur into the other directions and cannot be told apart.
Press “Try” to make the directions, add some of them and take the dot products.
Computed beforehand (10,000 directions; in how many of 1,000 tries all were found)
| Dimensions | 3 stacked | 10 stacked |
|---|---|---|
| 3 | 0 | 0 |
| 10 | 0 | 0 |
| 30 | 43 | 0 |
| 100 | 997 | 0 |
| 300 | 1,000 | 696 |
| 1,000 | 1,000 | 1,000 |
| 2,560 | 1,000 | 1,000 |
The bars are the 40 largest dot products. Yellow: a direction that was added. Red: a wrong one among the top. Left of the dashed line: the top places. The more you stack, and the more directions there are to tell apart, the more dimensions it takes.
TRY 4
Fewer dimensions break the neighbourhoods
The real embeddings of Gemma 3 4B (5,000 English words, 2,560 dimensions) are squashed: only the k most important directions are kept. Then we count how many of a word's original 10 nearest words are still among its 10 nearest.
2,560 dimensions: on average over the 5,000 words, 100% of the neighbours remain.
The 10 nearest words of “cat” in 2,560 dimensions. 10 of them are original neighbours.
Solid frame: an original neighbour. Dashed frame: a word that only came close because of the squashing. The “most important directions” come from principal component analysis (the directions along which the 5,000 words spread out most, in order). Even with this best choice, almost no neighbours survive in two dimensions. That is why the map of episode 21 could not keep them.
TRY 5
Measured on the real Gemma
The experiments above used random numbers. We measured whether the same thing happens in the real embedding table.
The angles of all 12,497,500 pairs of 5,000 words: 87.1° on average; 99.3% between 80° and 100°. Only 0.67% of the pairs are closer than 80°.
A “capital letter” direction: built from 2,100 pairs of a lower-case and a capitalised word, tested on the other 2,099 pairs, it points the right way for every pair. The largest single dimension carries 7% of this direction. Half of it takes 94 dimensions, nine tenths 861.
To be fair: dimension number 1,335 alone already tells capital from lower case in 99.4% of the cases. Some properties do sit largely in one dimension. And the angle between the “capital letter” direction and a “plural” direction is 84°: almost perpendicular.
Embedding dimensions of some models
| Model | Dimensions |
|---|---|
| Gemma 3 270M | 640 |
| GPT-2 | 768 |
| Gemma 3 1B | 1,152 |
| Gemma 3 4B | 2,560 |
| Gemma 3 12B | 3,840 |
| Gemma 3 27B | 5,376 |
The average angle of the real embeddings is a little below 90° because every word carries a small common component. Twice the dimensions means about four times the numbers in the matrices, so the number of dimensions is a trade-off with the cost of computing.
CHECK
Check yourself
Three questions on what this episode showed.
Q1How many directions that are exactly perpendicular to each other fit in a three-dimensional room?
Q2In 2,560 dimensions you draw two arrows at random. What is the angle?
Q3Is each of the 2,560 dimensions in charge of one meaning, like “capital letter” or “animal”?
NEXT #25
How Does an LLM Know Word Order? The Position Problem
Next time: word order. How does an LLM know the order of the words? In fact, there is a big problem here.
The random experiments (TRY 1–3) are computed in your browser; nothing is sent to a server. The tables and TRY 4–5 were computed beforehand (the real embeddings were read from Gemma 3 4B (Q4_K_M), gemma-3-4b-it-Q4_K_M.gguf).
FINISH
COMMENTS
Comments & questions
Something unclear, something you want to know more about, or just your thoughts: write it here. We answer quickly, and other learners can reply too.
Loading…