Hubungi kami

Understand LLMs from Zero#24

High Dimensions: Why LLM Embeddings Have Hundreds of Dimensions

In Gemma 3 4B, the word “cat” is 2,560 numbers: 2,560 dimensions. A dimension is one number you can move freely. With many of them, things happen that our three-dimensional intuition does not expect: two arrows drawn at random are almost always nearly perpendicular. Raise the number of dimensions and see for yourself.

  • POINT 1A dimension is one number you can move freely. Exactly perpendicular directions (dot product 0): only as many as there are dimensions.
  • POINT 2But in high dimensions, random directions are almost perpendicular to each other. If “almost” is good enough, the number of directions that fit explodes with the dimensions. In 2,560 dimensions we placed 10,000 without a single collision.
  • POINT 3That is why many properties can be stacked without blurring together. One dimension is not one meaning: it is directions that carry meaning. Take dimensions away, and the neighbourhoods break.

TRY 1

Add dimensions and measure the angle

Move the slider and the page draws pairs of arrows in random directions, right now, and measures the angle of each pair. A dot on the globe is the angle of the second arrow when the first one points at the north pole (the longitude means nothing). The green belt is 80° to 100°.

0°45°90°135°180°

Computed beforehand with 100,000 pairs: 17% between 80° and 100°; angles from 0.5° to 179.6°.

In two dimensions every angle is equally likely. In three, there is little room near the pole and a lot around the equator, so angles near 90° are more common. With every added dimension this “equator” gets wider, until almost every dot sits on it. The random numbers are drawn in your browser.

TRY 2

Fit as many “almost perpendicular” directions as you can

Random directions are drawn one at a time. A new one is kept if its angle to every direction already kept is between 80° and 100°; otherwise it is thrown away. Each press draws 2,000 more.

Dimensions

3 dimensions: drawn 0, kept 0, thrown away 0.

Computed beforehand (a million draws, or until 10,000 were kept)

DimensionsKeptThrown away
22999,998
33999,997
44999,996
55999,995
66999,994
88999,992
1010999,990
1514999,986
2017999,983
3026999,974
5041999,959
100119999,881
200670999,330
3003,093996,907
50010,000 *6,538
1,00010,000 *2
2,56010,000 *0

* means the run stopped because 10,000 had been kept. In 2,560 dimensions not a single arrow was thrown away on the way to 10,000. Placed exactly at right angles, as many directions as dimensions always fit. Drawing at random falls a little short of that between about 15 and 50 dimensions, and beyond 100 dimensions it leaves the number of dimensions far behind.

TRY 3

Stack properties and read them back

Each property gets one random direction. We prepare 1,500 directions, pick a few, add them up, and use dot products to tell which ones were added. In few dimensions they blur into the other directions and cannot be told apart.

Dimensions
Stacked

Press “Try” to make the directions, add some of them and take the dot products.

Computed beforehand (10,000 directions; in how many of 1,000 tries all were found)

Dimensions3 stacked10 stacked
300
1000
30430
1009970
3001,000696
1,0001,0001,000
2,5601,0001,000

The bars are the 40 largest dot products. Yellow: a direction that was added. Red: a wrong one among the top. Left of the dashed line: the top places. The more you stack, and the more directions there are to tell apart, the more dimensions it takes.

TRY 4

Fewer dimensions break the neighbourhoods

The real embeddings of Gemma 3 4B (5,000 English words, 2,560 dimensions) are squashed: only the k most important directions are kept. Then we count how many of a word's original 10 nearest words are still among its 10 nearest.

2,560 dimensions: on average over the 5,000 words, 100% of the neighbours remain.

Word

The 10 nearest words of “cat” in 2,560 dimensions. 10 of them are original neighbours.

catcatskittydogpetkatepuppydogsrabbitfoxhorse

Solid frame: an original neighbour. Dashed frame: a word that only came close because of the squashing. The “most important directions” come from principal component analysis (the directions along which the 5,000 words spread out most, in order). Even with this best choice, almost no neighbours survive in two dimensions. That is why the map of episode 21 could not keep them.

TRY 5

Measured on the real Gemma

The experiments above used random numbers. We measured whether the same thing happens in the real embedding table.

The angles of all 12,497,500 pairs of 5,000 words: 87.1° on average; 99.3% between 80° and 100°. Only 0.67% of the pairs are closer than 80°.

Real embeddings: average angle87.1°
Real: share in 80°–100°99.3%
Random, 2,560 dimensions: average90.0°
5,000 rows of the whole table: 80°–100°99.7%

A “capital letter” direction: built from 2,100 pairs of a lower-case and a capitalised word, tested on the other 2,099 pairs, it points the right way for every pair. The largest single dimension carries 7% of this direction. Half of it takes 94 dimensions, nine tenths 861.

To be fair: dimension number 1,335 alone already tells capital from lower case in 99.4% of the cases. Some properties do sit largely in one dimension. And the angle between the “capital letter” direction and a “plural” direction is 84°: almost perpendicular.

Embedding dimensions of some models

ModelDimensions
Gemma 3 270M640
GPT-2768
Gemma 3 1B1,152
Gemma 3 4B2,560
Gemma 3 12B3,840
Gemma 3 27B5,376

The average angle of the real embeddings is a little below 90° because every word carries a small common component. Twice the dimensions means about four times the numbers in the matrices, so the number of dimensions is a trade-off with the cost of computing.

CHECK

Check yourself

Three questions on what this episode showed.

Q1How many directions that are exactly perpendicular to each other fit in a three-dimensional room?

Q2In 2,560 dimensions you draw two arrows at random. What is the angle?

Q3Is each of the 2,560 dimensions in charge of one meaning, like “capital letter” or “animal”?

The random experiments (TRY 1–3) are computed in your browser; nothing is sent to a server. The tables and TRY 4–5 were computed beforehand (the real embeddings were read from Gemma 3 4B (Q4_K_M), gemma-3-4b-it-Q4_K_M.gguf).

FINISH

Finished the episode?

COMMENTS

Comments & questions

Something unclear, something you want to know more about, or just your thoughts: write it here. We answer quickly, and other learners can reply too.

Loading…