咨询

Understand LLMs from Zero#21

Word Embedding Space: Why Similar Words Sit Close Together

One row of the embedding table is 2560 numbers: one point in a space of 2560 dimensions. Those points are not scattered at random. Near “cat” you find “dog”. Type words into a map made from Gemma 3's real embeddings and see for yourself.

  • POINT 1A row of the embedding table is a point in 2560-dimensional space. Closeness is measured with cosine similarity. “cat” and “dog”: 0.37. The average over all pairs of the 5,000 words is 0.05, so that pair is very close.
  • POINT 2Nobody arranged them. Training to guess what comes next in text pulled words that are used in the same kinds of places, little by little, the same way.
  • POINT 3The 2D map is like a shadow play. Of the 10 truly closest words, only about 27% on average are also among the 10 nearest on the map. Real closeness is measured in 2560 dimensions.

TRY 1

Type a word: see its place on the map and its neighbors

5,000 words are laid out as dots. Type a word (or click a dot): the view flies to it, and lines go out to its 10 truly closest words.

Words to try

“cat” sits here among the 5,000 words (color group 6). Below are its 10 words with the highest cosine similarity.

10.72#102 on the map
20.51#46 on the map
30.37#1 on the map too
40.28#2 on the map too
50.26#1,825 on the map
60.24#42 on the map
70.24#92 on the map
80.23#20 on the map
90.21#11 on the map
100.21#8 on the map too

Green = also a neighbor on the map (within the nearest 10) / red = far away on the map. Click a word to go to it.

The words are 5,000 common words that are a single token of the 262,144-token vocabulary (lower case, with a leading space). Closeness is the cosine similarity of rows of the embedding table (2,560 numbers each). The positions come from t-SNE, a method that rearranges the points in 2 or 3 dimensions so that close words stay as close as possible.

TRY 2

How far can the map be trusted?

For the word chosen in TRY 1, here are its “10 truly closest words” next to “the 10 nearest dots on the map”. How many words are in both?

Chosen word: “cat”

The 10 truly closest words (cosine similarity in 2,560 dimensions)

  1. cats
  2. kitty
  3. dog
  4. pet
  5. kate
  6. puppy
  7. dogs
  8. rabbit
  9. fox
  10. horse

The 10 nearest on the map

  1. dog
  2. pet
  3. rat
  4. bug
  5. animal
  6. bird
  7. bee
  8. horse
  9. bull
  10. bear

3 of the 10 words are in both.

On average over the 5,000 words, only about 27% survive on the 2D map, and about 29% in the 3D cloud (with plain PCA, which just picks axes, about 2%). Squashing 2,560 dimensions into 2 cannot keep all the closeness. The map is for looking at the whole; real closeness is measured in 2,560 dimensions.

TRY 3

Look for islands

Where on the map do words of one kind sit? Press a button and those words light up on the map above (TRY 1).

Light up a group of words

Press a button and that group of words lights up on the map above.

The colors of the dots are 12 groups of similar rows. The first words of each group:

The color groups were made with k-means in the full 2,560 dimensions. They gather words that are used alike, not only by meaning: -ing forms, past forms, people's names. The word lists of the buttons were chosen by us.

CHECK

Check yourself

Three questions on what this episode covered.

Q1In Gemma 3's embeddings, the cosine similarity of “cat” and “dog” is 0.37. What is the average over all pairs of the 5,000 words?

Q2Why do similar words sit close together?

Q3Two words are far apart on the 2D map. Which is right?

Closeness, color groups and positions were computed from the embedding table (token_embd.weight) of the Gemma 3 4B (Q4_K_M) file (GGUF), read with llama.cpp's gguf-py (5,000 words, the 10 closest of each stored). Ranks on the map are computed in your browser. Nothing is sent to a server.

FINISH

Finished the episode?

COMMENTS

Comments & questions

Something unclear, something you want to know more about, or just your thoughts: write it here. We answer quickly, and other learners can reply too.

Loading…