Связаться

Understand LLMs from Zero#22

Word Vector Arithmetic: King − Man + Woman = Queen (word2vec)

A word is 2560 numbers. And numbers can be subtracted and added. If we compute “king − man + woman”, do we really get “queen”? Write your own formulas and check them against the real numbers of Gemma 3's embedding table.

  • POINT 1Word points can be subtracted and added. The difference of two points is an arrow. We add the arrow “woman − man” to “king” and look for the word nearest to the point we land on.
  • POINT 2The nearest word is still “king”, the word we put in (0.68). With the input words left out, first place is “queen” (0.52). Next to “king” alone, “kings” was nearer than “queen”; the arithmetic moves “queen” past it.
  • POINT 3It is not exact. The computed point and “queen” are far apart, and the two arrows are 77 degrees apart. The arrows line up only slightly. With opposites and numbers, it does not work.

TRY 1

Write your own formula and check the answer

Join words with “−” and “+” to write a formula (for example: king − man + woman). It is computed on the spot, and the words nearest to the computed point are listed, nearest first.

See the 560 words you can use — click one to add it

Computed “king − man + woman”. The nearest word is still “king”, a word of the formula (0.68). With the input words left out, first place is “queen” (cosine similarity 0.52, distance from the computed point 1.25).

1queen0.52
2kings0.49
3women0.46
4princess0.34
5female0.34
6donna0.33
7emperor0.31
8wife0.28

Why this order? Closeness to the computed point is almost exactly: closeness to “king” + closeness to “woman” − closeness to “man”. Here is the breakdown for the top three.

queen

+ king0.568

+ woman0.271

− man0.100

total0.739

kings

+ king0.709

+ woman0.066

− man0.085

total0.690

women

+ king0.101

+ woman0.717

− man0.176

total0.642

The words are 560 of the 5,000 words used in the video (frequent words that are a single token, out of a vocabulary of 262,144 tokens). From the cosine similarity of every pair of rows (2,560 numbers each) and the length of each row, the closeness of any sum or difference to any word is computed exactly. Because the search covers 560 words, places from second on can differ from the video, which searches 5,000. The 3D picture is the three-dimensional space spanned by the five points, shown as it is.

TRY 2

Count the formulas that work and those that fail

Take pairs with the same relation (male and female, country and capital …) and build one formula from two pairs. If “sister − brother + father” gives “mother”, it is correct. One lamp is one formula. Click a lamp to load its formula into TRY 1.

Kind of pair

12 pairs: man → woman, king → queen, boy → girl, father → mother, brother → sister, son → daughter, uncle → aunt, husband → wife, prince → princess, actor → actress, grandfather → grandmother, nephew → niece

Formula correct (subtract, then add)

124 / 13294%

No arithmetic (just the nearest word)

66 / 13250%

Click a lamp to see its formula and answer (green = correct, red = wrong).

Right and wrong come from the same search as in the video: the nearest of 5,000 words, with the three input words left out. “No arithmetic” simply takes the word nearest to the word being added. The gap between the two is what the subtraction and addition contribute.

TRY 3

Measure the angle between arrows

Pick two pairs: the angle between their arrows (from the first word to the second) is measured. 0 degrees means the same direction, 90 degrees means unrelated. The green dots are all combinations within that kind of pair. The grey band is where arrows between random words fall.

Kind of pair
First arrow
Second arrow
0°30°60°90°120°150°180°77°

The arrows “man → woman” and “king → queen” are 77 degrees apart (cosine similarity 0.22).

For “male → female”, the 66 pairs of arrows average 76 degrees. 66 of the 66 lie outside the random range (84 to 96 degrees).

For arrows between randomly chosen words, 99 pairs out of 100 fall between 84 and 96 degrees (measured for the video). No arrows are neatly parallel. A small lean is what does the work when the answer is picked out of many words.

CHECK

Check yourself

Three questions on what this episode showed.

Q1In Gemma 3's embeddings, which of the 5,000 words is nearest to the point computed by “king − man + woman”?

Q2The arrow from “man” to “woman” and the arrow from “king” to “queen”: how well do they line up?

Q3What happens when “four − one + two” is computed with word points?

Closeness between words was computed from the embedding table (token_embd.weight) of the Gemma 3 4B (Q4_K_M) file (GGUF), read with llama.cpp's gguf-py. The formulas are computed in your browser from the cosine similarity of every pair among 560 words and the length of each row. The hit counts and the random range are the values measured over 5,000 words for the video. Nothing is sent to a server.

FINISH

Finished the episode?

COMMENTS

Comments & questions

Something unclear, something you want to know more about, or just your thoughts: write it here. We answer quickly, and other learners can reply too.

Loading…