Napište nám

Understand LLMs from Zero#18

Dot Product and Cosine Similarity: How LLMs Measure Meaning

How much do two word vectors point the same way? The dot product and cosine similarity answer with one number. Move the arrows to see the calculation, then measure how close real words of the model are.

  • POINT 1The dot product multiplies the numbers in the same position and adds everything: (3, 1) · (1, 2) = 3 × 1 + 1 × 2 = 5. Nearly the same way: plus. Right angle: 0. Opposite: minus.
  • POINT 2The dot product contains the lengths too. Divided by the two lengths it becomes cosine similarity, which shows only the direction, from −1 to 1 (same way 1, right angle 0, opposite −1).
  • POINT 3Measured in Gemma 3, “cat” and “dog” give about 0.37. 98% of random pairs of tokens fall between −0.1 and 0.1, so that is clearly close. But opposite words come out close too.

TRY 1

Move two arrows, watch the similarity

Drag the tips (the balls) of the cyan arrow a and the yellow arrow b. The dot product, the lengths, the cosine similarity and the angle are worked out again at once. The buttons turn b to the same direction, a right angle or the opposite.

Make b
ab
|a|3.16
|b|2.24
dot product a · b5
cosine similarity0.71angle 45°
Dot product = multiply position by position, add(3, 1) · (1, 2) = 3 × 1 + 1 × 2 = 5Cosine similarity = dot product ÷ (|a| × |b|)5 ÷ (3.16 × 2.24) = 5 ÷ 7.07 ≈ 0.71

Positive dot product: the two point to the same side (the angle is under 90°).

Press “double a”: the dot product doubles, but the cosine similarity stays the same. It only looks at direction.

TRY 2

How close are real Gemma words?

In Gemma 3 4B every token is an arrow of 2560 numbers. Pick two words to see the cosine similarity of their arrows. Below, every measured word is sorted by its similarity to the first word.

Choosing
cosine similarity0.37
dot product (sum of 2560 products)0.391
lengths1.03 · 1.04

The cosine similarity of the 2560-dimensional arrows of “cat” and “dog” is 0.37.

Cosine similarity with “cat” (closest first). The pink band is −0.1 to 0.1.

cats0.72
kitten0.51
dog0.37
horse0.21
bird0.21
car0.21
mouse0.18
tiger0.18
cow0.17
lion0.17
bat0.15
fish0.14
man0.14
guitar0.13
bicycle0.13
train0.13
red0.12
woman0.12
piano0.12
apple0.11
book0.11
water0.11
music0.11
king0.11
blue0.10
small0.10
plane0.10
cold0.10
queen0.09
big0.09
table0.09
love0.09
Paris0.09
winter0.09
hot0.08
bank0.08
truck0.08
vehicle0.08
London0.08
sad0.08
Apple0.08
spring0.07
summer0.07
Tokyo0.07
left0.06
right0.06
happy0.06
hate0.05
Tuesday0.05
Monday0.04

Of 199,999 random pairs of tokens, 98% had a cosine similarity between −0.1 and 0.1 (almost a right angle). In 2560 dimensions, unrelated arrows are almost at right angles.

The English words are the tokens with a space in front (“ cat”), the form a word has inside a sentence.

TRY 3

The closest of all 262,144 tokens

Pick a word to see the 10 tokens of the whole vocabulary with the highest cosine similarity. Many are other spellings of the same word, or the same meaning in another language.

Tokens closest to “cat” (out of 262,144)

1. ␣Cat0.82
2. cat0.79
3. Cat0.77
4. ␣cats0.72
5. ␣CAT0.65
6. ␣Cats0.60
7. CAT0.60
8. 猫0.58
9. ␣kat0.57
10. ␣feline0.57

␣ marks a token with a space in front (in English, a word inside a sentence has this form).

Opposite words can be close too (for example “summer” and “winter”). A high cosine similarity does not always mean the same meaning.

CHECK

Quick check

Check what you learned with three questions.

Q1The dot product of (2, 1) and (−1, 2) is…

Q2Make arrow a twice as long. The cosine similarity of a and b…

Q3In Gemma, “hot” and “cold” have a high cosine similarity because…

The numbers in TRY 2 and 3 were computed from the embedding table (token_embd.weight) in the Gemma 3 4B (Q4_K_M) model file (GGUF), read with llama.cpp's gguf-py. The arrows in TRY 1 are numbers you choose. Nothing is sent to a server.

FINISH

Finished the episode?

COMMENTS

Comments & questions

Something unclear, something you want to know more about, or just your thoughts: write it here. We answer quickly, and other learners can reply too.

Loading…