Contact

Understand LLMs from Zero#23

What Is a Matrix? The Linear Transformations Behind Every LLM

A matrix is a table of numbers, arranged across and down. With two rows and two columns there are only four numbers. Those four numbers move the whole plane. Move them yourself and watch the picture of a cat change. The inside of an LLM is almost all matrices like this.

  • POINT 1Multiply a matrix by a vector and it takes one dot product per row, giving back a new vector. For two rows and two columns: two dot products.
  • POINT 2Multiply every point of the plane by the same matrix and the whole picture moves. The left column is where “one step right” goes, the right column is where “one step up” goes. Turn, stretch, slant, flip. When the area factor is 0, the picture is crushed and cannot be restored.
  • POINT 3The file of Gemma 3 4B holds 239 matrices. 99.99% of the model's numbers (about 3.88 billion) are inside matrices. The rule is the same as two by two. Only the size differs.

TRY 1

Move the four numbers of a matrix

Move the four sliders and the cat, drawn with beads, is reshaped by that matrix. The red dot is where “one step right (1, 0)” goes, the blue dot is where “one step up (0, 1)” goes.

Matrices with a name

The red dot (one step right) goes to (2, 0): the left column of the matrix. The blue dot (one step up) goes to (0, 1): the right column.

Area factor (the determinant) = 2 × 1 − 0 × 0 = 2. The picture is not crushed.

The matrix that undoes it: 0.5001

Each slider goes from −2 to 2 in steps of 0.1. Whatever the matrix, straight lines stay straight and the middle point does not move (a linear transformation). No matrix can bend the picture.

TRY 2

Matrix × vector, by hand

Multiply one point by the matrix of TRY 1. The dot product of the top row with the point is its new position across. The dot product of the bottom row is its new position up.

The matrix right now (change it in TRY 1): 2001

top row2 × 1 + 0 × 1.5 = 2
bottom row0 × 1 + 1 × 1.5 = 1.5

The point (1, 1.5) moves to (2, 1.5).

The dashed circle is the original point, the yellow one is where it goes. Every point of the picture moves by this same calculation.

TRY 3

The same matrix, again and again

You can multiply by a matrix as many times as you like. Each press of “multiply once more” adds one stamp of the cat. However many times, it all combines into a single matrix.

Matrix to repeat

Multiplied 3 times. 0.87−0.50.50.87 → combined: 0−110

“Turn 30°” comes around in 12 steps. “Shrink and turn” spirals in to the middle. “Grow × 1.1” is a small step, but repeated it grows without end. You can press up to 24 times.

TRY 4

Swap the order

Multiply by two different matrices, one after the other. Is the result the same if you swap the order?

First matrix
Second matrix

twice as wide → turn 90°

0−120

turn 90° → twice as wide

0−210

The order changes the result, and the four numbers of the combined matrix too.

Above each picture is the single matrix the two combine into. However many changes you stack up, the result is again just four numbers.

TRY 5

The real matrices inside Gemma

We opened the file of Gemma 3 4B and counted every matrix in it. Press a shape to see how many matrices have it, how many numbers they take in and how many they give back.

239 matrices (plus 205 plain lists of numbers). The model has 3,880,099,328 numbers in all. 99.99% of them are inside matrices.

68 matrices of 10,240 rows × 2,560 columns, 26,214,400 numbers each. Each takes in 2,560 numbers and gives back 10,240. More numbers come out than go in.

We pulled one out. Its name in the file is blk.0.attn_q.weight. 2,048 rows × 2,560 columns, 5,242,880 numbers. This is its top left corner, 6 rows × 6 columns.

0.015−0.0040.000−0.0040.012−0.008…
−0.024−0.015−0.0150.003−0.024−0.001…
0.0140.008−0.033−0.012−0.0120.029…
0.0180.0140.0020.0180.0300.026…
0.0150.010−0.010−0.026−0.0210.015…
0.0030.0150.015−0.0180.020−0.001…
⋮⋮⋮⋮⋮⋮

Multiply this matrix by the row of “cat” (2,560 numbers) and 2,048 numbers come out. That takes 5,242,880 multiplications. The first output is the dot product with row 1: 0.015 × −0.002 + −0.004 × 0.022 + … = −0.009.

row of “cat”

−0.002

0.022

0.018

0.024

−0.001

0.026

⋮

output

−0.009

0.011

−0.012

−0.009

−0.012

−0.015

⋮

shifted input

−0.003

0.061

−0.004

0.075

−0.010

0.061

⋮

its output

−0.009

0.011

−0.012

−0.009

−0.012

−0.015

⋮

We shifted the numbers of “cat” by 1.03 in a certain direction (the original row has length 1.03). The 2,048 outputs are still the same: the largest difference between the two outputs is 0.0000000000000002. Different inputs, the same output. A change that cannot be undone.

The numbers are decompressed from the file (which stores them in about 4 bits each). Inside the model, the row is rescaled before it reaches this matrix; here we multiply the raw row. What each matrix is for comes in later episodes.

CHECK

Check yourself

Three questions on what this episode showed.

Q1Multiply the matrix “2, 1 / 1, 3” by the vector (3, 2). What comes out?

Q2The matrix “1, 0 / 0, 0” has crushed the picture. Which matrix restores it?

Q3Of Gemma 3 4B's numbers (about 3.88 billion), how many are inside matrices?

The number of matrices, their shapes and the numbers inside were read from the Gemma 3 4B (Q4_K_M) file (gemma-3-4b-it-Q4_K_M.gguf) with llama.cpp's gguf-py. The two-by-two changes are computed in your browser. Nothing is sent to a server.

FINISH

Finished the episode?

COMMENTS

Comments & questions

Something unclear, something you want to know more about, or just your thoughts: write it here. We answer quickly, and other learners can reply too.

Loading…