संपर्क करें

Understand LLMs from Zero#6

How LLMs Turn Text into Numbers: Token IDs Explained

A neural network is a calculating mechanism, so it can only handle numbers. That is why the first thing an LLM does with a text is turn all of it into numbers. On this page you turn text into numbers, change the numbers, and turn them back into text with your own hands.

  • POINT 1What an LLM reads is not text, but a list of numbers called token IDs.
  • POINT 2A lookup in the vocabulary table turns text into numbers (encoding) and numbers into text (decoding).
  • POINT 3A token ID is only a label. How large it is, or how close to another, has nothing to do with meaning.

TRY 1

Characters already have numbers

Computers handled text as numbers long before LLMs existed, because every character has a number that is agreed on worldwide: its character code. Type any characters you like and look at their numbers. Move a number by one, and the character changes too.

The number of “c” is 99. That is 5 characters in all, so 5 numbers. Pick a character and move its number.

These numbers are set by a worldwide standard called Unicode. Your browser looks them up on the spot, and letters of the alphabet are numbered in order. An LLM, however, does not use these numbers directly. Go on to the next block.

TRY 2

Encoding: a sentence becomes a list of token IDs

The unit an LLM reads is not a character but a token (episode 2). The model has a table called the vocabulary that lists every token it knows, and a token's position in that table is its token ID. Cut the sentence, then look up the tokens one by one.

My cat loves fish.

First, cut the sentence into tokens.

This model's vocabulary table has 262,144 rows, counted from 0. The rules for cutting are covered in detail from episode 9.

TRY 3

Change the numbers and turn them back into text

Now change the numbers themselves. Pick a token and move its ID by one. The decoded result appears below. Is a token whose number is next door also a neighbour in meaning?

Pick the token to change

The current list of numbers[4754, 5866, 18178, 5385, 236761]

DecodedMy cat loves fish.

Pick a token and move its ID by + 1 or − 1.

Press a token below to add it to the end of the list (the small number is its ID)

An ID can be moved up to 6 steps either way (this page carries only that part of the table). Every token you see is from the real vocabulary table. Pieces of the world's languages and symbols sit side by side, in no order of meaning.

TRY 4

Looks the same, different number

The same token always has the same ID. But what looks like the same word to us is a different token as soon as it is written differently. Press the cards and check the numbers. Digits are just tokens, too.

They all look like the same word. Are the numbers the same? Press to find out.

The digits 0–9 are one token each. The small number below is the token ID.

  1. 0236771
  2. 1236770
  3. 2236778
  4. 3236800
  5. 4236812
  6. 5236810
  7. 6236825
  8. 7236832
  9. 8236828
  10. 9236819

The ID of the character “7” is not 7. The IDs are not even in the order 0–9 (a pink frame marks a digit whose ID is smaller than the one before it).

This is part of why an LLM sometimes gets long calculations or letter counts wrong. More in episode 16.

CHECK

Quick check

Three questions on what we found out this time.

Q1What does an LLM actually receive?

Q2What happens when you make a token's ID larger by one?

Q3How does the model's output become text?

Every token, token ID and table row on this page comes from a real language model (Gemma 3 4B, vocabulary of 262,144 rows). The results were extracted in advance, so nothing is sent to a server. The character codes are computed by your browser on the spot.