BPE Part 2: Training a Tokenizer Vocabulary
Last time we merged the most frequent pair, once. This time we repeat it all the way. Count, merge, count again: that is all it takes to build a tokenizer's vocabulary. On this page, you train a tokenizer on your own text.
- POINT 1Training a tokenizer = count the pairs, merge the most frequent one, write it down as a rule, and repeat.
- POINT 2Vocabulary size = starting characters + number of merges. The more often a string appears, the more likely it becomes one token.
- POINT 3Where to stop (the vocabulary size) is a setting people choose. Gemma's vocabulary has 262,144 tokens (2 to the 18th).
TRY 1
Train a vocabulary on your own text
Break the text into single characters, count the pairs, merge the most frequent one, and repeat until no pair appears twice. The text to start with is the 40 sentences this series has used since episode 4. Replace it with any text you like and watch the vocabulary grow.
986← 986Tokens
22= 22 + 0Vocabulary (characters + merges)
0Merges
Right now every character is one token: 986 tokens. The vocabulary is the 22 kinds of characters. Press “Merge once” or “Play”.
- t
- h
- e
- c
- a
- t
- w
- a
- k
- e
- s
- u
- p
- e
- a
- r
- l
- y
- i
- n
- t
- h
- e
- m
- o
- r
- n
- i
- n
- g
- t
- h
- e
- c
- a
- t
- l
- i
- k
- e
- s
- f
- i
- s
- h
- t
- h
- e
- d
- o
- g
- l
- i
- k
- e
- s
- m
- e
- a
- t
- t
- h
- e
- b
- i
- r
- d
- f
- l
- i
- e
- s
- i
- n
- t
- h
- e
- s
- k
- y
- t
- h
- e
- c
- a
- t
- w
- a
- t
- c
- h
- e
- s
- t
- h
- e
- b
- i
- r
- d
- f
- r
- o
- m
- t
- h
- e
- w
- i
- n
- d
- o
- w
- t
- h
- e
- d
- o
- g
- p
- l
- a
- y
- s
- i
- n
- t
- h
- e
- g
- a
- r
- d
- e
- n
- t
- h
- e
- c
- h
- i
- l
- d
- p
- l
- a
- y
- s
- w
- i
- t
- h
- t
- h
- e
- d
- o
- g
- i
- n
- t
- h
- e
- g
- a
- r
- d
- e
- n
- m
- o
- t
- h
- e
- r
- c
- o
- o
- k
- s
- f
- i
- s
- h
- i
- n
- t
- h
- e
- k
- i
- t
- c
- h
- e
- n
- t
- h
- e
- c
- a
- t
- w
- a
- i
- t
- s
- f
- o
- r
- t
- h
- e
- f
- i
- s
- h
- i
- n
- t
- h
- e
- k
- i
- t
- c
- h
- e
- n
- f
- a
- t
- h
- e
- r
- l
- e
- a
- v
- e
- s
- t
- h
- e
- h
- o
- u
- s
- e
- e
- a
- r
- l
- y
- i
- n
- t
- h
- e
- m
- o
- r
- n
- i
- n
- g
- t
- h
- e
- c
- h
- i
- l
- d
- g
- o
- e
- s
- t
- o
- s
- c
- h
- o
- o
- l
- t
- h
- e
- d
- o
- g
- g
- o
- e
- s
- t
- o
- t
- h
- e
- p
- a
- r
- k
- w
- i
- t
- h
- f
- a
- t
- h
- e
- r
- t
- h
- e
- r
- e
- i
- s
- a
- b
- i
- g
- t
- r
- e
- e
- i
- n
- t
- h
- e
- p
- a
- r
- k
- t
- h
- e
- b
- i
- r
- d
- s
- i
- n
- g
- s
- i
- n
- t
- h
- e
- t
- r
- e
- e
- t
- h
- e
- c
- a
- t
- w
- a
- t
- c
- h
- e
- s
- t
- h
- e
- b
- i
- r
- d
- i
- n
- t
- h
- e
- t
- r
- e
- e
- t
- h
- e
- d
- o
- g
- d
- r
- i
- n
- k
- s
- w
- a
- t
- e
- r
- t
- h
- e
- c
- a
- t
- d
- r
- i
- n
- k
- s
- m
- i
- l
- k
- t
- h
- e
- c
- h
- i
- l
- d
- l
- i
- k
- e
- s
- m
- i
- l
- k
- m
- o
- t
- h
- e
- r
- w
- a
- k
- e
- s
- u
- p
- e
- a
- r
- l
- y
- i
- n
- t
- h
- e
- m
- o
- r
- n
- i
- n
- g
- f
- a
- t
- h
- e
- r
- c
- o
- m
- e
- s
- h
- o
- m
- e
- l
- a
- t
- e
- a
- t
- n
- i
- g
- h
- t
- a
- t
- n
- i
- g
- h
- t
- t
- h
- e
- c
- a
- t
- w
- a
- i
- t
- s
- f
- o
- r
- f
- a
- t
- h
- e
- r
- t
- h
- e
- d
- o
- g
- s
- l
- e
- e
- p
- s
- i
- n
- t
- h
- e
- g
- a
- r
- d
- e
- n
- a
- t
- n
- i
- g
- h
- t
- t
- h
- e
- c
- a
- t
- s
- l
- e
- e
- p
- s
- i
- n
- t
- h
- e
- h
- o
- u
- s
- e
- t
- h
- e
- c
- h
- i
- l
- d
- s
- l
- e
- e
- p
- s
- w
- i
- t
- h
- t
- h
- e
- c
- a
- t
- i
- n
- t
- h
- e
- m
- o
- r
- n
- i
- n
- g
- t
- h
- e
- b
- i
- r
- d
- s
- i
- n
- g
- s
- i
- n
- t
- h
- e
- g
- a
- r
- d
- e
- n
- m
- o
- t
- h
- e
- r
- g
- r
- o
- w
- s
- f
- l
- o
- w
- e
- r
- s
- i
- n
- t
- h
- e
- g
- a
- r
- d
- e
- n
- t
- h
- e
- f
- l
- o
- w
- e
- r
- s
- l
- i
- k
- e
- w
- a
- t
- e
- r
- t
- h
- e
- c
- h
- i
- l
- d
- g
- i
- v
- e
- s
- w
- a
- t
- e
- r
- t
- o
- t
- h
- e
- f
- l
- o
- w
- e
- r
- s
- t
- h
- e
- d
- o
- g
- e
- a
- t
- s
- a
- b
- i
- g
- p
- i
- e
- c
- e
- o
- f
- m
- e
- a
- t
- t
- h
- e
- c
- a
- t
- e
- a
- t
- s
- a
- s
- m
- a
- l
- l
- f
- i
- s
- h
- t
- h
- e
- b
- i
- r
- d
- e
- a
- t
- s
- a
- s
- m
- a
- l
- l
- b
- u
- g
- t
- h
- e
- b
- u
- g
- f
- l
- i
- e
- s
- o
- v
- e
- r
- t
- h
- e
- f
- l
- o
- w
- e
- r
- s
- t
- h
- e
- c
- a
- t
- w
- a
- t
- c
- h
- e
- s
- t
- h
- e
- b
- u
- g
- f
- a
- t
- h
- e
- r
- p
- l
- a
- y
- s
- w
- i
- t
- h
- t
- h
- e
- d
- o
- g
- i
- n
- t
- h
- e
- p
- a
- r
- k
- m
- o
- t
- h
- e
- r
- g
- o
- e
- s
- t
- o
- s
- c
- h
- o
- o
- l
- w
- i
- t
- h
- t
- h
- e
- c
- h
- i
- l
- d
- t
- h
- e
- r
- e
- i
- s
- a
- b
- i
- g
- g
- a
- r
- d
- e
- n
- a
- t
- t
- h
- e
- s
- c
- h
- o
- o
- l
- t
- h
- e
- c
- h
- i
- l
- d
- s
- i
- n
- g
- s
- a
- t
- s
- c
- h
- o
- o
- l
- t
- h
- e
- d
- o
- g
- l
- i
- k
- e
- s
- t
- h
- e
- c
- h
- i
- l
- d
- t
- h
- e
- c
- a
- t
- l
- i
- k
- e
- s
- m
- o
- t
- h
- e
- r
- t
- h
- e
- r
- e
- i
- s
- a
- c
- a
- t
- i
- n
- t
- h
- e
- h
- o
- u
- s
- e
Upper case is turned into lower case. When pairs are tied, the one found first in the text wins. The orange outline marks the token just made.
TRY 2
Where to stop? Choosing the vocabulary size
Pick a vocabulary size for the same 40 sentences. The bigger it is, the shorter the sequence, but also the more tokens that are hardly ever used in the training text.
78= 22 + 56Vocabulary size
433← 986Tokens for the 40 sentences
17+ 3tokens never used + used once
With 78 kinds, the 40 sentences become 433 tokens. 17 kinds in the vocabulary are never used, and 3 are used only once.
- the
- cat
- wa
- kes
- u
- p
- earl
- y
- in
- the
- morning
- the
- cat
- likes
- fish
The vocabulary (numbers = how often each is used in the 40 sentences; struck out = never used)
- t6
- h3
- e12
- c3
- a12
- w2
- k12
- s30
- u8
- p12
- r9
- l12
- y7
- i5
- n3
- m9
- o9
- g14
- f6
- d2
- b3
- v3
- th0
- the63
- at6
- in21
- es11
- ch5
- cat13
- ther3
- ar3
- do1
- il2
- mo0
- dog9
- bi3
- li3
- en2
- chil0
- child8
- er4
- kes2
- ing3
- is3
- bir0
- bird6
- wat6
- gar0
- gard0
- garden6
- ee6
- likes5
- eat5
- wi0
- with5
- mother5
- oo1
- fa0
- father5
- ow1
- wa4
- mor0
- morn0
- morning4
- fis0
- fish4
- it4
- ho4
- to4
- sch0
- schoo0
- school4
- fl0
- flow0
- flower0
- flowers4
- ear0
- earl3
Where to stop is a setting people choose before training. Too small and the sequences are long; too big and you get many tokens that hardly ever appear, so the model sees almost no examples of them.
TRY 3
A real vocabulary: Gemma's 262,144 tokens
We read the vocabulary of the model used in this series straight from its file and counted it. Look at what it is made of, the order of its entries, and how a few words are cut.
The vocabulary of Gemma 3 4B (Q4_K_M, llama.cpp) has 262,144 tokens.
| Merged pieces | 236,249 |
| Single characters | 19,225 |
| Bytes (256) | 256 |
| Special symbols etc. | 6,414 |
| Total | 262,144 = 218 |
Entries 494 to 506 of the vocabulary (␣ = space)
| 494 | ␣t |
| 495 | in |
| 496 | ␣a |
| 497 | er |
| 498 | on |
| 499 | he |
| 500 | re |
| 501 | en |
| 502 | at |
| 503 | ␣s |
| 504 | or |
| 505 | ␣c |
| 506 | ␣the |
The IDs follow the order of the merges. The first ones are short strings that are common in English. Number 506 is “␣the”.
Words cut with the real vocabulary (small numbers = token IDs)
- the506
- garden7972
- morning5597
- ant2314
- idis33090
- establishment163524
- arian8225
- ism1929
- floc184655
- cin7047
- auc20683
- ini2818
- hil47425
- ip760
- il543
- ification2540
- <E9>471
- <B6>420
- <BA>424
- <E9>471
- <B4>418
- <92>384
“the” is one token. Common words are in the vocabulary as a whole.
The tokenizer is made before the model, and is fixed afterwards. The model then practises guessing the next token, using the tokens of this vocabulary.
CHECK
Quick check
Three questions on what this episode showed.
Q1You start with 22 kinds of characters and merge 8 times. How big is the vocabulary?
Q2Which strings are most likely to become one token?
Q3Who decides the vocabulary size (where to stop)?
NEXT #12
Tokenizer Encode and Decode: From Text to Token IDs and Back
Next time, we use the finished vocabulary and its rules to turn text into a sequence of token IDs (encoding), and turn the IDs back into the text (decoding).
The merges and token counts on this page are computed by your browser on the spot. The breakdown, order and token IDs of the real vocabulary were taken in advance from the file and tokenizer of a language model (Gemma 3 4B (Q4_K_M, llama.cpp), vocabulary of 262,144). Nothing is sent to a server.