Kontakta oss

Understand LLMs from Zero#10

Byte Pair Encoding (BPE) Part 1: Merging the Most Frequent Pair

Who decides the pieces of words (subwords), and how? The answer is one simple move: count, then merge. On this page you break a sentence into characters, count the pairs of neighbours, and merge pairs with your own hands.

  • POINT 1BPE starts from characters and counts how often each pair of neighbours appears.
  • POINT 2The most frequent pair is joined into one new token. That is a merge.
  • POINT 3One merge adds one entry to the vocabulary and makes the sequence shorter. Meaning plays no part.

TRY 1

Break it into characters, count the pairs

BPE starts from characters. Break the text into single letters, take out every pair of neighbours, and count how many times each pair appears. Type any text you like and count.

Examples:
  1. p
  2. e
  3. t
  4. e
  5. r
  6. p
  7. i
  8. p
  9. e
  10. r
  11. p
  12. i
  13. c
  14. k
  15. e
  16. d
  17. a
  18. p
  19. e
  20. c
  21. k
  22. o
  23. f
  24. p
  25. i
  26. c
  27. k
  28. l
  29. e
  30. d
  31. p
  32. e
  33. p
  34. p
  35. e
  36. r
  37. s

37tokens (one letter each for now)

13kinds

29pairs

37 tokens give 29 pairs. Let's count how many times each pair appears.

In a text with spaces, pairs are made only inside a word (a pair never crosses a space). Capital letters are turned into lowercase. Pairs with the same count are listed in the order they first appear in the text.

TRY 2

Pick a pair yourself and merge it

Now you choose which pair to merge. Press “Merge” in the table and every occurrence of that pair in the text is joined into one new token. Compare how much shorter the sequence gets when you pick the most frequent pair, and when you pick a rare one.

Examples:
  1. p
  2. e
  3. t
  4. e
  5. r
  6. p
  7. i
  8. p
  9. e
  10. r
  11. p
  12. i
  13. c
  14. k
  15. e
  16. d
  17. a
  18. p
  19. e
  20. c
  21. k
  22. o
  23. f
  24. p
  25. i
  26. c
  27. k
  28. l
  29. e
  30. d
  31. p
  32. e
  33. p
  34. p
  35. e
  36. r
  37. s

37← 37tokens

13← 13vocabulary (letters + merged tokens)

0merges done

Pick a pair in the table and press “Merge”. The top row is the most frequent pair.

The pairs right now and their counts (recounted after every merge)

  1. p+e5
  2. e+r3
  3. p+i3
  4. c+k3
  5. i+c2
  6. e+d2
  7. e+t1
  8. t+e1
  9. i+p1
  10. k+e1

A merge changes the counts of the pairs around it. Some pairs disappear, and new pairs that contain the new token are born. That is why we count again from scratch every time. A merged token can always be turned back into its letters by following the rule backwards.

TRY 3

Frequent pairs first, or rare pairs first?

Two strategies run side by side on the same text. On the left, BPE's way: always merge the most frequent pair. On the right: always merge the rarest pair. Press the button and compare them one move at a time.

Always the most frequent pair37

  1. p
  2. e
  3. t
  4. e
  5. r
  6. p
  7. i
  8. p
  9. e
  10. r
  11. p
  12. i
  13. c
  14. k
  15. e
  16. d
  17. a
  18. p
  19. e
  20. c
  21. k
  22. o
  23. f
  24. p
  25. i
  26. c
  27. k
  28. l
  29. e
  30. d
  31. p
  32. e
  33. p
  34. p
  35. e
  36. r
  37. s

Always the rarest pair37

  1. p
  2. e
  3. t
  4. e
  5. r
  6. p
  7. i
  8. p
  9. e
  10. r
  11. p
  12. i
  13. c
  14. k
  15. e
  16. d
  17. a
  18. p
  19. e
  20. c
  21. k
  22. o
  23. f
  24. p
  25. i
  26. c
  27. k
  28. l
  29. e
  30. d
  31. p
  32. e
  33. p
  34. p
  35. e
  36. r
  37. s

Both start with 37 tokens. Make one move.

With either strategy, each merge adds one entry to the vocabulary. For the same vocabulary size we want the sequence as short as possible. That is why BPE picks the most frequent pair.

TRY 4

A real vocabulary, and the bytes under the characters

Take the sentence from the video. Compare what you get by counting only this sentence and merging on and on, with how a real model's vocabulary cuts it. Then look at the “byte” in BPE's name.

Counting only this sentence, 1 merges of the most frequent pair: 32 tokens

  1. pe
  2. t
  3. e
  4. r
  5. p
  6. i
  7. pe
  8. r
  9. p
  10. i
  11. c
  12. k
  13. e
  14. d
  15. a
  16. pe
  17. c
  18. k
  19. o
  20. f
  21. p
  22. i
  23. c
  24. k
  25. l
  26. e
  27. d
  28. pe
  29. p
  30. pe
  31. r
  32. s

Cut with the vocabulary of a real model (Gemma 3 4B (Q4_K_M, llama.cpp)): 8 tokens (small numbers are token IDs)

  1. peter144328
  2. piper120622
  3. picked14917
  4. a496
  5. peck222406
  6. of529
  7. pickled103853
  8. peppers47948

37 letters become 32 tokens after 1 merges on this sentence alone. With the real vocabulary: 8 tokens.

The real model's vocabulary has 262,144 entries. Combinations that are common out in the world have been joined. How a vocabulary grows through many merges is the topic of the next episode.

  1. p70
  2. éC3 A9
  3. 🐔F0 9F 90 94

Characters: 3. Bytes: 7.

There are only 256 different bytes (00 to FF). An English letter is 1 byte, an accented letter 2, a Japanese character 3, an emoji 4 (the UTF-8 rule). BPE began as a compression technique that merges pairs of bytes, and GPT's tokenizer starts from bytes, not characters. That way any character can always be written.

CHECK

Check yourself

Three questions on what this episode showed.

Q1How does BPE choose the next pair to merge?

Q2What does one merge do?

Q3Why do we count the pairs again after a merge?

The pair counts and merges on this page are computed by your browser as you go. The real tokens and token IDs were taken in advance from the tokenizer of a language model (Gemma 3 4B (Q4_K_M, llama.cpp), vocabulary of 262,144 rows). Nothing is sent to a server.