Konzultáció

Understand LLMs from Zero#11

BPE Part 2: Training a Tokenizer Vocabulary

Last time we merged the most frequent pair, once. This time we repeat it all the way. Count, merge, count again: that is all it takes to build a tokenizer's vocabulary. On this page, you train a tokenizer on your own text.

  • POINT 1Training a tokenizer = count the pairs, merge the most frequent one, write it down as a rule, and repeat.
  • POINT 2Vocabulary size = starting characters + number of merges. The more often a string appears, the more likely it becomes one token.
  • POINT 3Where to stop (the vocabulary size) is a setting people choose. Gemma's vocabulary has 262,144 tokens (2 to the 18th).
The video is coming soonOnce it is published, you can watch it here.

TRY 1

Train a vocabulary on your own text

Break the text into single characters, count the pairs, merge the most frequent one, and repeat until no pair appears twice. The text to start with is the 40 sentences this series has used since episode 4. Replace it with any text you like and watch the vocabulary grow.

Text:
0 / 112

986← 986Tokens

22= 22 + 0Vocabulary (characters + merges)

0Merges

Right now every character is one token: 986 tokens. The vocabulary is the 22 kinds of characters. Press “Merge once” or “Play”.

  1. t
  2. h
  3. e
  4. c
  5. a
  6. t
  7. w
  8. a
  9. k
  10. e
  11. s
  12. u
  13. p
  14. e
  15. a
  16. r
  17. l
  18. y
  19. i
  20. n
  21. t
  22. h
  23. e
  24. m
  25. o
  26. r
  27. n
  28. i
  29. n
  30. g
  31. t
  32. h
  33. e
  34. c
  35. a
  36. t
  37. l
  38. i
  39. k
  40. e
  41. s
  42. f
  43. i
  44. s
  45. h
  46. t
  47. h
  48. e
  49. d
  50. o
  51. g
  52. l
  53. i
  54. k
  55. e
  56. s
  57. m
  58. e
  59. a
  60. t
  61. t
  62. h
  63. e
  64. b
  65. i
  66. r
  67. d
  68. f
  69. l
  70. i
  71. e
  72. s
  73. i
  74. n
  75. t
  76. h
  77. e
  78. s
  79. k
  80. y
  81. t
  82. h
  83. e
  84. c
  85. a
  86. t
  87. w
  88. a
  89. t
  90. c
  91. h
  92. e
  93. s
  94. t
  95. h
  96. e
  97. b
  98. i
  99. r
  100. d
  101. f
  102. r
  103. o
  104. m
  105. t
  106. h
  107. e
  108. w
  109. i
  110. n
  111. d
  112. o
  113. w
  114. t
  115. h
  116. e
  117. d
  118. o
  119. g
  120. p
  121. l
  122. a
  123. y
  124. s
  125. i
  126. n
  127. t
  128. h
  129. e
  130. g
  131. a
  132. r
  133. d
  134. e
  135. n
  136. t
  137. h
  138. e
  139. c
  140. h
  141. i
  142. l
  143. d
  144. p
  145. l
  146. a
  147. y
  148. s
  149. w
  150. i
  151. t
  152. h
  153. t
  154. h
  155. e
  156. d
  157. o
  158. g
  159. i
  160. n
  161. t
  162. h
  163. e
  164. g
  165. a
  166. r
  167. d
  168. e
  169. n
  170. m
  171. o
  172. t
  173. h
  174. e
  175. r
  176. c
  177. o
  178. o
  179. k
  180. s
  181. f
  182. i
  183. s
  184. h
  185. i
  186. n
  187. t
  188. h
  189. e
  190. k
  191. i
  192. t
  193. c
  194. h
  195. e
  196. n
  197. t
  198. h
  199. e
  200. c
  201. a
  202. t
  203. w
  204. a
  205. i
  206. t
  207. s
  208. f
  209. o
  210. r
  211. t
  212. h
  213. e
  214. f
  215. i
  216. s
  217. h
  218. i
  219. n
  220. t
  221. h
  222. e
  223. k
  224. i
  225. t
  226. c
  227. h
  228. e
  229. n
  230. f
  231. a
  232. t
  233. h
  234. e
  235. r
  236. l
  237. e
  238. a
  239. v
  240. e
  241. s
  242. t
  243. h
  244. e
  245. h
  246. o
  247. u
  248. s
  249. e
  250. e
  251. a
  252. r
  253. l
  254. y
  255. i
  256. n
  257. t
  258. h
  259. e
  260. m
  261. o
  262. r
  263. n
  264. i
  265. n
  266. g
  267. t
  268. h
  269. e
  270. c
  271. h
  272. i
  273. l
  274. d
  275. g
  276. o
  277. e
  278. s
  279. t
  280. o
  281. s
  282. c
  283. h
  284. o
  285. o
  286. l
  287. t
  288. h
  289. e
  290. d
  291. o
  292. g
  293. g
  294. o
  295. e
  296. s
  297. t
  298. o
  299. t
  300. h
  301. e
  302. p
  303. a
  304. r
  305. k
  306. w
  307. i
  308. t
  309. h
  310. f
  311. a
  312. t
  313. h
  314. e
  315. r
  316. t
  317. h
  318. e
  319. r
  320. e
  321. i
  322. s
  323. a
  324. b
  325. i
  326. g
  327. t
  328. r
  329. e
  330. e
  331. i
  332. n
  333. t
  334. h
  335. e
  336. p
  337. a
  338. r
  339. k
  340. t
  341. h
  342. e
  343. b
  344. i
  345. r
  346. d
  347. s
  348. i
  349. n
  350. g
  351. s
  352. i
  353. n
  354. t
  355. h
  356. e
  357. t
  358. r
  359. e
  360. e
  361. t
  362. h
  363. e
  364. c
  365. a
  366. t
  367. w
  368. a
  369. t
  370. c
  371. h
  372. e
  373. s
  374. t
  375. h
  376. e
  377. b
  378. i
  379. r
  380. d
  381. i
  382. n
  383. t
  384. h
  385. e
  386. t
  387. r
  388. e
  389. e
  390. t
  391. h
  392. e
  393. d
  394. o
  395. g
  396. d
  397. r
  398. i
  399. n
  400. k
  401. s
  402. w
  403. a
  404. t
  405. e
  406. r
  407. t
  408. h
  409. e
  410. c
  411. a
  412. t
  413. d
  414. r
  415. i
  416. n
  417. k
  418. s
  419. m
  420. i
  421. l
  422. k
  423. t
  424. h
  425. e
  426. c
  427. h
  428. i
  429. l
  430. d
  431. l
  432. i
  433. k
  434. e
  435. s
  436. m
  437. i
  438. l
  439. k
  440. m
  441. o
  442. t
  443. h
  444. e
  445. r
  446. w
  447. a
  448. k
  449. e
  450. s
  451. u
  452. p
  453. e
  454. a
  455. r
  456. l
  457. y
  458. i
  459. n
  460. t
  461. h
  462. e
  463. m
  464. o
  465. r
  466. n
  467. i
  468. n
  469. g
  470. f
  471. a
  472. t
  473. h
  474. e
  475. r
  476. c
  477. o
  478. m
  479. e
  480. s
  481. h
  482. o
  483. m
  484. e
  485. l
  486. a
  487. t
  488. e
  489. a
  490. t
  491. n
  492. i
  493. g
  494. h
  495. t
  496. a
  497. t
  498. n
  499. i
  500. g
  501. h
  502. t
  503. t
  504. h
  505. e
  506. c
  507. a
  508. t
  509. w
  510. a
  511. i
  512. t
  513. s
  514. f
  515. o
  516. r
  517. f
  518. a
  519. t
  520. h
  521. e
  522. r
  523. t
  524. h
  525. e
  526. d
  527. o
  528. g
  529. s
  530. l
  531. e
  532. e
  533. p
  534. s
  535. i
  536. n
  537. t
  538. h
  539. e
  540. g
  541. a
  542. r
  543. d
  544. e
  545. n
  546. a
  547. t
  548. n
  549. i
  550. g
  551. h
  552. t
  553. t
  554. h
  555. e
  556. c
  557. a
  558. t
  559. s
  560. l
  561. e
  562. e
  563. p
  564. s
  565. i
  566. n
  567. t
  568. h
  569. e
  570. h
  571. o
  572. u
  573. s
  574. e
  575. t
  576. h
  577. e
  578. c
  579. h
  580. i
  581. l
  582. d
  583. s
  584. l
  585. e
  586. e
  587. p
  588. s
  589. w
  590. i
  591. t
  592. h
  593. t
  594. h
  595. e
  596. c
  597. a
  598. t
  599. i
  600. n
  601. t
  602. h
  603. e
  604. m
  605. o
  606. r
  607. n
  608. i
  609. n
  610. g
  611. t
  612. h
  613. e
  614. b
  615. i
  616. r
  617. d
  618. s
  619. i
  620. n
  621. g
  622. s
  623. i
  624. n
  625. t
  626. h
  627. e
  628. g
  629. a
  630. r
  631. d
  632. e
  633. n
  634. m
  635. o
  636. t
  637. h
  638. e
  639. r
  640. g
  641. r
  642. o
  643. w
  644. s
  645. f
  646. l
  647. o
  648. w
  649. e
  650. r
  651. s
  652. i
  653. n
  654. t
  655. h
  656. e
  657. g
  658. a
  659. r
  660. d
  661. e
  662. n
  663. t
  664. h
  665. e
  666. f
  667. l
  668. o
  669. w
  670. e
  671. r
  672. s
  673. l
  674. i
  675. k
  676. e
  677. w
  678. a
  679. t
  680. e
  681. r
  682. t
  683. h
  684. e
  685. c
  686. h
  687. i
  688. l
  689. d
  690. g
  691. i
  692. v
  693. e
  694. s
  695. w
  696. a
  697. t
  698. e
  699. r
  700. t
  701. o
  702. t
  703. h
  704. e
  705. f
  706. l
  707. o
  708. w
  709. e
  710. r
  711. s
  712. t
  713. h
  714. e
  715. d
  716. o
  717. g
  718. e
  719. a
  720. t
  721. s
  722. a
  723. b
  724. i
  725. g
  726. p
  727. i
  728. e
  729. c
  730. e
  731. o
  732. f
  733. m
  734. e
  735. a
  736. t
  737. t
  738. h
  739. e
  740. c
  741. a
  742. t
  743. e
  744. a
  745. t
  746. s
  747. a
  748. s
  749. m
  750. a
  751. l
  752. l
  753. f
  754. i
  755. s
  756. h
  757. t
  758. h
  759. e
  760. b
  761. i
  762. r
  763. d
  764. e
  765. a
  766. t
  767. s
  768. a
  769. s
  770. m
  771. a
  772. l
  773. l
  774. b
  775. u
  776. g
  777. t
  778. h
  779. e
  780. b
  781. u
  782. g
  783. f
  784. l
  785. i
  786. e
  787. s
  788. o
  789. v
  790. e
  791. r
  792. t
  793. h
  794. e
  795. f
  796. l
  797. o
  798. w
  799. e
  800. r
  801. s
  802. t
  803. h
  804. e
  805. c
  806. a
  807. t
  808. w
  809. a
  810. t
  811. c
  812. h
  813. e
  814. s
  815. t
  816. h
  817. e
  818. b
  819. u
  820. g
  821. f
  822. a
  823. t
  824. h
  825. e
  826. r
  827. p
  828. l
  829. a
  830. y
  831. s
  832. w
  833. i
  834. t
  835. h
  836. t
  837. h
  838. e
  839. d
  840. o
  841. g
  842. i
  843. n
  844. t
  845. h
  846. e
  847. p
  848. a
  849. r
  850. k
  851. m
  852. o
  853. t
  854. h
  855. e
  856. r
  857. g
  858. o
  859. e
  860. s
  861. t
  862. o
  863. s
  864. c
  865. h
  866. o
  867. o
  868. l
  869. w
  870. i
  871. t
  872. h
  873. t
  874. h
  875. e
  876. c
  877. h
  878. i
  879. l
  880. d
  881. t
  882. h
  883. e
  884. r
  885. e
  886. i
  887. s
  888. a
  889. b
  890. i
  891. g
  892. g
  893. a
  894. r
  895. d
  896. e
  897. n
  898. a
  899. t
  900. t
  901. h
  902. e
  903. s
  904. c
  905. h
  906. o
  907. o
  908. l
  909. t
  910. h
  911. e
  912. c
  913. h
  914. i
  915. l
  916. d
  917. s
  918. i
  919. n
  920. g
  921. s
  922. a
  923. t
  924. s
  925. c
  926. h
  927. o
  928. o
  929. l
  930. t
  931. h
  932. e
  933. d
  934. o
  935. g
  936. l
  937. i
  938. k
  939. e
  940. s
  941. t
  942. h
  943. e
  944. c
  945. h
  946. i
  947. l
  948. d
  949. t
  950. h
  951. e
  952. c
  953. a
  954. t
  955. l
  956. i
  957. k
  958. e
  959. s
  960. m
  961. o
  962. t
  963. h
  964. e
  965. r
  966. t
  967. h
  968. e
  969. r
  970. e
  971. i
  972. s
  973. a
  974. c
  975. a
  976. t
  977. i
  978. n
  979. t
  980. h
  981. e
  982. h
  983. o
  984. u
  985. s
  986. e
Tokens 986Vocabulary (characters + merges) 22Merges 1120

Upper case is turned into lower case. When pairs are tied, the one found first in the text wins. The orange outline marks the token just made.

TRY 2

Where to stop? Choosing the vocabulary size

Pick a vocabulary size for the same 40 sentences. The bigger it is, the shorter the sequence, but also the more tokens that are hardly ever used in the training text.

78

78= 22 + 56Vocabulary size

433← 986Tokens for the 40 sentences

17+ 3tokens never used + used once

With 78 kinds, the 40 sentences become 433 tokens. 17 kinds in the vocabulary are never used, and 3 are used only once.

  1. the
  2. cat
  3. wa
  4. kes
  5. u
  6. p
  7. earl
  8. y
  9. in
  10. the
  11. morning
  12. the
  13. cat
  14. likes
  15. fish

The vocabulary (numbers = how often each is used in the 40 sentences; struck out = never used)

  • t6
  • h3
  • e12
  • c3
  • a12
  • w2
  • k12
  • s30
  • u8
  • p12
  • r9
  • l12
  • y7
  • i5
  • n3
  • m9
  • o9
  • g14
  • f6
  • d2
  • b3
  • v3
  • th0
  • the63
  • at6
  • in21
  • es11
  • ch5
  • cat13
  • ther3
  • ar3
  • do1
  • il2
  • mo0
  • dog9
  • bi3
  • li3
  • en2
  • chil0
  • child8
  • er4
  • kes2
  • ing3
  • is3
  • bir0
  • bird6
  • wat6
  • gar0
  • gard0
  • garden6
  • ee6
  • likes5
  • eat5
  • wi0
  • with5
  • mother5
  • oo1
  • fa0
  • father5
  • ow1
  • wa4
  • mor0
  • morn0
  • morning4
  • fis0
  • fish4
  • it4
  • ho4
  • to4
  • sch0
  • schoo0
  • school4
  • fl0
  • flow0
  • flower0
  • flowers4
  • ear0
  • earl3

Where to stop is a setting people choose before training. Too small and the sequences are long; too big and you get many tokens that hardly ever appear, so the model sees almost no examples of them.

TRY 3

A real vocabulary: Gemma's 262,144 tokens

We read the vocabulary of the model used in this series straight from its file and counted it. Look at what it is made of, the order of its entries, and how a few words are cut.

The vocabulary of Gemma 3 4B (Q4_K_M, llama.cpp) has 262,144 tokens.

Merged pieces236,249
Single characters19,225
Bytes (256)256
Special symbols etc.6,414
Total262,144 = 218

Entries 494 to 506 of the vocabulary (␣ = space)

494␣t
495in
496␣a
497er
498on
499he
500re
501en
502at
503␣s
504or
505␣c
506␣the

The IDs follow the order of the merges. The first ones are short strings that are common in English. Number 506 is “␣the”.

Words cut with the real vocabulary (small numbers = token IDs)

the
  1. the506
1
garden
  1. garden7972
1
morning
  1. morning5597
1
antidisestablishmentarianism
  1. ant2314
  2. idis33090
  3. establishment163524
  4. arian8225
  5. ism1929
5
floccinaucinihilipilification
  1. floc184655
  2. cin7047
  3. auc20683
  4. ini2818
  5. hil47425
  6. ip760
  7. il543
  8. ification2540
8
鶺鴒
  1. <E9>471
  2. <B6>420
  3. <BA>424
  4. <E9>471
  5. <B4>418
  6. <92>384
6

“the” is one token. Common words are in the vocabulary as a whole.

The tokenizer is made before the model, and is fixed afterwards. The model then practises guessing the next token, using the tokens of this vocabulary.

CHECK

Quick check

Three questions on what this episode showed.

Q1You start with 22 kinds of characters and merge 8 times. How big is the vocabulary?

Q2Which strings are most likely to become one token?

Q3Who decides the vocabulary size (where to stop)?

The merges and token counts on this page are computed by your browser on the spot. The breakdown, order and token IDs of the real vocabulary were taken in advance from the file and tokenizer of a language model (Gemma 3 4B (Q4_K_M, llama.cpp), vocabulary of 262,144). Nothing is sent to a server.