Skip to content
<UK/>
ENTR

Tokenizer playground

Accretion

Watch a tokenizer learn, merge by merge, and see why the same sentence can cost more tokens in Turkish.

Runs in your browser · a teaching tokenizer, not a production model

Watch it learn

Byte-pair encoding starts from single characters. At every step it finds the pair of neighbours that appears most often inside words and fuses it into one new token. Dust becomes pebbles; letters become subwords.

Training text
Read the training text (30)
Speed

Play and its speed are hidden because your system asks for reduced motion. Step, +10 and the slider still work.

Vocabulary
3029 characters + end-of-word + 0 merges
Corpus tokens
EN 1,611 · TR 1,607

Specimens

Three English and three Turkish words from this room’s example sentences, split by the tokenizer as it is right now.

  • ENplanets
    1. p
    2. l
    3. a
    4. n
    5. e
    6. t
    7. s
    8. end of word
  • ENtogether
    1. t
    2. o
    3. g
    4. e
    5. t
    6. h
    7. e
    8. r
    9. end of word
  • ENquestions
    1. q
    2. u
    3. e
    4. s
    5. t
    6. i
    7. o
    8. n
    9. s
    10. end of word
  • TRyıldızlara
    1. y
    2. ı (unknown character)
    3. l
    4. d
    5. ı (unknown character)
    6. z
    7. l
    8. a
    9. r
    10. a
    11. end of word
  • TRarkadaşlarımızla
    1. a
    2. r
    3. k
    4. a
    5. d
    6. a
    7. ş (unknown character)
    8. l
    9. a
    10. r
    11. ı (unknown character)
    12. m
    13. ı (unknown character)
    14. z
    15. l
    16. a
    17. end of word
  • TRbitiremedik
    1. b
    2. i
    3. t
    4. i
    5. r
    6. e
    7. m
    8. e
    9. d
    10. i
    11. k
    12. end of word

Latest merge

No merges yet. Press Step or Play.

CompressionEnglish half 1,611Turkish half 1,607

Merge history

The merges appear here as the tokenizer learns them.

Same meaning, different bill

Each pair says the same thing in English and Turkish. They are example sentences written for this room, and none of them is in the built-in training text. Both sides use the tokenizer from above, as trained right now.

Sentence pair
  1. w
  2. e
  3. end of word
  4. c
  5. o
  6. u
  7. l
  8. d
  9. end of word
  10. n
  11. o
  12. t
  13. end of word
  14. f
  15. i
  16. n
  17. d
  18. end of word
  19. o
  20. u
  21. r
  22. end of word
  23. n
  24. o
  25. t
  26. e
  27. b
  28. o
  29. o
  30. k
  31. s
  32. end of word
  33. .
  34. end of word
tokens
34
words
6
characters
32
UTF-8 bytes
32
  1. d
  2. e
  3. f
  4. t
  5. e
  6. r
  7. l
  8. e
  9. r
  10. i
  11. m
  12. i
  13. z
  14. i
  15. end of word
  16. b
  17. u
  18. l
  19. a
  20. m
  21. a
  22. d
  23. ı (unknown character)
  24. k
  25. end of word
  26. .
  27. end of word
tokens
27
words
2
characters
25
UTF-8 bytes
26

Up to 400 characters each. What you type stays on this page and never goes into the link.

Turkish needs fewer tokens here: 0.79× English.

A dot after a token means it ends a word: the paper’s end-of-word symbol. A chip holding only a dot is that symbol on its own, not merged yet. Dashed tokens are characters this tokenizer never saw in training. Byte-level tokenizers avoid that by starting from the 256 possible bytes.

Why Turkish often pays more, and when it does not

The training mix sets the price

A tokenizer spends its merges on whatever its training text repeats most. Train it on English and English words fuse into whole tokens while Turkish stays in pieces. Petrov et al. measured this on FLORES-200, the same 2,000 sentences in 200 languages: Turkish needs 2.43× the tokens of English with GPT-2’s tokenizer and 1.91× with cl100k_base (ChatGPT and GPT-4), but only 1.04× with the multilingual XLM-RoBERTa.

Try it above: switch the training text and watch the premium flip.

Suffixes spread the counts thin

Turkish builds words by adding suffixes to a stem. Toraman et al. give “veremedim” (“I could not give”) as ver-e-me-di-m: one word, four suffixes. Each stem appears in many surface forms, each of them rarer, so fewer whole-word merges pay off.

In this corpus, English has 155 different word forms in 286 words; Turkish has 159 in 212.

Morphemes
  1. ver
  2. e
  3. me
  4. di
  5. m
This tokenizer, now
  1. v
  2. e
  3. r
  4. e
  5. m
  6. e
  7. d
  8. i
  9. m
  10. end of word

BPE cuts where pairs are frequent, not where grammar puts a boundary, so its pieces need not match morphemes.

Language

Type a word to see how the current tokenizer splits it. The language sets the lowercasing rules: in Turkish, I becomes ı.

Turkish letters take two bytes

In UTF-8, ç, ğ, ı, ö, ş and ü take two bytes each, while the English alphabet takes one (RFC 3629). Byte-level tokenizers start from bytes, so a Turkish letter they have not merged can cost two tokens. This teaching tokenizer starts from characters, so here it costs one.

This corpus: English 1,576 characters in 1,576 bytes; Turkish 1,570 characters in 1,742 bytes.

Words are the wrong yardstick

Turkish says the same thing in fewer, longer words, so tokens per word flatters English. The fair measure is tokens for the same meaning, which Petrov et al. call the tokenization premium and the pairs above compute.

Model boundaries

A teaching model, deliberately small. It learns from 30 sentence pairs written for this room, or from up to 2,000 characters of your own text, lowercases the text, splits off punctuation and stops once no pair appears twice, so it learns a few hundred merges; production tokenizers such as cl100k_base work with vocabularies of about 100,000 tokens. The numbers here show the mechanism and the direction of the effect, not what any commercial model would count. Nothing you do here is sent to a server.

Sources

Why this room

Umut Korkmaz builds bilingual software, this site included. His open-source Orkhon language-model stack trains a byte-level BPE tokenizer and gates it before expensive training runs: the default target is at least 3.5 UTF-8 bytes per Turkish token.

Orkhon’s tokenizer fertility check on GitHub (opens in a new tab)