How Tokenizers Work in Language Models

Models do not read words

When you type a sentence into a large language model, the first thing that happens is not intelligence. It is chopping. The text is passed through a program called a tokenizer, which breaks it into small units called tokens. The model never processes the words you see on screen. It processes a sequence of numbers, one number per token, and everything it knows is compressed into that mapping between text and numbers. If the tokenizer is bad, the model has to work harder on every single input. If the tokenizer is good, ordinary English becomes a compact language the model can learn easily.

This matters for money and memory in the same way. Pricing for model APIs is quoted per token, so the tokenizer determines your bill. Model context is measured in tokens, so the tokenizer determines how much fits in a window. Even the model's internal arithmetic runs over token positions. The tokenizer is therefore not a minor preprocessing detail. It is one of the foundational architectural choices in a language model, and understanding how tokenizers are built explains a lot about why different models behave differently.

Why not just use words?

The obvious tokenizer splits on whitespace, one word per token. That fails fast. Even a modest English corpus contains tens of thousands of unique words, and the real problem is words you have never seen: names, misspellings, code identifiers, and rare compounds. A model that has never seen the token unforgettably cannot give it a meaningful representation, so a pure word-based vocabulary is hopelessly leaky around the edges. The solution used by essentially all modern models is subword tokenization: store common words as single tokens, but let rare words decompose into smaller pieces the model has seen before.

This decomposition also gives the model useful structure. The word unforgettably splits into familiar parts: un, forget, tabl, ly. Each piece carries real semantic weight, and a model that sees them can guess the meaning of a new word from its pieces. For languages with very rich morphology, where a single word can encode person, tense, case, and possession at once, subword splitting is not just a fallback for rare words. It is the main way the vocabulary stays manageable at all.

Byte-pair encoding: the workhorse algorithm

The dominant way to build subword vocabularies is byte-pair encoding, or BPE. It starts from a tiny vocabulary, often the individual characters of a text or, increasingly, the individual bytes so nothing can ever be unencodable. Then it repeatedly looks at all adjacent symbol pairs in the training text and merges the pair that occurs most often, turning it into a single new symbol. After the first round of merging you have two-character chunks. After many more rounds you have common words like the, ing, and http living in the vocabulary as single symbols, while rare material still decomposes into pieces.

The number of merge rounds is the knob that sets the vocabulary size. A GPT-style tokenizer with roughly 100,000 tokens has performed enough merges that the vast majority of ordinary English text compresses to about one token per 1.5 words, while still guaranteeing that every input, including text in any script or a stream of symbols, maps to some valid sequence. Training on a corpus tuned for the target use case is what makes a tokenizer good for its job. A tokenizer trained mostly on web text encodes English compactly but code and mathematics less well, which is one reason model providers have increasingly retokenized their models as their target use cases broadened.

SentencePiece and the BERT family

A second mainstream family is represented by SentencePiece, the library behind the tokenizers used by BERT and its descendants. Where classic BPE assumes you are handed pre-split text, SentencePiece was designed to learn tokenization from raw text without trusting any upstream space-splitting step. It inserts a special separator, the invisible word-piece marker, between every character so the model must learn word boundaries itself, and then runs a merge-based algorithm, either BPE or its nearest-neighbor variant unigram. The unigram mode is different in kind: instead of greedily merging frequent pairs, it holds a large candidate vocabulary and probabilistically prunes it down, keeping the split that maximizes overall text likelihood.

This design made SentencePiece the natural choice for multilingual work. Space-delimited text is a property of Latin letters, not of Chinese, Japanese, or Thai, so a tokenizer that assumes spaces is structurally biased toward European languages. By learning boundaries from raw characters, SentencePiece-based tokenizers handle all scripts on equal footing, and unigram pruning tends to produce vocabulary entries that are statistically clean. If you have ever wondered why a model handles a low-resource language surprisingly well, the tokenizer's choice is at least part of the answer.

Why tokens differ between models

There is no universal token format. Every model family ships its own vocabulary, trained on its own corpus and tuned to its own size target, so the same sentence can produce different token counts under different models. A short phrase might be nine tokens in one model and twelve in another, and that difference scales up over a long document. This is why prompt engineers measure token budgets in the specific model they are calling, and why a model upgrade that silently changes the tokenizer can shift costs for every stored prompt. Special tokens, the reserved slots for things like the start-of-sequence marker or the assistant role, also differ between model families, which is one reason model adapters have to be learned rather than translated.

Modern tokenizers also have to do a job that goes beyond text. Multimodal models reserve token positions for images, audio, and other modalities, often compressing a visual frame into a fixed number of patch tokens. As these models absorb more of the workaday interface, the token has quietly stopped meaning a piece of a word. It now means a unit of billing, context, and computation, and the tokenizer is the gatekeeper that decides what gets counted.

Sources: Hugging Face - How to train your own tokenizer; OpenAI - GPT-4 tokenizer documentation.

Corrections: Report an error at corrections@wireshaman.com.