The Tokenizer Decides What the Model Can See

A language model never sees your text. It sees a sequence of integer tokens produced by a tokenizer — a fixed, lossy mapping decided before training ever starts. That upstream decision quietly determines what the model finds easy and what it finds nearly impossible.

Subword tokenization

Modern tokenizers use subword schemes like byte-pair encoding: frequent strings become single tokens, rare ones fragment into many. "the" is one token; an unusual surname or a long chemical name might be five or ten. The model spends its fixed context and attention budget on those fragments, and it has fewer, noisier examples of each to learn from.

Where it shows up

• Arithmetic: numbers split inconsistently ("1234" may be one token, "1235" two), so digit-level reasoning fights the tokenization. This is a real source of math errors.
• Rare and non-English words: languages underrepresented in the tokenizer's corpus fragment more, inflating token counts and cost and lowering quality — a measurable multilingual tax.
• Code: whitespace, indentation, and symbols tokenize in ways that can make structurally identical code look different to the model.

Why it matters in practice

Token count is cost and latency, and the tokenizer sets it. Two models with similar parameter counts can have very different effective context lengths for your text depending on how efficiently their tokenizers encode it. And a class of "the model is dumb at X" complaints — counting letters, reversing strings, exact arithmetic — trace straight back to the fact that the model cannot see the characters you can.

The takeaway

Before blaming the model, look at the tokens. The tokenizer decided what the model was allowed to perceive.

References

1. Sennrich, R., Haddow, B., & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. ACL 2016. arXiv:1508.07909. https://arxiv.org/abs/1508.07909

2. Kudo, T., & Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer. EMNLP 2018. arXiv:1808.06226. https://arxiv.org/abs/1808.06226