#tokenization
2 papers
-
inspiration
The Tokenizer Decides What the Model Can See
A model never sees characters or words — it sees tokens. The tokenizer is a lossy, fixed decision made before training, and it quietly shapes what the model is good and bad at: arithmetic, rare words, code, and non-English text all live or die by tokenization.
-
inspiration
The Tokenizer Is the Bug
Every LLM failure starts with the same invisible step: tokenization. It runs before inference, produces no logs, and degrades outputs silently. Most debugging sessions end at the model. They should start at the tokenizer.