Use Ctrl+P (or Cmd+P) to save as PDF. Back to paper
Perplexity is the first metric most practitioners reach for. It is easy to compute. It moves in the right direction when training is going well. It feels like signal.
It is not a proxy for capability.
Perplexity is the exponentiated average negative log-likelihood a model assigns to a token sequence. A lower number means the model was less surprised by the tokens it saw. That is all it measures.
It does not measure whether the model produces correct outputs. It does not measure task performance. It does not measure reasoning ability. It measures distributional surprise over training-like text.
In January 2026, Veličković et al. proved this in a rigorous, non-empirical way. Their result: if a compact decoder-only Transformer model predicts any sufficiently long sequence accurately and confidently — a necessary condition for strong generalisation — then there must exist another sequence where the model is wrong but assigns very low perplexity.
The implication is structural. A model that generalises well on some sequences is provably capable of being confidently wrong on others, with low perplexity as cover. You cannot rule this out by measuring perplexity alone.
Prior to the formal proof, the empirical picture was already clear:
Model selection. Do not choose between checkpoints or quantisation schemes using perplexity alone. A Q4 quant with lower perplexity than a Q5 quant may still perform worse on your actual task. Measure on task.
Evaluation pipelines. Perplexity is a cheap pre-screening signal, not a verdict. Use it to eliminate clearly broken runs. Do not use it to compare models that are close in capability.
Quantisation validation. This is where practitioners get burned most often. Quantisation changes weight distributions in ways that can preserve perplexity while degrading targeted capabilities — especially reasoning, multi-step arithmetic, and instruction following.
Fine-tuning ablations. Pre-training perplexity does not predict which data mixture or objective will produce the best downstream model. Running ablations solely on perplexity is measuring the wrong thing.
Perplexity tells you that training is working. It does not tell you what the model can do.
If you are comparing models, compare them on tasks. If you are validating a quantised model, test it on representative prompts. If you are running ablations, hold out an evaluation set that reflects your actual use case.
A model that surprises you less is not necessarily a model that is right more often.
1. Veličković, P., Barbero, F., Perivolaropoulos, C., Osindero, S., & Pascanu, R. (2026). Perplexity Cannot Always Tell Right from Wrong. arXiv:2601.22950. https://arxiv.org/abs/2601.22950
2. Hu, Y. et al. (2024). What is Wrong with Perplexity for Long-context Language Modeling? arXiv:2410.23771. https://arxiv.org/abs/2410.23771
3. Zeng, H. et al. (2025). Can Pre-training Indicators Reliably Predict Fine-tuning Outcomes of LLMs? arXiv:2504.12491. https://arxiv.org/abs/2504.12491
4. Wang, Y. et al. (2024). MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. arXiv:2406.01574. https://arxiv.org/abs/2406.01574