Perplexity Is Not a Proxy

Perplexity is the first metric most practitioners reach for. It is easy to compute. It moves in the right direction when training is going well. It feels like signal.

It is not a proxy for capability.

What Perplexity Measures

Perplexity is the exponentiated average negative log-likelihood a model assigns to a token sequence. A lower number means the model was less surprised by the tokens it saw. That is all it measures.

It does not measure whether the model produces correct outputs. It does not measure task performance. It does not measure reasoning ability. It measures distributional surprise over training-like text.

The Formal Problem

In January 2026, Veličković et al. proved this in a rigorous, non-empirical way. Their result: if a compact decoder-only Transformer model predicts any sufficiently long sequence accurately and confidently — a necessary condition for strong generalisation — then there must exist another sequence where the model is wrong but assigns very low perplexity.

The implication is structural. A model that generalises well on some sequences is provably capable of being confidently wrong on others, with low perplexity as cover. You cannot rule this out by measuring perplexity alone.

The Empirical Evidence Agrees

Prior to the formal proof, the empirical picture was already clear:

• Xu et al. (2024) and Luden et al. (2024) showed perplexity does not reliably predict downstream task performance.
• On long-context benchmarks specifically, perplexity degrades as a signal as input length grows — models can use local token statistics to achieve low perplexity without any genuine long-text understanding.
• At fixed model scale, pre-training perplexity correlates poorly with fine-tuning outcomes. A 2025 study across 50 1B-parameter variants found that conventional perplexity is a weak predictor of which model performs better after supervised fine-tuning.

What This Means for Practice

Model selection. Do not choose between checkpoints or quantisation schemes using perplexity alone. A Q4 quant with lower perplexity than a Q5 quant may still perform worse on your actual task. Measure on task.

Evaluation pipelines. Perplexity is a cheap pre-screening signal, not a verdict. Use it to eliminate clearly broken runs. Do not use it to compare models that are close in capability.

Quantisation validation. This is where practitioners get burned most often. Quantisation changes weight distributions in ways that can preserve perplexity while degrading targeted capabilities — especially reasoning, multi-step arithmetic, and instruction following.

Fine-tuning ablations. Pre-training perplexity does not predict which data mixture or objective will produce the best downstream model. Running ablations solely on perplexity is measuring the wrong thing.

The One Rule

Perplexity tells you that training is working. It does not tell you what the model can do.

If you are comparing models, compare them on tasks. If you are validating a quantised model, test it on representative prompts. If you are running ablations, hold out an evaluation set that reflects your actual use case.

A model that surprises you less is not necessarily a model that is right more often.

References

1. Veličković, P., Barbero, F., Perivolaropoulos, C., Osindero, S., & Pascanu, R. (2026). Perplexity Cannot Always Tell Right from Wrong. arXiv:2601.22950. https://arxiv.org/abs/2601.22950

2. Hu, Y. et al. (2024). What is Wrong with Perplexity for Long-context Language Modeling? arXiv:2410.23771. https://arxiv.org/abs/2410.23771

3. Zeng, H. et al. (2025). Can Pre-training Indicators Reliably Predict Fine-tuning Outcomes of LLMs? arXiv:2504.12491. https://arxiv.org/abs/2504.12491

4. Wang, Y. et al. (2024). MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. arXiv:2406.01574. https://arxiv.org/abs/2406.01574

devinfo.dev. (2026). "Perplexity Is Not a Proxy." devinfo.dev:2026.0061. https://devinfo.dev/d/2026.0061