#benchmarks
2 papers
-
inspiration
Perplexity Is Not a Proxy
A model can assign low perplexity to a sequence it gets wrong. This is not an edge case — it is a theorem. Perplexity measures how surprised a model is by token sequences. It does not measure whether the model is right.
-
whitepaper
Evals Are Not Optional
Benchmark scores are not evaluations. Contamination is widespread, Goodhart's Law is in effect, and the gap between a leaderboard number and production behaviour is unbridged without a real eval pipeline. This paper defines what evals are, why the major benchmarks are unreliable in isolation, and how to build an evaluation practice that actually catches failures.