Prompt Caching Is the Cheapest Speedup

The fastest token is the one you never recompute. Prompt caching is the most underused speedup in LLM serving, because it looks like an optimization and is really just not repeating work.

Most real prompts share a large, stable prefix: a system prompt, tool definitions, a few-shot preamble, a long document. On every call the model recomputes attention over that prefix from scratch, even though it has not changed. Prompt caching stores the computed key and value state for that prefix and reuses it, so only the new suffix is processed.

The wins are concrete. Time to first token drops, because the expensive prefill over the shared prefix is skipped. Cost drops, because many providers bill cached input tokens at a fraction of the normal rate. And the effect compounds exactly where it hurts most: long system prompts and big retrieved contexts, called repeatedly.

There is a discipline to it. Put the stable content at the front and the variable content at the back, so the cacheable prefix is as long as possible. Do not reshuffle the preamble between calls, or you invalidate the cache every time. Order is now a performance decision.

You do not always need a faster model. Often you just need to stop paying twice for the same prefix.