Latency and Throughput Are Not the Same Goal

Two systems can serve the same model and feel completely different, because they optimize different things. Latency is how long one request takes end to end. Throughput is how many requests the system finishes per second across everyone. They pull in opposite directions.

Batching is the clearest example. Grouping many requests into one forward pass raises throughput, because the accelerator does more useful work per cycle, but any single request now waits for the batch to form and for the slowest member to finish. You bought aggregate speed with individual delay.

The mistake is optimizing one number and reporting the other. A benchmark that shows high tokens per second can hide a p99 latency that makes the product feel sluggish. A demo that feels instant for one user can collapse under concurrency because nothing was batched.

Decide which you are serving. An interactive assistant lives and dies on tail latency; a nightly enrichment job only cares about total work done by morning. Continuous batching narrows the gap, but it does not erase the trade. Measure both, name the one you are optimizing, and hold the other as a stated constraint.