Continuous Batching Is Why Your Server Is Fast

A GPU serving an LLM is almost never compute-limited by a single request. It is limited by how well you keep it busy. Static batching — collect N requests, run them together, return them together — sounds efficient but wastes most of that opportunity.

The problem with static batches

Requests in a static batch finish at different times, but the batch does not release until the longest one is done. A 20-token reply waits on a 500-token reply sharing its batch. New requests wait for the whole batch to clear before they can start. The GPU sits partly idle, holding finished sequences in memory and doing nothing useful for them.

Token-level scheduling

Continuous batching (also called in-flight batching) schedules at the granularity of a single token step, not a whole request. After every forward pass, finished sequences leave and waiting requests join — immediately, mid-flight. The batch is reassembled each step to stay as full as the hardware allows. Introduced in Orca and popularized by vLLM, this is the main reason a modern server delivers many times the throughput of a naive one on the same GPU.

Memory is the real constraint

Keeping many sequences in flight means keeping many KV caches resident. vLLM's PagedAttention manages that memory in fixed pages, the way an operating system manages RAM, so fragmentation does not strand capacity. More usable KV memory means a larger in-flight batch, which means more throughput.

The trade-off to name

Continuous batching optimizes throughput, not single-request latency. Under heavy load an individual request may see more per-token delay because it shares each step with more sequences. For a self-hosted server with bursty, mixed-length traffic that trade is almost always worth it — you are turning idle silicon into completed requests.

References

1. Yu, G., et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022. https://www.usenix.org/conference/osdi22/presentation/yu

2. Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. arXiv:2309.06180. https://arxiv.org/abs/2309.06180

devinfo.dev. (2026). "Continuous Batching Is Why Your Server Is Fast." devinfo.dev:2026.0068. https://devinfo.dev/d/2026.0068