#llm-serving
12 papers
-
inspiration
Continuous Batching Is Why Your Server Is Fast
Static batching wastes the GPU: every request waits for the slowest one in its batch to finish. Continuous batching — the idea behind Orca and vLLM — schedules at the token level instead of the request level, and it is the single biggest reason modern serving throughput is what it is.
-
inspiration
Speculative Decoding Is a Bet on the Draft
Speculative decoding makes a large model generate faster by letting a small model guess ahead. It is lossless — the output is identical to decoding from the large model alone. But the entire speedup is a function of how often the draft is right, which makes the technique only as good as the match between your draft and target models.
-
inspiration
Quantization Is a Memory-Bandwidth Decision
Dropping a model from FP16 to INT4 is usually framed as a way to fit it in less VRAM. That is the smaller half of the story. When you serve a single stream, token generation is bound by memory bandwidth, not arithmetic — every token reads the entire model from memory once. Quantization shrinks that read, so it buys throughput, not just capacity.
-
inspiration
The Adapter Is Not the Model
The obvious way to serve ten fine-tuned variants is to run ten models. That is wrong. A LoRA adapter is a thin correction on top of a base model — and the base model is the same for all of them. Merging the adapter back into the weights before serving discards the one fact that makes multi-tenant fine-tuning cheap.
-
inspiration
The Draft Model Does the Work
Speculative decoding uses a small draft model to propose tokens and a large model to verify them in parallel. The large model runs once per batch, not once per token. That single change converts a sequential bottleneck into a parallel verification step — and delivers 2–3x latency reduction at zero quality cost.
-
inspiration
The Scheduler Is Not the Model
You can swap in a faster model and get slower responses. The model is not the bottleneck — the scheduler is. How the serving system decides what to run, in what order, and in what chunks determines your latency and throughput.
-
inspiration
Prefill Is the Stall
The gap between submitting a prompt and receiving the first token is not network lag. It is compute. Prefill is a matrix multiplication over every token in your input — and it blocks decode entirely until it finishes.
-
inspiration
Sparse Is Not Small
A model with 671 billion parameters can cost less to run than a 70 billion dense model. That is not a marketing claim — it is arithmetic. Mixture of Experts replaces a full forward pass with a routing decision, and the routing decision is the cost model.
-
inspiration
The Router Is the System
Routing between models is not a configuration detail. It is a measurable, trainable system boundary — and treating it as one cuts inference costs by 40–85% without sacrificing quality.
-
whitepaper
The While Loop Is the Easy Part: Engineering Agents for Production
Every LLM agent converges on the same structure: call the model, execute tools, repeat. That loop is not where the engineering lives. The hard parts are termination conditions, context budget management, error classification, tool safety rails, and observability infrastructure — and most agents that fail in production fail there, not in the model.
-
inspiration
Parallelism Is a Topology Decision
Tensor parallelism and pipeline parallelism are not interchangeable scaling knobs. They encode different assumptions about your hardware, your model shape, and what you are optimizing for. Choosing wrong does not just waste GPUs — it locks in a latency-throughput tradeoff you did not knowingly make.
-
inspiration
The KV Cache Is Your Real Memory Budget
The KV cache — not the model weights — is what limits how many tokens you can generate and how many requests you can serve. Understanding it changes how you provision hardware and tune inference.