Quantization Is a Memory-Bandwidth Decision

Ask why a 4-bit model runs faster than the same model at 16-bit and the common answer is "less memory." True, but incomplete. The deeper reason is bandwidth.

Decoding is memory-bound

Autoregressive generation produces one token at a time. To produce each token, the GPU must read every weight in the model out of high-bandwidth memory and into the compute units — once per token. At batch size one, the arithmetic per weight is tiny (a single multiply-accumulate), so the chip spends most of its time waiting on memory, not computing.

That gives a useful ceiling:

tokens/sec ≈ memory_bandwidth / bytes_read_per_token

Bytes read per token is dominated by the model's weight footprint. Halve the bytes and you roughly double the token rate. This is why FP16 to INT8 to INT4 tracks almost linearly with decode speed on a single stream, even though the math being done is nearly identical.

Prefill is different

Processing the prompt (prefill) is compute-bound. It multiplies large matrices with high arithmetic intensity, so the chip is busy, not waiting. Quantization helps prefill far less than it helps decode. If your workload is long prompts and short answers, quantization buys you less than the headline number suggests.

The accuracy cost is real but bounded

Modern methods are not naive rounding. GPTQ solves a layer-wise reconstruction to place the quantization error where it matters least. AWQ protects the small fraction of weight channels that carry the most signal. At 4-bit, both stay close to lossless on many models. Below 4-bit, quality falls off quickly and the calibration data starts to matter a lot.

What this means for self-hosting

At batch size one — the self-hosted default — you are memory-bound. So 4-bit quantization is usually the highest-leverage single change you can make: it fits a larger model in the same card and speeds up generation at the same time.

The trade-off to name honestly: batched, high-throughput serving is compute-bound, so it benefits less. And quantization is lossy — validate on your own evals before trusting the smaller model. But for one person running a capable model on one GPU, the decision is easy, and it is a bandwidth decision.

References

1. Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. arXiv:2208.07339. https://arxiv.org/abs/2208.07339

2. Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323. https://arxiv.org/abs/2210.17323

3. Lin, J., et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. arXiv:2306.00978. https://arxiv.org/abs/2306.00978

devinfo.dev. (2026). "Quantization Is a Memory-Bandwidth Decision." devinfo.dev:2026.0065. https://devinfo.dev/d/2026.0065