#quantization
4 papers
-
inspiration
Quantization Is a Memory-Bandwidth Decision
Dropping a model from FP16 to INT4 is usually framed as a way to fit it in less VRAM. That is the smaller half of the story. When you serve a single stream, token generation is bound by memory bandwidth, not arithmetic — every token reads the entire model from memory once. Quantization shrinks that read, so it buys throughput, not just capacity.
-
inspiration
Calibration Is the Work
Round-to-nearest quantization distributes precision evenly across all weights. That is the wrong allocation. A tiny fraction of weight channels — the ones multiplied by large activations — dominate model output. AWQ finds those channels first. RTN never looks.
-
whitepaper
The Memory Wall: A Field Guide to LLM Inference on Consumer Hardware
LLM inference is not compute-bound. It is memory-bandwidth-bound. Understanding that single fact — and the arithmetic that follows from it — determines every sensible hardware and quantization decision you will make when running models on consumer devices.
-
inspiration
Quantization Is a Design Decision
Quantization is not just compression. It is a tradeoff you are making about accuracy, speed, and memory — and it belongs in your architecture docs, not your deployment scripts.