Calibration Is the Work

Round-to-nearest quantization (RTN) does exactly what the name says: it rounds each weight to the nearest representable value in the target precision. No data required. No passes through the model. Fast, free, and wrong for the wrong reasons.

The error is not in the rounding. The error is in the assumption that all weights are equally important.

The Assumption RTN Makes

A layer does not output its weights. It outputs Wx, where x is the activation. Two weights with identical rounding error contribute differently to the output depending on the magnitude of the activation they multiply.

RTN is blind to x. It treats every weight the same. In practice, roughly 1% of weight channels are multiplied by activations that are 10–100× larger than the rest — these are salient channels, and compressing them at the same precision as ordinary weights causes disproportionate output degradation.

This is why RTN at 4-bit often produces perplexity scores of 43 or higher on models that score 5–6 at full precision. The weights are rounded correctly. The model is wrecked.

What AWQ Does Instead

Activation-aware Weight Quantization (AWQ) runs a small number of calibration forward passes — typically 128 samples — to collect activation statistics. From those statistics it identifies the salient channels: the small subset of weights that, when perturbed, cause the largest output change.

Instead of adjusting those weights (which would require retraining), AWQ applies a per-channel scale factor s before quantization and divides by s after: the transformation is mathematically equivalent to the original weight but redistributes the quantization grid toward the values that matter most. Salient channels get more precision. Ordinary channels absorb the residual error.

The result: INT4 perplexity within 0.5–1.0 points of FP16 on standard benchmarks. RTN at the same bit-width often loses 5–10+ points.

What GPTQ Does Instead

GPTQ takes a second-order approach. It uses the Hessian of the layer's output loss with respect to each weight — a measure of sensitivity — to compensate for quantization error column by column. When it compresses a weight, it immediately adjusts all remaining uncompressed weights in that row to partially cancel the error introduced.

GPTQ is more expensive than AWQ at quantization time (Cholesky decomposition per layer), but it can recover quality that activation-scaling alone cannot, particularly in very low-bit regimes (INT3 and below).

The Engineering Consequence

Calibration data is not a nice-to-have. It is the mechanism that converts uniform rounding into output-aware compression. The 128 samples AWQ uses are not training the model — they are measuring where the model is sensitive, so the quantizer can allocate its limited precision budget accordingly.

Choosing an RTN quantized model over an AWQ or GPTQ model at 4-bit is not choosing a simpler tool. It is leaving available quality on the table for no reason. The calibration cost is minutes. The quality gap compounds every time the model encounters a salient channel — which is every forward pass.

Pick your quantization method after asking what the activations look like. RTN never does.

References

• Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. arXiv:2306.00978. https://arxiv.org/abs/2306.00978
• Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323. https://arxiv.org/abs/2210.17323
• Fujimoto, K. (2026). A practical guide to INT4 quantization for SLMs: GPTQ vs AWQ, Olive, and real-world results. Microsoft Data Science + AI Blog. https://medium.com/data-science-at-microsoft/a-practical-guide-to-int4-quantization-for-slms-gptq-vs-awq-olive-and-real-world-results-2f63d6963d1d
• Jarvis Labs. (2026). The Complete Guide to LLM Quantization with vLLM: Benchmarks & Best Practices. https://jarvislabs.ai/blog/vllm-quantization-complete-guide-benchmarks

devinfo.dev. (2026). "Calibration Is the Work." devinfo.dev:2026.0059. https://devinfo.dev/d/2026.0059