When not to quantize: the small-model energy penalty
2026-06-25 · Hongping Zhang
Update, 2026-07-25: Ada now has INT8 and 7B measurements. On an RTX 4090 NF4 is already at break-even by 3B (+0.8%) and saves −28% at 7B, and INT8 was an energy penalty at all five sizes we tested (n = 1 each, bitsandbytes LLM.int8()) — see the newer note. The "INT8 on A800 only" caveat below applied to the data available when this was written.
Quantization is usually sold as a free win: cast your weights to 4-bit (NF4) or 8-bit (INT8), use a fraction of the VRAM, ship. And for memory, that story holds. For energy, it does not — at least not for small models.
Across 360+ direct GPU power measurements (NVML, 10 Hz, 256 tokens/run, 10 iterations) on four NVIDIA architectures, weight-only quantization increases decode energy for small models and only starts saving above a model-size threshold that depends on the GPU. Below that threshold, you pay more joules per token to save memory you may not have needed.
Why a 4-bit model can burn more energy
Weight-only quantization stores weights in 4/8-bit but computes in FP16: every matmul dequantizes on the fly. On small models the decode loop is not strictly memory-bound, so the bandwidth you save by reading smaller weights doesn't outweigh the extra dequantization work — net energy goes up. As models grow, weight traffic dominates, the memory savings win, and quantization flips to a net saving. The size where it flips is the crossover point.
What the measurements show
NF4 vs FP16 decode energy (negative = quantization saves):
GPU
~1–3B (small)
7B
Crossover
RTX 4090D (Ada)
+25% to +56%
not measured
none in measured range
RTX 4090 (Ada, July 2026)
+1% to +16%
−28%
just above 3B on this card (pooled Ada fit, incl. 4090D: ≈ 3.7B)
RTX 5090 (Blackwell)
+12% to +29%
−11%
≈ 4.8B
T4 (Turing)
+0% to +5%
−14%
≈ 2.1B
A800 (Ampere)
not measured
−4%
< 7B
Small models measured on consumer cards (Ada/Blackwell/Turing); 7–14B measured on A800. Values outside a GPU's measured range are fitted extrapolations, not measurements — the tool marks these explicitly. INT8 was measured on A800 and RTX 4090. The two Ada cards were measured on different stacks and disagree by roughly 20 points at the same model size, so the fitted Ada band is the widest of the four classes.
Rule of thumb: on consumer GPUs, below roughly 3–5B parameters, weight-only quantization is more likely to cost energy than save it. Above ~7B it generally saves. But the exact crossover is architecture-specific — measure, don't assume.
When quantization is still the right call
This is not "never quantize." Quantization is the right choice when:
The model would otherwise not fit in VRAM — fitting on one GPU beats spilling or adding hardware.
You're above the crossover (larger models), where it saves both memory and energy.
Latency/throughput at your batch size benefits and the small energy cost is acceptable.
The point is to make it a measured decision, not a reflex.
Find your crossover
The EcoCompute tool lets you check any GPU + model size and see whether quantization saves or costs energy, with each number labelled measured / interpolated / extrapolated. Example permalink you can share: quantenergy.tech/?tab=estimate&arch=blackwell&size=3&prec=NF4