When not to quantize: the small-model energy penalty
2026-06-25 · Hongping Zhang
Update, 2026-07-25: Ada now has INT8 and 7B measurements. On an RTX 4090 NF4 is already at break-even by 3B (+0.8%) and saves −28% at 7B, and INT8 was an energy penalty at all five sizes we tested (n = 1 each, bitsandbytes LLM.int8()) — see the newer note. The "INT8 on A800 only" caveat below applied to the data available when this was written.
Quantization is usually sold as a free win: cast your weights to 4-bit (NF4) or 8-bit (INT8), use a fraction of the VRAM, ship. And for memory, that story holds. For energy, it does not — at least not for small models.
Across 360+ direct GPU power measurements (NVML, 10 Hz, 256 tokens/run, 10 iterations) on four NVIDIA architectures, weight-only quantization increases decode energy for small models and only starts saving above a model-size threshold that depends on the GPU. Below that threshold, you pay more joules per token to save memory you may not have needed.
Why a 4-bit model can burn more energy
Weight-only quantization stores weights in 4/8-bit but computes in FP16: every matmul dequantizes on the fly. On small models the decode loop is not strictly memory-bound, so the bandwidth you save by reading smaller weights doesn't outweigh the extra dequantization work — net energy goes up. As models grow, weight traffic dominates, the memory savings win, and quantization flips to a net saving. The size where it flips is the crossover point.
What the measurements show
NF4 vs FP16 decode energy (negative = quantization saves):
GPU
~1–3B (small)
7B
Crossover
RTX 4090D (Ada)
+25% to +56%
not measured
none in measured range
RTX 4090 (Ada, July 2026)
+1% to +16%
−28%
just above 3B on this card (pooled Ada fit, incl. 4090D: ≈ 3.7B)
Small models measured on consumer cards (Ada/Blackwell/Turing); 7–14B measured on A800. Values outside a GPU's measured range are fitted extrapolations, not measurements — the tool marks these explicitly. INT8 was measured on A800 and RTX 4090, and — outside the fitted curves — on RTX 5090 in the September 2026 session, where it is positive at every size (+55% to +259%). The two Ada cards were measured on different stacks and disagree by roughly 20 points at the same model size, so the fitted Ada band is the widest of the four classes.
* The 5090 row is stack-dependent, and we have since proved it on that card. Re-measuring the
same RTX 5090 in September 2026 with bitsandbytes 0.50.2 put NF4 at
+0.6%/+1.9% at 1.1–1.5B and −32% at 7B — a crossover near 1.8B rather than 4.8B, with no
hardware change. And the shift is not measurement noise: within the September stack, the A/B restart repeat
agrees to 1.1 pp (NF4 average), while between stacks the same four cells sit
23.3 pp apart on average — a factor of ~20. The crossover moved because the stack changed,
not because the measurement wobbled. Read every crossover in this table as “this card with the libraries
of that session”; the numbers above are the stack the fitted curves are built from.
Rule of thumb: on consumer GPUs, below roughly 3–5B parameters, weight-only quantization is more likely to cost energy than save it. Above ~7B it generally saves. But the exact crossover is architecture-specific — measure, don't assume.
When quantization is still the right call
This is not "never quantize." Quantization is the right choice when:
The model would otherwise not fit in VRAM — fitting on one GPU beats spilling or adding hardware.
You're above the crossover (larger models), where it saves both memory and energy.
Latency/throughput at your batch size benefits and the small energy cost is acceptable.
The point is to make it a measured decision, not a reflex.
Find your crossover
The EcoCompute tool lets you check any GPU + model size and see whether quantization saves or costs energy, with each number labelled measured / interpolated / extrapolated. Example permalink you can share: quantenergy.tech/?tab=estimate&arch=blackwell&size=3&prec=NF4