EcoCompute · Notes

When not to quantize: the small-model energy penalty

2026-06-25 · Hongping Zhang

Update, 2026-07-25: Ada now has INT8 and 7B measurements. On an RTX 4090 the NF4 crossover lands between 3B and 7B (−29% at 7B), and INT8 turned out to be an energy penalty at every size — see the newer note. The "INT8 on A800 only" caveat below applied to the data available when this was written.

Quantization is usually sold as a free win: cast your weights to 4-bit (NF4) or 8-bit (INT8), use a fraction of the VRAM, ship. And for memory, that story holds. For energy, it does not — at least not for small models.

Across 360+ direct GPU power measurements (NVML, 10 Hz, 256 tokens/run, 10 iterations) on four NVIDIA architectures, weight-only quantization increases decode energy for small models and only starts saving above a model-size threshold that depends on the GPU. Below that threshold, you pay more joules per token to save memory you may not have needed.

Why a 4-bit model can burn more energy

Weight-only quantization stores weights in 4/8-bit but computes in FP16: every matmul dequantizes on the fly. On small models the decode loop is not strictly memory-bound, so the bandwidth you save by reading smaller weights doesn't outweigh the extra dequantization work — net energy goes up. As models grow, weight traffic dominates, the memory savings win, and quantization flips to a net saving. The size where it flips is the crossover point.

What the measurements show

NF4 vs FP16 decode energy (negative = quantization saves):

GPU	~1–3B (small)	7B	Crossover
RTX 4090D (Ada)	+25% to +56%	not measured	none in measured range
RTX 5090 (Blackwell)	+12% to +29%	−11%	≈ 4.8B
T4 (Turing)	+0% to +5%	−14%	≈ 2.1B
A800 (Ampere)	not measured	−4%	< 7B

Small models measured on consumer cards (Ada/Blackwell/Turing); 7–14B measured on A800. Values outside a GPU's measured range are fitted extrapolations, not measurements — the tool marks these explicitly. INT8 was measured on A800 only.

Rule of thumb: on consumer GPUs, below roughly 3–5B parameters, weight-only quantization is more likely to cost energy than save it. Above ~7B it generally saves. But the exact crossover is architecture-specific — measure, don't assume.

When quantization is still the right call

This is not "never quantize." Quantization is the right choice when:

The model would otherwise not fit in VRAM — fitting on one GPU beats spilling or adding hardware.
You're above the crossover (larger models), where it saves both memory and energy.
Latency/throughput at your batch size benefits and the small energy cost is acceptable.

The point is to make it a measured decision, not a reflex.

Find your crossover

The EcoCompute tool lets you check any GPU + model size and see whether quantization saves or costs energy, with each number labelled measured / interpolated / extrapolated. Example permalink you can share: quantenergy.tech/?tab=estimate&arch=blackwell&size=3&prec=NF4