EcoCompute · Notes

We ran our own container on a rented RTX 4090: NF4 breaks even just above 3B, INT8 lost at every size we tested

2026-07-25 · Hongping Zhang

Publishing a measurement methodology is cheap; running it is not. So we rented a single NVIDIA GeForce RTX 4090 (Ada Lovelace, 24 GB) for an afternoon and pointed our own container at it — the EcoCompute energy MLCube, the same MLCube-compatible container anyone can run on their own card. Five models from 0.5B to 7B, three precisions each: 15 of 15 configurations came back as real hardware measurements (basis: "measured", measurement_source: "direct-nvml"), not estimates.

Two things came out of it that our earlier Ada data could not show: INT8 on Ada (previously we only had INT8 on A800) and 7B models (the earlier Ada anchor stopped at 3B).

What we measured

Decode-phase energy, integrated from direct NVML power samples at 10 Hz over the generation loop: 256 tokens per run, 10 iterations after 2 warmups, batch 1, greedy decoding from one fixed prompt.

Model	FP16 (mJ/token)	NF4 (mJ/token)	NF4 vs FP16	INT8 (mJ/token)	INT8 vs FP16
Qwen2-0.5B	1550.9	2103.6	+35.6%	5302.5	+241.9%
TinyLlama-1.1B	1619.7	1883.9	+16.3%	3986.2	+146.1%
Qwen2-1.5B	2200.7	2533.3	+15.1%	6177.9	+180.7%
Qwen2.5-3B	3558.0	3584.9	+0.8%	8354.9	+134.8%
Qwen2-7B	5467.9	3911.8	−28.5%	8176.8	+49.5%

Positive = quantization costs more energy than FP16. Every cell is a measurement on the same card in the same session. Each Δ is recomputed here from the two energy columns next to it, so you can check every percentage with a calculator.

Correction, 2026-08-11: this table originally quoted the vs_fp16_pct column of rtx4090_results.csv (Zenodo 10.5281/zenodo.22037483), which does not match that file's own energy, power and throughput columns — those three agree with each other to within 0.7 points everywhere, while the percentage column drifts by up to 14. The worst case was Qwen2.5-3B NF4: the column said +10.1%, the energies say +0.8%. The Δ column above (and every chart on this site, which is fitted from measured.csv) is now derived from the energies. The energy measurements themselves never changed; a corrected version of the Zenodo record is pending.

NF4: the crossover is real, and on this card it sits just above 3B

The NF4 penalty shrinks with model size — +35.6% at 0.5B, +16.3% and +15.1% at 1.1B and 1.5B — is already gone at 3B (+0.8%, i.e. break-even), and flips to a −28.5% saving at 7B. Interpolating between the 3B and 7B points puts break-even at roughly 3.1B on this card. The pooled Ada fit on the front page still lands near 3.7B because it also carries the RTX 4090D anchors, which are consistently more penalised at the same size — that gap between two cards of the same architecture is itself one of the honest limitations of a per-architecture curve.

INT8: lower power, more energy

INT8 (bitsandbytes LLM.int8()) was a penalty at all five sizes we measured, from +47% at 7B to +256% at 0.5B — one card, one backend, n = 1 per size, so read it as "this kernel on this card" rather than a verdict on 8-bit inference. The mechanism is visible in the throughput panel below: INT8 genuinely draws less instantaneous power (73–95 W vs 88–269 W for FP16), but decode throughput collapses to 9–19 tok/s against FP16's 39–62 tok/s. Energy is power × time, so a run that takes 3–5× longer loses even when it draws less power.

Two panels for RTX 4090: absolute decode energy per token, where INT8 is highest at every model size, and decode throughput, where INT8 collapses to 9–19 tokens per second. — Left: energy per token. Right: decode throughput. INT8 loses on energy precisely because it loses on time.

Practical read: on a 24 GB Ada card, use NF4 when the model is large enough (break-even is just above 3B here, and 7B saves clearly) or when it simply would not fit otherwise — but do not reach for weight-only INT8 at batch 1 expecting an energy win. On this hardware, with this backend, there wasn't one. Other kernels (GPTQ, AWQ, TensorRT-LLM) are a different measurement we have not made.

What this run is not

Being explicit about scope is the point of the project, so: each configuration was run once (n = 1), with the 10 decode iterations integrated into a single energy total — there is no standard deviation or CV to report yet. We have since re-measured one of these cells — TinyLlama-1.1B INT8 — on a second rented RTX 4090 in August 2026: +137.9% against July's +146.1%, i.e. two single trials 8.2 points apart (different session, different bitsandbytes version). That is the resolution these numbers deserve to be read at. NVML gives GPU-package power, not whole-system wall power. The workload is one short prompt at a fixed 256-token length and batch 1, so it speaks to single-stream decode, not to batched serving. And the card is an RTX 4090, not an RTX 4090D — both are Ada, but they are different SKUs, which is why the site now lists them as separate columns instead of overwriting the older one. We publish this as a supplementary case study, not as a certified benchmark.

Run it yourself, or take the data

The container does the same thing on your GPU and writes an energy.json with the exact fields the charts on this site read:

git clone https://github.com/hongping-zh/ecocompute-mlcube.git
cd ecocompute-mlcube

python3 entrypoint.py energy_estimate \
    --model Qwen/Qwen2-7B --precision NF4 \
    --gpu_arch ada --params_b 7 \
    --output_dir workspace/outputs --share

cat workspace/outputs/energy.json

No GPU? It still runs and returns dataset-derived reference values, clearly flagged — we never dress an estimate up as a measurement. The complete run above (raw per-configuration reports, aggregated CSV, environment metadata, figures) is archived on Zenodo under 10.5281/zenodo.22037483, CC BY 4.0, and the numbers are live in the tool under GPU → RTX 4090 · Ada · 24GB. Cite that DOI for this run specifically; for the broader benchmark behind the rest of the site, cite the main dataset v1.1.0, 10.5281/zenodo.19647290 (360+ configurations, four architectures).