Everything we have published so far measures one axis: joules. That is a defensible thing to measure and an indefensible thing to give advice from. "NF4 saves 39% energy at 7B" is only a recommendation if the quantized model is still the model you wanted. So the container now scores perplexity in the same run as the energy, and this note is the first dataset where both numbers for a configuration come from the same process, the same weights and the same session.
Ten measured configurations on one rented RTX 4090 (Ada, 24 GB), five model sizes ×
NF4/INT8, each with its own FP16 baseline measured in the same run. Every row is direct NVML,
basis: measured, n = 1. Archived at
DOI 10.5281/zenodo.22037483
(CC BY 4.0, raw energy.json included).
| Model | N | NF4 Δenergy | NF4 Δppl | INT8 Δenergy | INT8 Δppl |
|---|---|---|---|---|---|
| Qwen2-0.5B | 0.5B | +31.1% | +9.45% | +594.6% | +0.52% |
| TinyLlama-1.1B | 1.1B | +5.5% | +5.01% | +301.6% | +0.54% |
| Qwen2-1.5B | 1.5B | −2.2% | +6.93% | +365.1% | +0.51% |
| Qwen2.5-3B | 3.0B | −15.1% | +27.56% | +273.2% | +3.79% |
| Qwen2-7B | 7.0B | −39.0% | +12.06% | +105.8% | +1.19% |
Read the energy column alone and you would recommend 3B NF4: it saves 15% of the joules. Read the quality column and it is the worst configuration in the table — perplexity rises 27.6%. Conversely INT8 is nearly free on quality (about +0.5% at three of five sizes) and catastrophic on energy: it never once saved a joule, and at 0.5B it cost almost seven times the FP16 energy, because decode throughput collapses to 6–14 tok/s while the package still draws 85–100 W.
Apply the obvious filter — "recommend nothing that degrades the model by more than 1% perplexity" — and this session rejects every NF4 configuration and admits every INT8 configuration except 3B. Which is the exact opposite of the energy ranking. Two honest measurements, two opposite recommendations.
We measured the same card model in July. Comparing the two sessions, the INT8 energy penalty is 2.0–2.5× larger in August at every one of the five sizes (INT8 decode throughput 0.62–0.65× July's), while the FP16 baselines of the two sessions agree to within −10%…+19% and every pinned dependency matched (bitsandbytes 0.43.3, transformers 4.57.6).
| INT8 Δenergy | 0.5B | 1.1B | 1.5B | 3B | 7B |
|---|---|---|---|---|---|
| July 2026 | +241.9% | +146.1% | +180.7% | +134.8% | +49.5% |
| August 2026 | +594.6% | +301.6% | +365.1% | +273.2% | +105.8% |
It is not noise, and we now have the number to say so. We re-ran all five INT8 configurations three times each the next day on the same instance (rtx4090_int8_repeats_2026-08-20.csv): the coefficient of variation of Δenergy is 0.6–3.9%, and the n=3 means land on the single trials above. Run-to-run noise is 30–50× smaller than the 100–140 point gap with July.
What has to be explained, then, is a property of the session. The decomposition: INT8 energy per
token rose 1.41–1.58× with unchanged package power (74–91 W against July's ~76 W)
— so it is entirely a throughput effect — and at the small sizes the August FP16 baseline was also
lower (0.5B: 0.78×), which is what stretches that size's ratio to 2.0×. The leading hypothesis is
the LLM.int8() kernel path, which changes with the torch build: July's anchors were
recorded with torch 2.4.1+cu121 (Python 3.8), the native path used in August got
torch 2.5.1+cu121, and the container image pins torch 2.13.0 — three
different builds, no two sessions alike. That hypothesis is untested. Testing it costs one
1.1B INT8 run pinned back to the July build, or one run of the image on any Docker-capable RTX 4090
(the rented instance cannot nest Docker) — back near +146% confirms the kernel explanation,
still near +300% moves suspicion to the instance.
We are stating it rather than quietly picking the nicer session, because it has a consequence for everyone publishing numbers like ours: a quantization energy penalty measured once, on one rented instance, can be off by a factor of two — and almost nobody reports that layer of variance. What reproduced across our two sessions is the shape: the NF4 penalty falls monotonically with model size and crosses into savings (near 1.4B in August, near 3.2B in July); INT8 never saves. The magnitudes did not. So the two sessions are archived as separate files, must not be pooled, and the site's fitted curves still use only the July session — mixing them would make the published error bars describe a mixture of two backends rather than an estimate of either.
LLM.int8(). AWQ,
GPTQ and TensorRT-LLM have different kernels and may behave differently — an INT8 result here is
a result about this INT8 implementation.Every number above comes out of the same open container you can run in one command; the quality probe is on by default and adds a few seconds:
docker run --rm --gpus all -v "$PWD/out:/out" \ ghcr.io/hongping-zh/ecocompute-mlcube:latest \ --model Qwen/Qwen2-7B --precision NF4 --output_dir /out
The report it writes (energy.json, schema
ecocompute-energy/1.1) carries the energy, the FP16 baseline, both perplexities and
the full version stack, and you can drop it on
the replications page to see it plotted against our curve — including
when it disagrees. A run that contradicts this note is more useful to us than one that agrees.