EcoCompute · Notes

How repeatable is a quantization energy number?

2026-08-21 · Hongping Zhang

Last note ended on an uncomfortable result: the same INT8 configurations, on the same GPU model, with the same pinned dependencies, measured 2.0–2.5× worse in August than in July. We had no way to say whether that was two unlucky draws from a noisy measurement or a real property of the sessions, because every configuration had been measured exactly once.

So we measured the noise. Five INT8 configurations, three independent runs each, same rented RTX 4090, same day, same stack — the first points on this site that carry a real coefficient of variation. New version of the dataset: DOI 10.5281/zenodo.22037483 (CC BY 4.0; 15 raw energy.json reports and their logs included).

The noise

Model	N	Δenergy vs FP16, mean of 3	SD	CV	CV of raw J/1k tok	CV of the FP16 baseline
Qwen2-0.5B	0.5B	+581.3%	12.5	2.15%	0.78%	2.45%
TinyLlama-1.1B	1.1B	+307.1%	1.8	0.57%	2.01%	1.97%
Qwen2-1.5B	1.5B	+347.4%	7.6	2.19%	0.30%	1.41%
Qwen2.5-3B	3.0B	+271.2%	7.4	2.72%	0.50%	1.51%
Qwen2-7B	7.0B	+105.9%	4.1	3.87%	1.93%	0.12%

n = 3 independent runs per row; SD is the sample standard deviation (ddof = 1) of the three, CV = SD/mean. Direct NVML at 10 Hz, batch 1, 256 generated tokens, 2 warmup + 10 decode iterations per run, bitsandbytes LLM.int8(). Throughput CV 0.0–2.6%, average-power CV 0.3–0.8%. Raw rows: rtx4090_int8_repeats_2026-08-20.csv, aggregates in the .summary.csv beside it.

The measurement itself is tight: 0.6–3.9% CV on the delta, 0.3–2.0% on the raw joules. And the single trials we published the day before were not flukes — every n=3 mean lands on them, the largest deviation being 1.5B at −17.7 points, about 2.3 SD.

What that buys us: the 2× gap is not sampling

Put the two numbers next to each other. Run-to-run noise on the INT8 delta is a few percent. The gap between our July and August sessions is 100–140 percentage points — for example TinyLlama-1.1B at +146.1% against +307.1%. That is 30–50× the noise. Before this note, "the sessions really differ" was an argument from the tidiness of the ratios; now it is a measurement.

Which promotes the discrepancy from an anomaly to a constraint that any explanation has to fit:

Power is not the mechanism. INT8 energy per token rose 1.41–1.58× at essentially unchanged GPU package power (74–91 W against July's ~76 W). The whole effect is throughput, which fell to 0.63–0.72×.
At the small sizes the FP16 baseline moved too (0.5B: 0.78× of July's), which is what stretches that size's ratio all the way to 2.0× — the INT8 side alone does not explain it.
It is not a scaling bug. Every report is internally consistent: total_energy / tokens agrees with avg_power / throughput (0.5B: 21043 J / 2560 tok = 8.22 J/tok versus 74.1 W / 9.0 tok/s = 8.23 J/tok), and the schema and quality probe both pass.

The leading hypothesis is still the LLM.int8() kernel path, which changes with the torch build — and no two of these sessions ran the same build. July's anchors were recorded with torch 2.4.1+cu121 on Python 3.8; the native path we had to use in August got torch 2.5.1+cu121; the container image pins torch 2.13.0. It is a hypothesis, not an attribution. The cheapest control is one 1.1B INT8 run pinned back to the July build on the August instance; the other is one run of the image on any Docker-capable RTX 4090 (that rented instance cannot nest Docker). Back near +146% and the kernel path explains it, still near +300% and suspicion moves to the instance. If you have such a card, that single run is the most useful contribution anyone could make to this dataset right now.

The finding we did not expect: absolute joules drift overnight

Between 2026-08-19 and 2026-08-20, on the same host, absolute energy per token came out 12–17% lower — for INT8 and FP16 alike — while the FP16-normalised delta barely moved. We do not know what changed on the machine, and that is rather the point: on rented infrastructure you do not get to know.

Practical consequence. An absolute figure like "8250 J per 1000 tokens" from a single session is worth much less than it looks. A delta measured against an FP16 baseline from the same run survives a 15% day-to-day shift in the machine; the joule count does not. This is why every chart on this site is a percentage against FP16, and why the container always measures its own baseline instead of reusing a published one.

What this does not show

n = 3 is a small sample, and all three runs sat on one rented instance within one day. It bounds run-to-run noise on that host; it says nothing about machine-to-machine variance — which the July/August gap suggests is the dominant term.
The quality axis has CV = 0 by construction. Perplexity came out bit-identical in all three replicates (7B: 3.9003 against an FP16 3.8543, +1.193%, every time) because teacher forcing is deterministic. That is a check that the pipeline replays exactly, not evidence that a quality result reproduces.
The 10 decode iterations inside a run are still one number, integrated into a single energy total. The n here is three runs, not thirty.
INT8 only. NF4 was not repeated in this session; its numbers remain n=1.
GPU package power via NVML — no CPU, DRAM, PSU losses, PUE or CO₂e — decode phase, batch 1, one card, one backend.

What we are not doing with it

Now that there is a session with error bars, the tempting move would be to fold it into the fitted curves. We have not: the August magnitudes disagree with July's by a factor of two, and a fit over both would produce error bars describing a mixture of two backends rather than an estimate of either. build/measured.csv and the published curves are unchanged and still July-only; the repeatability data ships as its own file, alongside the method page's statement of what does and does not carry across sessions. When a second compatible session exists — or when the kernel hypothesis is tested — that decision gets revisited in public.

Reproduce it, or contradict it

Same one command as always; run it three times into three directories and you have your own CV:

for i in 1 2 3; do
  docker run --rm --gpus all -v "$PWD/out:/out" \
    ghcr.io/hongping-zh/ecocompute-mlcube:latest \
    --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --precision INT8 \
    --output_dir /out/rep$i
done

Each energy.json carries its own FP16 baseline, both perplexities and the full version stack, so the three are comparable without any bookkeeping on your side. Drop one on the replications page to see it against our curve — especially if it disagrees.