Protocol v1.1 — the measurement specification

This is the normative layer of the site: what makes two submissions comparable, so that anyone following this page on any card produces numbers that can be checked against what we have measured. It states no measured results of its own — every number lives in the evidence layer; every count on the site is defined in the number glossary. Execution guides: the container, run it yourself; the machine-checkable contract: the schema.

1 · What the protocol requires · v1.1 normative

• Workload: single-stream decode, batch 1, 256 generated tokens, 10 decode iterations after a warmup, greedy.
• Baseline: an FP16 run measured in the same session on the same card — a quantized run on its own cannot produce a ΔE%.
• Measurement: NVML GPU-package power at ≥ 10 Hz, integrated over the decode loop (the generation window: model load, quantization and warm-up excluded). Not wall power. State the window and the output length with every energy claim — re-integrating one session over different windows moves its 64-token figure by up to 12.5 percentage points.
• Required in the report: GPU and architecture, driver, runtime/engine and version, model and params_b, precision, batch size, context length, J/token, tok/s, mean package watts — schema ecocompute-energy/1.2 (1.3 adds whole-run power-trace sidecars so the window can be re-cut post hoc).
• Thermal state (new in v1.1): the report records a thermal block — the warmup count and temperature at start, steady state, end and peak — plus the achieved (not requested) sample rate, an enforced cooldown between arms and randomised arm order. These are corrections we had to make to our own numbers, written up here. The 42 seed measurements have no thermal block at all, because it did not exist when they were taken — their thermal state is unknown, not cold, which is the honest reason the default stayed cold.
• Cold and steady are two different measurements. The container defaults to --thermal_mode cold: cool to idle, fixed warmup, measure. --thermal_mode steady instead keeps warming, one pass per second, until three consecutive readings fall within 2 °C, then measures a hot card — which clocks lower and reports more energy per token. Compare within a mode, not across one. A run that never settles says steady_state_reached: false rather than pretending it did, and a card with no temperature sensor says basis: "unavailable" rather than inventing a number.

2 · Contributing a measurement · the flow

One GPU, one model, about half an hour (first run including downloads: up to an hour — see the time budget). You do not need a GPU of your own: a free Colab T4 is enough.

• 1 · Pick a configuration nobody has measured — the coverage matrix shows which ones (26 of 56 cells have data today).
• 2 · Run it. Either the Colab notebook (no card needed) or the container — both write the same energy.json.
• 3 · Drop that file on /replications/. It is parsed in your browser, drawn against the published curve, and one button copies the submission and opens a GitHub issue for you to send. There is no upload endpoint; nothing is submitted behind your back.

The gaps that could change what this site says (not just add rows): runtimes — almost everything here is bitsandbytes, llama.cpp has exactly one cell, vLLM/TensorRT-LLM/SGLang have zero; quant formats — Q4_K_M, GPTQ, AWQ, FP8 on Ada: none measured under this protocol; batch size — every point here is batch 1; cards — no 3090, A100, H100, L4, no AMD (native ROCm telemetry is tracked here — leave your card model and ROCm version to help prioritise it); quality — only 13 of 42 seed rows carry a paired perplexity.

What you get: every accepted submission is published in /replications/ credited by your handle, and drawn on every chart it belongs on — including when it contradicts us; a run that breaks the curve is published as prominently as one that confirms it. Accepted points go into the next Zenodo release of the dataset, which carries its own DOI and names its contributors. Submissions are reviewed by a human for schema and plausibility — never for result — so publication is not instant and is not automatic.

3 · What is measured vs modelled

Every result carries a basis field, and the site renders it as a tag next to the number:

• measured — a real NVML run exists for exactly this configuration.
• interpolated — the model size falls between two measured anchors on the same architecture and precision; the value comes from the fitted crossover curve (curves.json).
• extrapolated — the size lies outside the measured range; treat the value as directional only.
• estimated — latency, throughput and VRAM figures from a roofline-style model, not from a power measurement.

Replication is encoded in the marker, not just in the text

measured says a run exists; it does not say how many times it was repeated. Because the site now shows two measurement layers on the same axes, every chart also encodes replication:

• Filled marker, solid line, solid bar — n ≥ 2 repeated trials from the main dataset (v1.1.0, CV < 2%).
• Hollow marker, faded line, hatched bar — n = 1, a single trial with no measured spread: the RTX 4090 deep dive and one early RTX 4090D point. Hovering a marker reports its n.

The replication count travels with the data: n_trials in build/measured.csv and n on every anchor in curves.json. Single-trial points still enter the fit — discarding real measurements would be worse — but they are visually marked so no one mistakes one observation for a distribution.

How far apart two single trials land

An n = 1 point has no spread of its own, so we re-ran the same configuration on separate rented RTX 4090 instances to get a sense of the between-session scatter the charts do not show:

• TinyLlama-1.1B INT8: +146.1% (July 2026) and +137.9% (August 2026) — 8.2 points apart.
• TinyLlama-1.1B NF4: three runs of one build gave +11.8 / +9.7 / +10.6% — a range of 2.1 points.

These are separate sessions on different instances with different library versions (quantization kernels change between bitsandbytes releases), so the scatter mixes hardware, session and software variation — it is an upper bound on run-to-run noise, not a CV. We keep the runs as separate n = 1 anchors rather than pooling them into an n = 2 point: filled markers on this site mean repeated trials of the same configuration with a measured CV < 2%, and these are not that. The NF4 numbers above come from container validation runs and are not in the dataset.

Run-to-run noise vs between-session scatter: they differ by 30–50×

On 2026-08-20 we ran all five INT8 configurations three times each on one instance, which finally separates the two kinds of variation the point above conflates (raw, per-size mean/SD/CV):

• Within one session, ΔE% is highly repeatable: CV = 0.6–3.9% (energy CV 0.3–2.0%, FP16-baseline CV 0.1–2.5%) across 0.5B–7B.
• Between sessions it is not: the same five configurations measured 100–140 percentage points higher in August than in July (e.g. 1.1B +146.1% → +307.1%). That is 30–50× the within-session noise, so it cannot be attributed to sampling.
• Absolute joules are the least stable quantity of all. Between two consecutive days on the same host, absolute energy per token drifted 12–17% lower for INT8 and FP16 alike, while the FP16-normalised ΔE% moved only a few points. This is why every number on this site is a delta against a baseline measured in the same run, and why absolute J/token should not be compared across days.

Consequence for anyone using or citing energy deltas, ours included: a quantization penalty measured once on one rented instance can be off by a factor of two, and the run-to-run CV does not warn you about it. The leading hypothesis for our own gap is the LLM.int8() kernel path, which changes with the torch build — and until recently no two of our sessions ran the same one: the 2026-07-24 anchors were recorded with torch 2.4.1+cu121 (Python 3.8), the August native path got torch 2.5.1+cu121, and the container image pins torch 2.13.0. So the kernel explanation could not be separated from everything else that differs between two rented instances two months apart.

That control now exists. On 2026-09-30 four July anchor cells (Qwen2-1.5B and Qwen2-7B at NF4 and INT8) were repeated on the identical stack — torch 2.4.1+cu121, transformers 4.46.3, bitsandbytes 0.45.5, accelerate 1.0.1 — on another RTX 4090. Holding the kernel version fixed did not make the drift go away; it redistributed it by size. The two 7B cells returned to within 1.2 points of July (−28.46% → −28.25% NF4, +49.54% → +50.75% INT8), while the two 1.5B cells moved 7.5 and 16.3 points (+15.11% → +22.6% NF4, +180.73% → +197.05% INT8). Read that with the sentence above: this is what a single measurement is worth when nothing about the software changed — about a point at 7B, and up to sixteen points at 1.5B. Two trials 48 minutes apart on one card agreed to 0.1–4.5 points, so the number by which these rows disagree with July is several times their own within-session noise. (Changelog · 2026-09-30; data: rtx4090_anchor_2026-09-30.csv.)

Still untested: whether the same holds on today's image build. The LLM.int8() decomposition constrains the mechanism: INT8 energy rose 1.41–1.58× at unchanged package power, i.e. purely a throughput effect. One caveat on the repeats: the perplexity column is bit-identical across replicates because teacher forcing is deterministic, so the quality axis has CV = 0 by construction and is not independently replicated.

What the shaded band around a fitted curve is

Every estimate carries a 95% band, and the curve chart shades it. It is not a bootstrap over replicates — most anchors are n = 1, so there are no replicates to resample, and a band built from the within-session CV (0.6–3.9%) would be about a tenth as wide as our own sessions disagree. It is built from how unstable the fit is under its own anchors:

• σ floor = max(residual spread of the fit, leave-one-out error, 2 pts). The leave-one-out error refits the curve with each anchor withheld and measures how far it moves at that anchor — the honest answer to “how wrong is this curve about a point it has not seen”. Current values: Ada NF4 16.1, Ada INT8 47.5, Blackwell NF4 4.8, Turing NF4 5.3 points (Ampere has 3 anchors per precision, below the 4 needed for leave-one-out, so its residual spread is used).
• Extrapolation term, growing with log-distance outside the measured size range.
• Borrowed-curve and batch/context terms when the answer leans on another architecture or on the modelled batch effect.
• For a size that is measured, the band instead covers the disagreement between the cards in that class rather than a fitted σ.

What the band does not cover: between-user and between-session variance. Two of our own sessions on the same card, same protocol, different torch build differ by 100–140 points — far outside any band fitted to one of them. Read the band as the precision of the curve, not as a prediction interval for your machine; the only way to close that gap is independent runs.

The fitted curves and the estimator are open: see estimate.js, optimize.js and the build/ scripts in the site repository.

4 · The finding this protocol produced

Weight-only quantization (NF4/INT8) reliably reduces memory footprint, but it does not always reduce energy. On small models the dequantization overhead outweighs the memory-bandwidth savings, so quantization increases energy per token; above a crossover point that depends on the model size, the GPU architecture and the software stack the numbers were measured on, it starts to save — on one unchanged RTX 5090, a newer stack moved the crossing from ≈5B to ≈1.8B (re-test). Explore your own case on the Your model tab.

5 · Reproduce it on your GPU

The measurement code is a container, not a description. The EcoCompute energy MLCube runs the same NVML sampling on your hardware and writes an energy.json with the exact fields these charts read:

git clone https://github.com/hongping-zh/ecocompute-mlcube.git
cd ecocompute-mlcube

pip install mlcube mlcube-docker
mlcube run --mlcube=. --task=energy_estimate --platform=docker

Then drop the resulting energy.json on the Run it yourself tab to overlay your point on the crossover curve — the overlay is encoded in the URL, nothing is uploaded. The container’s output schema, honesty guarantees, scope and how to contribute a run are documented on the container page.

6 · Number glossary · every count on this site, with its denominator

The site quotes several counts — 42, 29, 144, 26/56, 360+ — that answer different questions and must never be read interchangeably. This table is the single normative definition of each; every page that quotes one links here. Denominators verified against the build scripts, not prose.

Rule of thumb: 56 answers "what could exist", 26 answers "what has a number", 29 answers "what the curves stand on", 42 and 360+ answer "what is archived in the seed dataset", and 144 counts a separate sweep including its baselines.

7 · Limitations

• NVML reports GPU-package power, not whole-system draw: CPU, DRAM, PSU losses and cooling are excluded.
• Single-stream generation only; batched serving with continuous batching has a different energy profile.
• Curves are fitted per architecture class, not per card: the Ada class pools RTX 4090D and RTX 4090 measurements, which differ by roughly 20 percentage points at the same model size, so Ada bands are the widest of the four classes.
• The RTX 4090 (Ada) July 2026 column is n = 1 per configuration — a supplementary deep dive, not a replacement for the main dataset. Two single trials of the same configuration have landed 8.2 points apart (see above), which is the resolution to read these points at.
• Results are backend-specific: NF4 and INT8 here mean bitsandbytes NF4 and LLM.int8() at the versions recorded in each report. GPTQ, AWQ, TensorRT-LLM, llama.cpp and newer bitsandbytes releases implement different kernels and can behave differently. "INT8 cost energy at every size we tested" is a claim about this backend on these cards — not about INT8 as a format.
• What would falsify the central claim: a measured run, from this container or an equivalent NVML-based protocol, showing quantization saving energy below the crossover we publish for that architecture, or costing energy above it. We publish such runs on the replications page whether they agree with us or not.
• This is a research protocol, not a certified benchmark, and not affiliated with any energy-consulting body.

8 · How to cite

Full artifact map, concept vs version DOIs and BibTeX: Artifacts & citation. In short:

• The methodology and argument — preprint “Weight-Only Quantization Does Not Always Save Energy: An Empirical Study of LLM Inference Across NVIDIA GPU Platforms”, SSRN #6854700 (archived: 10.5281/zenodo.21066652).
• The numbers on this site — main dataset v1.1.0, 10.5281/zenodo.19647290 (CC BY 4.0).
• The RTX 4090 (Ada) deep dive — 10.5281/zenodo.22037483 (CC BY 4.0).