How EcoCompute measures LLM inference energy
Direct on-device power sampling, not TDP arithmetic. This page is the citable description of the protocol behind every number on quantenergy.tech.
In one paragraph. Energy per configuration (model × precision × GPU) is obtained by sampling GPU
package power with NVML at 10 Hz while the model generates 256 tokens per run, repeating the
run and integrating power over wall-clock time. Nothing is derived from thermal design power. Values that are
not measured — sizes between or beyond our measured anchors, and latency/throughput — are modelled and
labelled as interpolated, extrapolated or estimated in the UI and in the
API response. Measurements and estimates are never mixed silently.
1 · Measurement protocol
| Quantity | How it is obtained |
|---|
| GPU power | NVML (nvmlDeviceGetPowerUsage) polled at 10 Hz for the duration of generation; GPU-package power, board-level |
| Workload | Single-stream text generation, 256 output tokens per run, fixed prompt and sampling settings |
| Repeats | Main dataset (v1.1.0): n = 2 with CV < 2%. RTX 4090 deep dive (July 2026): n = 1 per configuration — one trial whose 10 decode iterations are integrated into a single energy total, so it has no CV. The two layers are drawn differently (see below), never merged into one visual tier |
| Energy | Time-integral of sampled power over the generation window, normalised to joules per 1M output tokens |
| Baseline | Each model's own FP16 run on the same GPU; ΔE% is relative to that baseline |
| Precisions | FP16, NF4, INT8 (FP8 where the hardware supports it) — weight-only quantization |
| Architectures | Turing (T4), Ampere (A800), Ada (RTX 4090 / 4090D), Blackwell (RTX 5090) |
2 · What is measured vs modelled
Every result carries a basis field, and the site renders it as a tag next to the number:
measured — a real NVML run exists for exactly this configuration.interpolated — the model size falls between two measured anchors on the same architecture and precision; the value comes from the fitted crossover curve (curves.json).extrapolated — the size lies outside the measured range; treat the value as directional only.estimated — latency, throughput and VRAM figures from a roofline-style model, not from a power measurement.
Replication is encoded in the marker, not just in the text
measured says a run exists; it does not say how many times it was repeated. Because the site now
shows two measurement layers on the same axes, every chart also encodes replication:
- Filled marker, solid line, solid bar — n ≥ 2 repeated trials from the main dataset
(v1.1.0, CV < 2%).
- Hollow marker, faded line, hatched bar — n = 1, a single trial with no measured spread: the
RTX 4090 deep dive and one
early RTX 4090D point. Hovering a marker reports its n.
The replication count travels with the data: n_trials in
build/measured.csv
and n on every anchor in curves.json. Single-trial points still enter the fit —
discarding real measurements would be worse — but they are visually marked so no one mistakes one observation for a distribution.
How far apart two single trials land
An n = 1 point has no spread of its own, so we re-ran the same configuration on separate rented
RTX 4090 instances to get a sense of the between-session scatter the charts do not show:
- TinyLlama-1.1B INT8: +146.1% (July 2026) and +137.9% (August 2026) — 8.2 points apart.
- TinyLlama-1.1B NF4: three runs of one build gave +11.8 / +9.7 / +10.6% — a range of 2.1 points.
These are separate sessions on different instances with different library versions
(quantization kernels change between bitsandbytes releases), so the scatter mixes hardware, session and
software variation — it is an upper bound on run-to-run noise, not a CV. We keep the runs as separate
n = 1 anchors rather than pooling them into an n = 2 point: filled markers on this site mean
repeated trials of the same configuration with a measured CV < 2%, and these are not that.
The NF4 numbers above come from container validation runs and are not in the dataset.
Run-to-run noise vs between-session scatter: they differ by 30–50×
On 2026-08-20 we ran all five INT8 configurations three times each on one instance, which finally
separates the two kinds of variation the point above conflates
(raw,
per-size mean/SD/CV):
- Within one session, ΔE% is highly repeatable: CV = 0.6–3.9% (energy CV 0.3–2.0%,
FP16-baseline CV 0.1–2.5%) across 0.5B–7B.
- Between sessions it is not: the same five configurations measured 100–140 percentage points
higher in August than in July (e.g. 1.1B +146.1% → +307.1%). That is 30–50× the within-session noise,
so it cannot be attributed to sampling.
- Absolute joules are the least stable quantity of all. Between two consecutive days on the
same host, absolute energy per token drifted 12–17% lower for INT8 and FP16 alike, while the
FP16-normalised ΔE% moved only a few points. This is why every number on this site is a delta against a
baseline measured in the same run, and why absolute J/token should not be compared across days.
Consequence for anyone using or citing energy deltas, ours included: a quantization
penalty measured once on one rented instance can be off by a factor of two, and the run-to-run CV does not
warn you about it. The leading hypothesis for our own gap is the LLM.int8() kernel path, which changes
with the torch build — and no two of our sessions ran the same one: the 2026-07-24 anchors were recorded
with torch 2.4.1+cu121 (Python 3.8), the August native path got torch 2.5.1+cu121, and
the container image pins torch 2.13.0. It is untested; the control is a repeat of one 1.1B INT8 run
with the July build (or with the image, on a host that can nest Docker — ours cannot). The decomposition constrains it: INT8 energy rose 1.41–1.58× at unchanged package
power, i.e. purely a throughput effect. One caveat on the repeats: the perplexity column is bit-identical
across replicates because teacher forcing is deterministic, so the quality axis has CV = 0 by
construction and is not independently replicated.
What the shaded band around a fitted curve is
Every estimate carries a 95% band, and the curve chart shades it. It is not a bootstrap over
replicates — most anchors are n = 1, so there are no replicates to resample, and a band built from the
within-session CV (0.6–3.9%) would be about a tenth as wide as our own sessions disagree. It is built from how
unstable the fit is under its own anchors:
- σ floor = max(residual spread of the fit, leave-one-out error, 2 pts). The leave-one-out
error refits the curve with each anchor withheld and measures how far it moves at that anchor — the honest
answer to “how wrong is this curve about a point it has not seen”. Current values:
Ada NF4 16.1, Ada INT8 47.5, Blackwell NF4 4.8, Turing NF4 5.3 points
(Ampere has 3 anchors per precision, below the 4 needed for leave-one-out, so its residual spread is used).
- Extrapolation term, growing with log-distance outside the measured size range.
- Borrowed-curve and batch/context terms when the answer leans on another architecture or on the
modelled batch effect.
- For a size that is measured, the band instead covers the disagreement between the cards in that
class rather than a fitted σ.
What the band does not cover: between-user and between-session variance. Two of our
own sessions on the same card, same protocol, different torch build differ by 100–140 points — far outside any
band fitted to one of them. Read the band as the precision of the curve, not as a prediction interval for your
machine; the only way to close that gap is independent runs.
The fitted curves and the estimator are open: see estimate.js, optimize.js and the build/ scripts in the site repository.
3 · The finding this protocol produced
Weight-only quantization (NF4/INT8) reliably reduces memory footprint, but it does not always reduce
energy. On small models the dequantization overhead outweighs the memory-bandwidth savings, so quantization
increases energy per token; above a crossover point that depends on both model size and GPU
architecture, it starts to save. Explore your own case on the
Your model tab.
4 · Reproduce it on your GPU
The measurement code is a container, not a description. The EcoCompute energy MLCube runs the same NVML
sampling on your hardware and writes an energy.json with the exact fields these charts read:
git clone https://github.com/hongping-zh/ecocompute-mlcube.git
cd ecocompute-mlcube
pip install mlcube mlcube-docker
mlcube run --mlcube=. --task=energy_estimate --platform=docker
Then drop the resulting energy.json on the Run it yourself tab to overlay your
point on the crossover curve — the overlay is encoded in the URL, nothing is uploaded. The container’s output
schema, honesty guarantees, scope and how to contribute a run are documented on
the container page.
5 · Limitations
- NVML reports GPU-package power, not whole-system draw: CPU, DRAM, PSU losses and cooling are excluded.
- Single-stream generation only; batched serving with continuous batching has a different energy profile.
- Curves are fitted per architecture class, not per card: the Ada class pools RTX 4090D and RTX 4090 measurements, which differ by roughly 20 percentage points at the same model size, so Ada bands are the widest of the four classes.
- The RTX 4090 (Ada) July 2026 column is n = 1 per configuration — a supplementary deep dive, not a replacement for the main dataset. Two single trials of the same configuration have landed 8.2 points apart (see above), which is the resolution to read these points at.
- Results are backend-specific: NF4 and INT8 here mean bitsandbytes NF4 and LLM.int8() at the versions recorded in each report. GPTQ, AWQ, TensorRT-LLM, llama.cpp and newer bitsandbytes releases implement different kernels and can behave differently. "INT8 cost energy at every size we tested" is a claim about this backend on these cards — not about INT8 as a format.
- What would falsify the central claim: a measured run, from this container or an equivalent NVML-based protocol, showing quantization saving energy below the crossover we publish for that architecture, or costing energy above it. We publish such runs on the replications page whether they agree with us or not.
- This is a research protocol, not a certified benchmark, and not affiliated with any energy-consulting body.
6 · How to cite
Full artifact map, concept vs version DOIs and BibTeX: Artifacts & citation. In short: