EcoCompute.
"Quantization doesn't always save energy. See for yourself."
Every quantization tool tells you how to quantize. We tell you whether you should — and ship the container that lets you check us.
Scope of every number here: GPU-package power (NVML), not whole-system draw — no PSU losses, CPU, DRAM, cooling, PUE or CO₂e. Single-stream decode, batch 1, 256 tokens. Replication is stated per point: the main dataset is n = 2 (CV < 2%), the RTX 4090 deep dive is n = 1 per configuration. Limits and how to disprove us →
Open measurement stack: an MLCube-compatible container doing direct NVML power sampling, the datasets it produced (Zenodo, CC BY 4.0), and the preprint that interprets them.
Replicate this number · 复现环境
The stacks below are not identical to each other, and saying so is the point. The published Ada curves were measured in the 2026-07-24 session; the container image you would pull today pins a different torch; and a native (non-Docker) install pins a third one, on which the same INT8 configurations measured about twice the energy penalty. Which of these you run changes the INT8 number you get. Run-to-run vs cross-session, with the CVs →
Hardware · 硬件
NVIDIA GeForce RTX 4090 · Ada Lovelace · 24 GB · compute capability 8.9 · CUDA build
12.1 · NVIDIA driver 535.146.02 in the August sessions (the July driver is in
run_metadata/gpu_summary.csv of that session's raw archive). Rented AutoDL instances.
Other anchors in the fit: RTX 4090D (Ada), RTX 5090 (Blackwell), A800 (Ampere), T4 (Turing).
Power: NVML nvmlDeviceGetPowerUsage at 10 Hz, integrated over the decode loop —
GPU-package power only, not wall power.
Software · 软件
Container image (what you get today): Python 3.10 · torch 2.13.0 ·
transformers 4.57.6 · bitsandbytes 0.43.3 · accelerate 1.14.0 ·
nvidia-ml-py 13.610.43, base nvidia/cuda:12.1.1-cudnn8-runtime-ubuntu22.04
(requirements.lock.txt).
2026-07-24 session — the anchors the Ada curves are fitted on: Python 3.8 ·
torch 2.4.1+cu121 · transformers 4.46.3 · bitsandbytes 0.45.5 ·
accelerate 1.0.1. Older than the current pins.
2026-08-19 / 08-20 native sessions: Python 3.10.20 · torch 2.5.1+cu121, the other four
matching the pins above.
Data · 数据
Main dataset v1.1.0 — DOI 10.5281/zenodo.19647290, n = 2 per configuration, CV < 2%.
RTX 4090 deep dive — DOI 10.5281/zenodo.22037483
(concept 10.5281/zenodo.22019741): 2026-07-24 energy, n = 1; 2026-08-19 energy + same-run perplexity, n = 1;
2026-08-20 INT8 n = 3 per size, CV of ΔE% 0.6–3.9%.
Protocol: batch size 1 · context 2048 · 256 generated tokens per run · 10 decode iterations after 2
warmups · greedy · each model's own FP16 baseline in the same session.
Quality probe corpus SHA-256 22ac091a6383740d30f8e41ae144032c873c772db5e1c901112c1c330fdc5504.
Container · 容器
Image ghcr.io/hongping-zh/ecocompute-mlcube:latest, rebuilt by CI on every merge to
main. One INT8 run of TinyLlama-1.1B:
docker run --rm --gpus all -v "$PWD/out:/workspace/outputs" ghcr.io/hongping-zh/ecocompute-mlcube:latest energy_estimate --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --params_b 1.1 --precision INT8 --gpu_arch auto --output_dir /workspace/outputs
Writes out/energy.json, schema ecocompute-energy/1.1, carrying
results.vs_fp16_energy_pct, results.basis and the software block that
recorded the versions above. Drop it into /replications/ to overlay it on the curve.
Inference energy per 1M tokens
Crossover curve · in what we measured, the bigger the model, the more quantization saves
Solid lines connect measured points only (direct NVML). Marker fill encodes replication, not accuracy: filled dots are points repeated at least twice in the main dataset (v1.1.0, n = 2, CV < 2%); hollow dots on a faded line were measured once (n = 1) — the July 2026 RTX 4090 deep dive and one early RTX 4090D point — so they carry no measured spread and should be read as a single observation, not as a distribution. INT8 was measured on A800 and RTX 4090 only, and sizes outside each GPU's measured range aren't drawn here. For fitted extrapolation beyond the measured range, use the Your model tab.
Where these numbers come from · the open measurement container
Direct measurement, never TDP
NVML on-device power sampling at 10 Hz over the decode phase, 256 tokens per run with warmup, repeated for the configured iterations. Energy, average power and throughput come out of that trace — no datasheet wattage, no wall-meter guesswork.
It refuses to fake a measurement
No GPU, or NVML power telemetry unavailable? The run still completes, but the report is derived from the published dataset and labelled basis ≠ measured with the fallback named in measurement_source. A fallback is never dressed up as a measurement.
Portable and contract-checked
An MLCube-compatible descriptor with one task, energy_estimate, exercised with the official mlcube CLI on the Docker platform; a CPU descriptor verifies the build/mount/report contract without a GPU. Dependencies are pinned with a full transitive lock, because bitsandbytes and torch change NF4/INT8 kernels between releases.
Machine-checkable output
Every report validates against schema/energy.schema.json and carries its own scope: certified_benchmark_result: false, plus a scenario_note saying the SingleStream/Offline label is nominal and not enforced by LoadGen.
Latest run · RTX 4090 (Ada) · measured in our own open container
NF4 reaches break-even just above 3B on this card: +36% energy penalty at 0.5B → +0.8% at 3B → −28% saving at 7B. These points are now folded into the fitted Ada curve, which puts the crossover at ≈3.7B — later than this card alone, because the fit also carries the more heavily penalised RTX 4090D anchors. INT8 (bitsandbytes LLM.int8()) did not save energy at any size we tested (+50% … +242%, five sizes, one card, n = 1 each): it does lower instantaneous power, but decode throughput collapses to 9–19 tok/s (FP16: 39–62 tok/s), so a token ends up costing more. That is a statement about this backend on this card, not about INT8 in general — a different kernel, card or serving stack could well reverse it.
INT8 energy penalty by model size · vs FP16 on the same card
Every bar here points the wrong way: no model size we tested saved energy in INT8 on this card, and the penalty does not fall monotonically with size (1.5B is worse than 1.1B). The penalty does shrink towards 7B, but we have no measurement above it, so we do not claim INT8 breaks even at some larger size — on this card INT8 bought memory, not energy. Bars are hatched because these are single trials (n = 1) on one RTX 4090. The 1.1B bar has since been re-measured on a second RTX 4090 instance (August 2026, same container): +138% against the July run's +146% — two single trials 8 points apart, on different software versions. Both are in measured.csv as separate n = 1 anchors; neither is a replication of the other. Also note: INT8 was measured on Ada and Ampere only, H100 borrows the Ampere curve (estimated, not measured), and T4 and RTX 5090 have no INT8 curve at all.
Honest scope · what this run does and does not support
| ✅ Do | ⚠️ Don't |
|---|---|
| Treat each value as a real measurement of this RTX 4090 under this workload (256 tokens, batch 1, single stream) | Read n = 1 as a tight distribution: every configuration was run once (ten decode iterations integrated into one energy total), so there is no std and no CV — unlike the main dataset's n = 2 / CV < 2% |
| Compare precisions on the same card — that is what ΔE% is | Compare cards across these two layers without noting that one is replicated and one is not (hollow markers and hatched bars mark n = 1 site-wide) |
| Use the numbers as GPU-package energy from direct NVML sampling | Present them as whole-system, datacenter or carbon numbers — there is no PUE, no CPU/DRAM and no grid model here |
| Cite it as a supplementary single-platform case study (DOI) alongside v1.1.0 | Cite it as a certified MLPerf/MLCommons result, or as a replacement for the main dataset |
| Reproduce or contradict it — one card, one hour: run the container and publish your point | Assume our card generalises to yours: cooling, driver, power limit and BIOS all move absolute energy |
All raw energy.json reports, the aggregated CSV, environment metadata and both figures are archived
under DOI 10.5281/zenodo.22037483,
CC BY 4.0 — a single-platform deep dive that sits alongside the main
v1.1.0 dataset, which remains
the reference for every other GPU here. Honest scope: n = 1 per configuration
(no std/CV yet, unlike v1.1.0's n = 2 / CV < 2%) and NVML measures GPU-package power,
not whole-system wall power — a supplementary case study, not a certified benchmark. Reproduce it on your own card
from the Run it yourself tab.