An open micro-benchmark for LLM quantization energy: a measurement protocol, a runnable container and an archived dataset. Independent open research project — building toward a reproducible inference-energy standard.
Measure whether LLM quantization actually saves energy — on your GPU.
Not TDP estimates: power is sampled directly from the GPU (NVML). Enter your GPU and model size to get a direction; run the container to get your own measured number.
Enter a GPU and a model size and get a directional verdict. The tool is right below: Tested models picks a card and model we actually measured; Your model takes your own model size and extrapolates the fit; Optimize searches precision × batch × context under your constraints. Directional is not a production decision. See the limitations for the boundary of every number.
Inference energy per 1M tokens
Chart — inference energy per 1M tokens. Bars compare FP16, NF4 and INT8 on the selected GPU. Above zero means quantization costs more energy, below zero means it saves; the verdict line states the same result in words.
⚠ Conclusion boundary — read this before quoting the number
This is not a universal law for all GPUs, models or inference frameworks. Within the GPUs, software stacks, batch sizes, context lengths and measurement windows measured so far, NF4 shows a crossover that depends on model size and hardware — not “NF4 saves energy on big models, INT8 never does”.
NVIDIA GPUs onlysingle-stream decodebatch size = 1256 tokensmain dataset n = 2GPU-package energy, not wall-plugoutside the fitted range is not a measurement
Filled repeated runs — most trustworthyHollow a single run — directional onlyCurve fitted / extrapolated — not a new measurement
Paste the verdict + citation into your tech spec, or share this exact config.
Crossover curve · in what we measured, the bigger the model, the more quantization saves
NF4INT8Your modelFilled = n ≥ 2 repeated trialsHollow = n = 1 single trialAbove zero = penalty (more energy) · below = savings
Chart — measured crossover curve. Energy change vs FP16 against model size. Marker shape encodes replication: filled = repeated runs, hollow = a single run. Shape and sign, not colour alone.
Filled repeated runsHollow single run — directionalCurve fitted / extrapolated
Not a universal law: NVIDIA GPUs · single-stream decode · batch 1 · 256 tokens · n = 2 · GPU-package energy, not wall-plug. Beyond the fitted range the curve is not a measurement. Full scope →
Solid lines connect measured points only (direct NVML). Marker fill encodes replication, not accuracy: filled dots are points repeated at least twice in the main dataset (v1.1.0, n = 2, CV < 2%); hollow dots on a faded line were measured once (n = 1) — the July 2026 RTX 4090 deep dive and one early RTX 4090D point — so they carry no measured spread and should be read as a single observation, not as a distribution. INT8 was measured on A800 and RTX 4090 only, and sizes outside each GPU's measured range aren't drawn here. For fitted extrapolation beyond the measured range, use the Your model tab.
⚠ Conclusion boundary — read this before quoting the number
This is not a universal law for all GPUs, models or inference frameworks. Within the GPUs, software stacks, batch sizes, context lengths and measurement windows measured so far, NF4 shows a crossover that depends on model size and hardware — not “NF4 saves energy on big models, INT8 never does”.
NVIDIA GPUs onlysingle-stream decodebatch size = 1256 tokensmain dataset n = 2GPU-package energy, not wall-plugoutside the fitted range is not a measurement
Filled repeated runs — most trustworthyHollow a single run — directional onlyCurve fitted / extrapolated — not a new measurement
Fitted crossover curve · your model marked with its uncertainty
Fitted curve95% band (fit instability)Measured anchors (n ≥ 2)Anchors measured once (n = 1, provisional)Independent replication (not fitted)Your model ± CIAbove zero = penalty · below = savings
Chart — fitted crossover curve with a 95% band. Your model is marked with its uncertainty. The line is a fit through measured anchors, not new measurements.
Filled repeated runsHollow single run — directionalCurve fitted / extrapolated
Not a universal law: NVIDIA GPUs · single-stream decode · batch 1 · 256 tokens · n = 2 · GPU-package energy, not wall-plug. Beyond the fitted range the curve is not a measurement. Full scope →
Hollow anchors were measured once (n = 1, the RTX 4090 deep dive) and filled anchors come from the replicated main dataset (n = 2, CV < 2%); both feed the fit, but a hollow point carries no measured spread. The curve is a fitted extrapolation calibrated to the measured anchors (dots) — not new measurements. Beyond the largest measured anchor the slope is not measurement-anchored, and INT8 on non-A800 architectures is fitted. Treat large-model (≈≥7B) points as directional — see About → How we measure. Each fitted curve also inherits the software stack its anchors were measured on — the crossover is sensitive to the stack as well as the architecture (on one unchanged RTX 5090, a newer stack moved it from ≈5B to ≈1.8B; the re-test), so treat the fitted crossing as card-plus-stack, not hardware destiny. What each fitted curve rests on: Ada NF4 9 anchors (RTX 4090 ×5, RTX 4090D ×4) · Ada INT8 6 (RTX 4090 only) · Blackwell NF4 4 · Turing NF4 4 · Ampere NF4 3 and INT8 3 (A800). Ada is the only class that pools two different cards, and they disagree by roughly 20 points at the same model size, so the Ada band is the widest of the four — see the methodology page for the leave-one-out error per class.
Share your exact model size + GPU + precision.
Report schema output (JSON)
From static query to constrained optimization: give your GPU, model size, an objective and optional
budgets — the tool exhaustively searches precision × batch × context, filters by your constraints and returns
the objective-optimal config, alternatives and the energy↔latency Pareto frontier. Energy is measured-anchored;
latency/throughput are a roofline model; VRAM is computed — each field is labelled below.
Filled repeated runsHollow single run — directionalCurve fitted / extrapolated
Not a universal law: NVIDIA GPUs · single-stream decode · batch 1 · 256 tokens · n = 2 · GPU-package energy, not wall-plug. Beyond the fitted range the curve is not a measurement. Full scope →
Alternatives
Energy ↔ latency Pareto frontier
Non-dominated trade-offs: no other config is both lower-energy and lower-latency. Down-left is better; ★ is the recommended config.
Recommended (★)Frontier config
Share this constrained scenario + result.
Every published measurement on this site is accompanied by its reproducibility artifacts. The EcoCompute energy MLCube container runs the same
measurement on your GPU — direct NVML power sampling, no TDP guesses — and writes an
energy.json with the exact same fields the charts here use. Run it, then drop the result
below to overlay your measurement on the crossover curve. No GPU of your own? The
Colab notebook
does the same measurement on a free T4.
Full guide, output schema, scope and how to contribute a run: the container page.
② I want to verify it
The answer should not rest on us alone. This path gets you a comparable number on your own hardware — a result that contradicts ours is as welcome as one that confirms it.
1 · No GPU? Run the notebook in Colab
A free Colab T4 is enough — no GPU of your own, no Docker, nothing installed locally. The notebook runs the same NVML measurement as the container and writes the same energy.json, then prints an overlay link and a block of text you can paste into an issue.
It measures your chosen precision and its own FP16 baseline, in randomized order, waiting for the card to come back to idle before each arm — the two things we got wrong the first time and wrote up. Both arms have to fit: on a 16 GB T4 that means models up to about 3B, and the notebook refuses to start rather than failing with an out-of-memory error twenty minutes in. Budget 20–35 minutes; most of it is cooldown and the model download.
One pass is n=1 and energy is integrated from sampled power — on our RTX 4090 the GPU's hardware energy counter read on average 14.9% higher than integrating the same traces, so read these figures at that resolution. The last cell repeats both arms 5× if you want a repeatability figure. Same scope as everything else here: GPU-package power, not wall AC; not a certified benchmark result.
2 · Have a GPU? Run the container
One command, nothing to configure. Needs an NVIDIA GPU. With Docker it runs the prebuilt image; on a rented GPU instance that cannot nest containers (AutoDL, vast.ai …) the same command installs into a venv instead and runs the identical code. No GPU? It still runs and returns dataset-derived reference values, clearly flagged (never faked).
Download from an immutable release, verify against the published checksum, then run — what executes is a specific checksummed version, not whatever main holds today (bash quickstart.sh --dry-run prints the whole plan first). The container runs non-root, as your own uid. Measures NF4 and its own FP16 baseline on TinyLlama-1.1B, writes a schema-validated energy.json to ./ecocompute-out/ and prints an overlay link. The measurement itself takes a few minutes; the first run is dominated by downloads — the one-time image pull (or dependency install) plus a ~2 GB model download — and later runs reuse the caches. Our one timed end-to-end run took 57 minutes on a China-hosted rented RTX 4090 (mostly fetching torch/CUDA wheels; since fixed to use a domestic index, but not re-timed). No flags: the architecture comes from the NVML device name, so your card cannot be mislabelled. Other models/precisions and the plain docker run form: the container page.
Repo & full guide: github.com/hongping-zh/ecocompute-mlcube. This is a supplemental energy-methodology container (MLCube-compatible), not a certified benchmark. The script ends with a link like quantenergy.tech/?tab=run&overlay=… — the point is encoded in the URL, so opening it restores your marker below with nothing uploaded.
3 · What you get — the same structure as this site
Example energy.json. The two fields the charts read are highlighted: vs_fp16_energy_pct (ΔE% vs FP16) and basis (measured / interpolated / extrapolated).
Shown here is the shipped example (examples/energy.measured.illustrative.json): its power/throughput numbers are synthetic only to illustrate the output shape. When you run on a real GPU, the same fields carry your actual NVML measurement (basis: "measured").
4 · Overlay your result on the crossover curve
Drop your energy.json here, or click to choose a file
ModelGPUPrecision
ΔE vs FP16source
Fitted curve (your GPU arch)95% band (fit instability)Measured anchors (n ≥ 2)Anchors measured once (n = 1, provisional)Your energy.json
Your point is placed at (model size, ΔE%) from the uploaded report and coloured by its basis. The fitted curve is calibrated to measured anchors for the report's GPU architecture — filled dots are replicated (n = 2), hollow dots were measured once (n = 1) — and beyond the largest anchor it is extrapolated. Nothing is uploaded to a server; parsing is 100% in your browser.
Compare several runs — drop more files to stack points.
Deep re-test (Ada): repeatability, paired perplexity
The five layers version independently. “Running the project” today means the container release schema-1.3-r3, whose reports validate as ecocompute-energy/1.3 against Protocol v1.1 — and every published number traces to a dataset DOI above. Full version-by-version correspondence: spec §8.
26 of 56 coverage-matrix cells have a number; the other 30 are empty. The seed data's thermal state is unknown, which is not cold.
Backends are almost entirely bitsandbytes (llama.cpp has exactly one cell), and everything is batch 1.
Stack versions drift: the fitted curves inherit the software stack their anchors were measured on — the crossover is card-plus-stack, not hardware destiny (the RTX 5090 re-test).
Not a company. It is three things: an open measurement protocol (Protocol v1.1 — what makes two submissions comparable), an MLCube container (anyone can reproduce the same measurement on their own GPU), and a dataset archived on Zenodo (360+ measured configurations, with a DOI); plus an SSRN preprint stating the conclusions. Single maintainer (Hongping Zhang), Apache-2.0.
Should I read the conclusions or the protocol first?
Conclusions first: spend a minute with the tool above to get a direction. You only need the protocol and the limitations when you are about to cite it or make a decision on it — the site deliberately splits “what we require / what we measured / what changed / what we are still exploring” into four layers so you never have to wade through methodology up front.
I have no GPU — can I use this?
Yes. The estimator is pure front-end and needs no hardware at all. To get your own measured number, a free Colab T4 is enough (20–35 minutes, nothing to install) and produces the same energy.json as the container; both arms have to fit, which on a 16 GB T4 means models up to about 3B.
Can these results guide a production deployment?
At this stage they give a direction, not a production decision. Every number is GPU-package power (NVML) — excluding PSU losses, CPU, DRAM, cooling and PUE — measured at single-stream batch 1 over 256 tokens, in the generation window only. It is neither whole-system power nor any certified benchmark result. For a production decision, measure your own stack, batch and workload — which is exactly what the container is for.
I'm a researcher, a cloud provider, or a developer — where should I click?
Developer / just deciding whether to quantize → the estimator above · when not to quantize · ask Watt · or run Colab on your own model
Does “NF4 saves energy” hold for my model?
It depends on your model size and your GPU — there is no unconditional answer. The fitted crossovers sit near Turing 2.1B / Ada 3.7B / Blackwell 4.8B: below the crossover quantization usually costs more energy, above it, it starts to save. And coverage is almost entirely bitsandbytes at batch 1, so changing the runtime can flip the sign — on the same card, llama.cpp Q4_0 is −61.9% while bitsandbytes LLM.int8() is +106%. So: get a direction from the estimator, then verify on your own stack with the container.
Full content index · how to read this site
Four layers · how to read this site
The site is deliberately split so that what we require,
what we measured, what changed and what we are still exploring never blur into one page.
1 · Spec — the normative protocol
What makes two submissions comparable. The normative protocol, the container and schema that enforce it.
2026-09-25 · data RTX 4090 NF4 re-tested across two physical cards — July's +0.8% anchor holds (mean +1.6%, n=3); the −15.1% historical reading was not reproduced and stays a divergent observation; no direction flip at the 3B anchor under the current stack. → 2026-09-24 · data All energy numbers unified on the generation measurement window; the llama.cpp row re-cut from its 100 Hz traces (−62.9% at 576 tokens, counter basis). → 2026-09-25 · site The site split into four layers — spec, evidence, changelog, findings; every count now has a defined denominator. → Full history, each entry linked to its evidence: the changelog →
Public research layer. This site is static: documentation, charts, dataset, DOIs and reproducible
materials — Apache-2.0, no login, no result tracking. A future service layer (benchmark API, report
generation, organisation & project management, private measurement results, enterprise permissions)
is defined as a contract only and is deliberately not deployed. The two layers share exactly one
thing: the energy.json schema — which is what keeps any future private result comparable
with the curves published here.