EcoCompute · MLPerf Client

MLPerf Client v2.0 measures speed. Here is how to measure the energy of the same run.

Last updated 2026-09-03 · written against mlcommons/mlperf_client at version.json = 2.0.0 · announcement: mlcommons.org · signed downloads

MLPerf Client v2.0 is a good release: signed Windows and macOS applications, a CLI as well as a GUI, agentic and image-generation tests, and far broader hardware coverage. Its headline is that you can now generate results on your own machine instead of in a lab.

Those results are performance results — geomean time to first token and 2nd+ token generation rate. This page exists because two questions are left over, and people arrive here asking them:

1 · What did quantization cost in energy? Every LLM configuration shipped in v2.0 runs an already quantized model — Llama-3.1-8B-Instruct-Q4_0.gguf on the llama.cpp CUDA path, *_ov-int4-* on OpenVINO, *-mlx-q4 on Apple MLX, and nvfp4 / fp8 / quanto_int8 for image generation. There is no FP16 configuration anywhere in the suite, which is reasonable — it benchmarks what vendors actually ship — but it means the difference between quantized and unquantized is not a quantity the suite produces.

2 · What did it cost in quality, in that same run? v2.0 has accuracy harnesses (tools/accuracy/ifeval, tools/accuracy/mmlu), but they score a model, not a delta against that model's own FP16 baseline on the same machine in the same session.

Both are exactly what the EcoCompute container measures, in software, on hardware you already own. Neither project replaces the other: MLPerf Client is a benchmark with certified results and a submission process; this is a research protocol with none of that.

Update, 2026-09-03: we ran the missing FP16 baseline, then re-ran it properly

The earlier version of this page said that closing the runtime gap — measuring GGUF F16 against GGUF Q4_0 inside one runtime, rather than comparing our bitsandbytes numbers to llama.cpp's — was the next thing on our list. It is done, against the same Llama-3.1-8B-Instruct-Q4_0.gguf that v2.0 itself pulls. The first pass (2026-08-31, n = 3) shipped with a list of known defects and a promise to redo it; the numbers below are the redo, and the older ones are kept beside them rather than overwritten.

One RTX 4090 at 450 W, one llama.cpp build (b10643-192067b72), Llama-3.1-8B-Instruct, batch 1, decode-only. 45 runs: 3 arms × 3 token lengths (64/320/576) × n = 5, randomized order, forced cooldown to ~25 W before every run, NVML hardware energy counter.

llama.cpp armmJ/tokentokens/Jtok/sdecode powerΔenergyΔperplexityprovisional 2026-08-31
F1649800.20154.7273 W——4815 mJ/tok
Q4_0 (ours)18990.527155.9296 W−61.9%+5.60%1754 mJ/tok, −63.6%
Q4_0 (the v2.0 file)18340.545173.8319 W−63.2%+5.61%1795 mJ/tok, −62.7%

The saving is not lower power — it is shorter time, and the clean run says so more sharply than the first one did. Q4_0 decodes at 296–319 W against F16's 273 W: it draws more power and still costs 62–63% less energy per token, because it emits 2.8–3.2× as many tokens per second. On this same card, bitsandbytes LLM.int8() costs 106% more energy per token than its FP16 baseline. Same idea, opposite sign — which is the point: it was never the format, it's the kernel.

Left: process energy against decoded tokens for 45 runs on one RTX 4090, three arms, each fitting a straight line with R-squared of at least 0.9966; the fitted slopes are 4980 mJ/token for F16, 1898 for our Q4_0 and 1834 for the MLPerf Client Q4_0 file, and the intercepts of 171 to 378 J are the weight-loading term that differencing removes. Right: mean decode power against decode throughput, showing the quantized arms drawing more power, 296 and 319 W against 273 W, at 2.8 to 3.2 times the throughput.

Because energy is linear in token count across all three lengths (R² ≥ 0.9966), differencing a 576-token run against a 64-token one is a legitimate way to strip the weight load out; the fitted slope and the differenced value agree to the printed digit. That was an assumption in the first pass and is a measurement in this one.

Still not a result in the MLPerf sense. It is not an MLPerf result and carries no MLCommons review or endorsement — it is an independent measurement that happens to use the same model file, with GPU-package power rather than wall AC power, on one card in one session. Three things a reader should weigh: (1) llama.cpp's timing line still did not parse in our wrapper, so the token counts are inferred from --ignore-eos plus a zero exit code rather than read back from the runtime; (2) the NVML energy counter reads 14.9% higher than trapezoidal integration of the same run's 100 Hz power trace (per run; after differencing, which cancels the constant part, the gap falls to 0.2–4.3%) — the counter is the reported quantity and both columns are in the CSV; (3) perplexity was not re-measured — the +5.6% is carried over from the 2026-08-31 session on the same files and the same build. Full caveats and the raw rows are in the write-up; the complete archive — 45 raw run records, scripts, environment snapshot, model hashes — is deposited at 10.5281/zenodo.22295184.
Three weight-only quantizations of a 7-8B model on one RTX 4090: bitsandbytes LLM.int8() costs 105.8% more decode energy per token than FP16, bitsandbytes NF4 saves 39.0%, and llama.cpp GGUF Q4_0 saves 61.9%. Their perplexity costs, +1.19%, +12.06% and +5.60%, rank them differently again.

Each delta is against an FP16 baseline measured in its own session on an RTX 4090. The card is held constant; the model, runtime and generated-token count are not. Read the sign and the mechanism from this, not the exact magnitudes.

The energy axis of MLPerf Client is real — and it is not available at home

v2.0 does define power measurement, and defines it well: tools/power/README_MLPerf-client_Power-measurements.md specifies a Yokogawa-class power analyzer, the SPEC PTDaemon interface, a second "director" machine on the same network, NTP-synchronised clocks, and an EULA you request by email. tools/power/power_log_parser.py then computes tokens/Joule per prompt category with a bootstrap 95% CI, pools them, and reports the geometric mean, with warmup iterations excluded. That is a rigorous, auditable, wall-power number.

It is also a bench setup. The people the signed applications just enabled — someone with one laptop and no analyzer — can produce the tok/s figure and not the tokens/J figure. That gap is the only reason this page exists.

Mapping the two, honestly

MLPerf Client v2.0 powerEcoCompute
Unittokens/JoulemJ/token — the same quantity, tokens/J = 1000 / (mJ/token); the site now prints both
Measurement boundaryWall AC, whole system: CPU, DRAM, display, fans, PSU losses includedGPU package only (nvmlDeviceGetPowerUsage) — a lower bound on system energy, so our tokens/J is an upper bound on a wall-power tokens/J
InstrumentYokogawa analyzer + SPEC PTDaemon + director hostNVML software telemetry @ 10 Hz, integrated over the decode loop
Barrier to entryHardware, a second machine, an EULAOne command, one machine
Precisions coveredWhatever the vendor ships: Q4_0, int4, q4, NVFP4, FP8FP16 baseline plus NF4 / INT8 in the same session, which is the whole point
QualityIFEval, MMLU/TinyMMLU as separate harnessesTeacher-forcing perplexity of the quantized model and its FP16 baseline, same run, fixed corpus
StatisticsBootstrap CI per prompt category, geomean across categories, warmup excludedWarmup excluded; repeats where we have them (same-machine CV 0.6–3.9%)
StatusAn official MLCommons benchmark with certified resultsAn independent research protocol. certified_benchmark_result: false. Not an MLPerf result.
Do not compare the two numbers directly. A GPU-package joule and a wall-plug joule are different physical quantities, and on a laptop the gap is large — display, CPU and fans are not in ours. If you have both, report them as two rows, not as one ratio. Anyone presenting our number as an MLPerf Client power result is misreporting it, and so would we be.

Running both on the same machine

1 · Speed, from MLPerf Client

Signed app, GUI or CLI. NVIDIA GPUs on Windows x64 use the llama.cpp CUDA execution provider.

mlperf-windows.exe -c NVIDIA_llamacpp-CUDA_GPU.json

Gives geomean TTFT and 2nd+ token rate for Llama-3.1-8B-Instruct-Q4_0.

2 · Energy and quality, from the container

Same machine. Measures the quantized model and its FP16 baseline, then scores both for perplexity.

docker run --rm --gpus all \
  -v "$PWD/out:/workspace/outputs" \
  ghcr.io/hongping-zh/ecocompute-mlcube:latest \
  energy_estimate \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --params_b 8 --precision NF4 \
  --gpu_arch auto \
  --output_dir /workspace/outputs

Writes out/energy.json. No Docker? The native path works on Windows — the first independent replication of this project came from a Windows laptop.

One difference you must keep in view when reading the two side by side: the runtimes are not the same. MLPerf Client's NVIDIA path is llama.cpp with a GGUF Q4_0 file; the container is transformers + bitsandbytes NF4. Same idea (4-bit weight-only), different kernels, so the energy deltas are not interchangeable — the measurement at the top of this page is exactly how far apart they can land (−61.9% against +106% on one card). The llama.cpp numbers were produced by a standalone NVML wrapper around llama-cli, not by the container; folding a GGUF backend into the container, so that one command reproduces the whole table, is now the next thing on our list.

Why this matters more on a laptop than in a datacentre

Our published curves say the crossover — the model size above which 4-bit weight-only quantization starts to save energy rather than cost it — sits in the low single-digit billions of parameters on the cards we have measured, because at small sizes the throughput loss outweighs the memory-traffic saving. An 8B model on a client GPU sits close enough to that boundary that the sign of the effect is a measurement question, not a theory question. Which is why we would rather have your measurement than our extrapolation.

The first independent replication of this project makes the point: on an RTX 3050 Ti Laptop GPU running TinyLlama-1.1B in NF4, the contributor measured 16.2% less energy per token, where every card we own spends between 4.6% and 33% more at that size. Our own curve had nothing useful to say there — it is fitted on 7–14B data. See the write-up, or the gallery.

If you just installed MLPerf Client v2.0: you already have a machine that can produce the missing axis. Sixty seconds of container time turns "how fast is it" into "what did it cost". Submit an energy.json →

A note on what we would like to see upstream

Nothing in the above is a criticism of the power methodology — a PTDaemon-audited wall-power number is strictly better evidence than software telemetry, and it should stay the standard for submitted results. The gap is that there is currently no tier below it, so the home user gets nothing. An explicitly informative-only, non-submittable software-telemetry tier (NVML on Windows and Linux, powermetrics on macOS), labelled as not comparable to PTD, would let the population v2.0 just enabled see an energy figure at all.

We would bring a warning with that suggestion rather than only enthusiasm. In our own record, repeated runs on one machine agree to within a coefficient of variation of 0.6–3.9%, but the same five INT8 configurations measured on the same card in two different sessions differed by 100 to 140 percentage points — 30 to 50 times the run-to-run noise — and absolute joules per token drifted 12–17% between two consecutive days. The numbers are here. Any at-home energy tier has to be designed around that, which is an argument for reporting deltas against a same-session baseline rather than absolute joules.

Sources