Last updated 2026-08-26 · written against
mlcommons/mlperf_client at
version.json = 2.0.0 · announcement:
mlcommons.org ·
signed downloads
MLPerf Client v2.0 is a good release: signed Windows and macOS applications, a CLI as well as a GUI, agentic and image-generation tests, and far broader hardware coverage. Its headline is that you can now generate results on your own machine instead of in a lab.
Those results are performance results — geomean time to first token and 2nd+ token generation rate. This page exists because two questions are left over, and people arrive here asking them:
1 · What did quantization cost in energy? Every LLM configuration shipped in v2.0
runs an already quantized model — Llama-3.1-8B-Instruct-Q4_0.gguf on the llama.cpp CUDA
path, *_ov-int4-* on OpenVINO, *-mlx-q4 on Apple MLX, and nvfp4 /
fp8 / quanto_int8 for image generation. There is no FP16 configuration anywhere in the
suite, which is reasonable — it benchmarks what vendors actually ship — but it means the difference between
quantized and unquantized is not a quantity the suite produces.
2 · What did it cost in quality, in that same run? v2.0 has accuracy harnesses
(tools/accuracy/ifeval, tools/accuracy/mmlu), but they score a model, not a delta
against that model's own FP16 baseline on the same machine in the same session.
Both are exactly what the EcoCompute container measures, in software, on hardware you already own. Neither project replaces the other: MLPerf Client is a benchmark with certified results and a submission process; this is a research protocol with none of that.
v2.0 does define power measurement, and defines it well:
tools/power/README_MLPerf-client_Power-measurements.md specifies a Yokogawa-class power analyzer, the
SPEC PTDaemon interface, a second "director" machine on the same network, NTP-synchronised clocks, and an EULA you
request by email. tools/power/power_log_parser.py then computes tokens/Joule per prompt
category with a bootstrap 95% CI, pools them, and reports the geometric mean, with warmup iterations excluded.
That is a rigorous, auditable, wall-power number.
It is also a bench setup. The people the signed applications just enabled — someone with one laptop and no analyzer — can produce the tok/s figure and not the tokens/J figure. That gap is the only reason this page exists.
| MLPerf Client v2.0 power | EcoCompute | |
|---|---|---|
| Unit | tokens/Joule | mJ/token — the same quantity, tokens/J = 1000 / (mJ/token); the site now prints both |
| Measurement boundary | Wall AC, whole system: CPU, DRAM, display, fans, PSU losses included | GPU package only (nvmlDeviceGetPowerUsage) — a lower bound on system energy, so our tokens/J is an upper bound on a wall-power tokens/J |
| Instrument | Yokogawa analyzer + SPEC PTDaemon + director host | NVML software telemetry @ 10 Hz, integrated over the decode loop |
| Barrier to entry | Hardware, a second machine, an EULA | One command, one machine |
| Precisions covered | Whatever the vendor ships: Q4_0, int4, q4, NVFP4, FP8 | FP16 baseline plus NF4 / INT8 in the same session, which is the whole point |
| Quality | IFEval, MMLU/TinyMMLU as separate harnesses | Teacher-forcing perplexity of the quantized model and its FP16 baseline, same run, fixed corpus |
| Statistics | Bootstrap CI per prompt category, geomean across categories, warmup excluded | Warmup excluded; repeats where we have them (same-machine CV 0.6–3.9%) |
| Status | An official MLCommons benchmark with certified results | An independent research protocol. certified_benchmark_result: false. Not an MLPerf result. |
mlperf-windows.exe -c NVIDIA_llamacpp-CUDA_GPU.json
docker run --rm --gpus all \ -v "$PWD/out:/workspace/outputs" \ ghcr.io/hongping-zh/ecocompute-mlcube:latest \ energy_estimate \ --model meta-llama/Llama-3.1-8B-Instruct \ --params_b 8 --precision NF4 \ --gpu_arch auto \ --output_dir /workspace/outputs
Our published curves say the crossover — the model size above which 4-bit weight-only quantization starts to save energy rather than cost it — sits in the low single-digit billions of parameters on the cards we have measured, because at small sizes the throughput loss outweighs the memory-traffic saving. An 8B model on a client GPU sits close enough to that boundary that the sign of the effect is a measurement question, not a theory question. Which is why we would rather have your measurement than our extrapolation.
The first independent replication of this project makes the point: on an RTX 3050 Ti Laptop GPU running TinyLlama-1.1B in NF4, the contributor measured 16.2% less energy per token, where every card we own spends between 4.6% and 33% more at that size. Our own curve had nothing useful to say there — it is fitted on 7–14B data. See the write-up, or the gallery.
Nothing in the above is a criticism of the power methodology — a PTDaemon-audited wall-power number is strictly
better evidence than software telemetry, and it should stay the standard for submitted results. The gap is that
there is currently no tier below it, so the home user gets nothing. An explicitly informative-only,
non-submittable software-telemetry tier (NVML on Windows and Linux, powermetrics on macOS), labelled
as not comparable to PTD, would let the population v2.0 just enabled see an energy figure at all.
We would bring a warning with that suggestion rather than only enthusiasm. In our own record, repeated runs on one machine agree to within a coefficient of variation of 0.6–3.9%, but the same five INT8 configurations measured on the same card in two different sessions differed by 100 to 140 percentage points — 30 to 50 times the run-to-run noise — and absolute joules per token drifted 12–17% between two consecutive days. The numbers are here. Any at-home energy tier has to be designed around that, which is an argument for reporting deltas against a same-session baseline rather than absolute joules.
tools/power/README_MLPerf-client_Power-measurements.md and tools/power/power_log_parser.py in that repositorydata/configs/vendors_default/ in that repository