It was never the format. It's the kernel.

MLPerf Client v2.0 shipped with signed Windows and macOS builds, agentic tasks, and a much wider hardware matrix — a lot of people can now benchmark a local LLM on their own machine without building anything. What comes out is speed: time to first token, tokens per second. Its power path is a different animal, needing a Yokogawa analyzer, SPEC PTDaemon and a second machine, so in practice a home run gives you throughput and nothing about joules.

We wrote a page about that gap last week, and it ended with a promise: our own energy numbers were measured with transformers + bitsandbytes NF4, while MLPerf Client's NVIDIA path is llama.cpp with a GGUF Q4_0 file, so the two could not be read against each other; putting a GGUF path next to ours was "the next thing on our list". This note is that measurement. It came back more interesting than we expected.

What we ran

One rented RTX 4090 (Ada, 24 GB, 450 W limit), one llama.cpp build (b10643-192067b72, CUDA 12.x), one model, three GGUF files:

• Llama-3.1-8B-Instruct-F16.gguf — the FP16 baseline that a benchmark suite of already-quantized models cannot give you;
• ours-Q4_0.gguf — quantized by us from the same weights;
• Llama-3.1-8B-Instruct-Q4_0.gguf — the file MLPerf Client v2.0 itself pulls for its NVIDIA llama.cpp path.

Each file was run at -n 64 and -n 576, three times each, batch 1, -ngl 99 -fa 1 --no-mmap -c 2048 --temp 0 --seed 1234, with GPU package power sampled from NVML at 10 Hz around the whole llama-cli process. Eighteen runs. Quality is a separate llama.cpp perplexity pass over the WikiText-2 raw test set, 564 chunks at context 512, which touches the energy figure not at all.

The two token counts are the point. A raw process energy total includes loading the weights — 16 GB for F16 against 4.7 GB for Q4_0 — which hands the quantized arm a structural head start that has nothing to do with decoding. Subtracting the 64-token run from the 576-token run leaves the energy of exactly 512 additional decoded tokens, with load and prefill cancelling out. Every number below is that difference.

The numbers

Decode-only, by differencing the 576- and 64-token runs; n = 3 per cell. SD of the differenced energy is 7–52 J (0.8–2.1% of the value). Perplexity is ±0.047–0.050 (1σ) on 564 WikiText-2 chunks; absolute values depend on the corpus and tokenizer and are not comparable with published WikiText numbers — only the within-row delta is. Raw rows: rtx4090_llamacpp_gguf_2026-08-31.csv, aggregates in the .summary.csv beside it.

The saving is throughput. It is not lower power.

Look at the decode power column before the energy column. All three arms pull 296–310 W while decoding — within a few percent of each other, on a card capped at 450 W. The GPU is about equally busy in all three cases. What changes is that Q4_0 moves a quarter of the bytes per token and runs at 2.6–2.7× the tokens per second. Energy per token is power divided by throughput, so the entire 63.6% is the throughput term.

That is the same mechanism, running the other way, that we reported for INT8: there too package power barely moved (74–91 W), and throughput fell to 0.63–0.72× of FP16, so energy per token went up. Which is the whole argument of this note:

Three implementations of "keep the activations, shrink the weights", one card, and the answer to "does quantizing save energy?" is +106%, −39% and −64% depending only on which kernel executes the result. LLM.int8() pays for its outlier-preserving mixed-precision decomposition in decode throughput; llama.cpp's Q4_0 path is a memory-bandwidth win on a bandwidth-bound workload and collects nearly the full ratio. A number like "4-bit" or "8-bit" predicts the memory footprint. It does not predict the sign of the energy effect, and anyone quoting an energy saving without naming the runtime is quoting a property of their kernel.

The quality axis does not rescue the ranking either. Ordered by energy the winner is llama.cpp Q4_0; ordered by perplexity damage it is LLM.int8() at +1.19%, which is also the worst on energy. There is no single "efficiency" ordering here, which is why this site publishes the two columns side by side and refuses to fuse them into one score.

A zero result worth publishing: our Q4_0 and MLPerf Client's are the same file, in effect

The two Q4_0 arms differ by 0.0002 in final perplexity — 7.7364 against 7.7366. Chunk by chunk across all 564, the largest absolute difference is 0.0003, and 472 of them agree to three decimals. On energy they differ by 1.4–2.3%, which is inside our run-to-run spread, and the sign of that gap flips depending on whether the cold-start runs discussed below are excluded. Both variants are in the summary CSV, deliberately.

So: we are not claiming our quantization is more efficient than the one MLPerf Client ships. It isn't, within this measurement. What the comparison buys is provenance — the numbers in the table are not an artifact of one particular GGUF file, and anyone who quantizes Llama-3.1-8B to Q4_0 themselves should land where the shipped file lands.

What this does not show

• This is not an MLPerf result and carries no MLCommons review or endorsement. It is an independent research measurement that happens to use the same model file. MLPerf Client's own power methodology measures wall AC power for the whole system with external instrumentation; we measure GPU package power via NVML. Those two boundaries are not interchangeable and the numbers must never be subtracted from each other.
• n = 3, one card, one session. The site's own repeatability note puts run-to-run CV at a few percent but the gap between two of our sessions at 100+ points; a 63% effect is far larger than either, but the third digit is not defensible.
• There is a real cold-start artifact in this sweep. The driving script runs F16 → ours → MLPerf in a fixed order with no cooldown, so 16 of the 18 runs began at 145–248 W of thermal and power tail left by the run before. The two that began near idle (22 W) are exactly the two low outliers in the short runs — f16 n64 r1 at 550.9 J against 679/766 J, and q4ours n64 r1 at 178.5 J against 317/323 J. The headline uses the warm runs only; the CSV flags every run with a cold_start column and the summary carries the all-runs variant too (−62.5% and −63.6%, i.e. the two Q4_0 arms swap places). A rerun with cooldowns and randomized order is needed before these become anchors.
• We cannot yet prove the token counts. The wrapper captured stdout into the report but every field came back empty, so llama.cpp's own llama_perf_context_print — the actual n_eval — is missing. That matters: against the RTX 4090's ~1008 GB/s, the F16 arm's 63.8 tok/s implies about 101% of theoretical memory bandwidth, which is impossible, so the F16 runs most likely stopped early on EOS and generated fewer than the requested tokens. If so the true F16 mJ/token is lower than 4815 and the 63.6% is an overestimate. The Q4_0 arms sit at a believable ~80% of bandwidth. This is the single reason the numbers here are labelled provisional, and fixing it is the first item on the rerun.
• Decode phase, batch 1, single stream, one prompt length. No prefill-heavy or agentic workload, no concurrency, no CPU, DRAM, PSU, PUE or CO₂e.
• The cross-backend comparison in the figure holds the card constant but not the model (Qwen2-7B against Llama-3.1-8B), the runtime, or the generated-token count. Read the sign and the mechanism from it, not the exact magnitudes.

If you are running MLPerf Client v2.0 this week

The two things measure different quantities and compose rather well on one machine. Run the signed client for speed on the workloads MLCommons has defined; run the container for the joules and the perplexity of a precision pair you choose, with the FP16 baseline measured in the same session:

docker run --rm --gpus all -v "$PWD/out:/out" \
  ghcr.io/hongping-zh/ecocompute-mlcube:latest \
  --model Qwen/Qwen2-7B --precision NF4 --output_dir /out

Keep the boundary in view when you put the two reports side by side: client tokens/second is a whole-system, vendor-optimized number; our joules are GPU-package only. And the container's current backend is transformers + bitsandbytes, so it reproduces the method of this note, not the llama.cpp rows — a GGUF backend, which is what would let anyone reproduce the table above with one command, is now the top of our list. All eighteen runs are in the per-run CSV linked above, with the undifferenced process energy, the first and peak power sample and a cold_start flag on each, so you can redo the arithmetic differently; drop your own result on the replications page, especially if it disagrees.