Short, evidence-based notes on the energy cost of LLM inference — grounded in direct NVML power measurements across NVIDIA GPUs. Honesty over hype.
2026-09-07
We published an energy measurement and called it a probable overestimate in the same breath. This is the rerun, written as a protocol note: forced cooldown before every run, randomized order, --ignore-eos, a third token length to test the differencing assumption (R² ≥ 0.9966), and NVML's hardware energy counter — which reads 14.9% higher than integrating the same run's 100 Hz power trace. The headline moved 1.7 points; of the 50 minutes of GPU time, 45 were spent waiting for the card to cool down.
Read →
2026-08-31 · corrected 2026-09-03
MLPerf Client v2.0 shipped an already-quantized suite, so we measured the FP16 baseline it cannot produce. Inside one llama.cpp build on one RTX 4090, GGUF Q4_0 costs 61.9% less decode energy per token than GGUF F16 for +5.60% perplexity — while drawing more power, 296–319 W against 273 W, so the saving is throughput and nothing else. On the same card bitsandbytes LLM.int8() costs 106% more. Three weight-only quantizations of a 7–8B model, spanning +106% to −62%: the bit width does not predict the sign. Re-run 2026-09-03 over 45 runs with cooldowns, randomized order and a hardware energy counter; the provisional −63.6% was an overestimate and the correction is documented in the note.
Read →
2026-08-25
Someone else ran the container. On an RTX 3050 Ti Laptop GPU, TinyLlama-1.1B in NF4 saves 16.2% energy — where every card we own spends 4.6–33% more at that size, and where our own fitted curve was extrapolating six-fold beyond its anchors. The paired perplexity, meanwhile, reproduced our RTX 4090 measurement to four decimal places: quality travels between machines, energy does not.
Read →
2026-08-21
Five INT8 configurations re-run three times each on the same RTX 4090: the coefficient of variation of the energy delta is 0.6–3.9% — 30–50× smaller than the 100–140 point gap between two of our own sessions, so that gap is not sampling. Also, absolute joules drifted 12–17% lower overnight on the same host while the FP16-normalised delta held.
Read →
2026-08-20
Ten RTX 4090 configurations where the joules and the perplexity come from the same run. NF4 saves 39% energy at 7B but costs 12% perplexity; INT8 costs almost nothing in quality and never saves energy — a Δppl ≤ 1% filter rejects every NF4 row and admits nearly every INT8 one, the exact opposite of the energy ranking. Also: the same INT8 configuration measured 2.0–2.5× worse than in July, on the same card model and the same software stack.
Read →
2026-08-18 · 中文
在你自己的 GPU 上复现测量的逐步手册：三种运行方式（Docker 一行命令 / 租卡 bootstrap / 源码）、每个参数的作用、energy.json 各字段（含新增的困惑度质量块）、故障排查，以及每个数字的边界。A Chinese-language how-to for running the measurement container on your own GPU.
阅读 → PDF 版
2026-08-16 · 中文
中文版站点导览：四个标签怎么用、每个数字的测量边界（GPU 芯片功耗、单流 batch 1、逐点标注的 n）、该引用哪个 DOI，以及四条必须记住的诚实声明。A Chinese-language guide to the site, its data boundaries and its citable artifacts.
阅读 → PDF 版
2026-07-25
15 of 15 configurations measured with direct NVML on one Ada card, using the open MLCube container anyone can run. NF4 goes from a +36% penalty at 0.5B to break-even at 3B and a −28% saving at 7B; INT8 cost more energy at all five sizes tested (n = 1 each, bitsandbytes LLM.int8()), because throughput collapses.
Read →
2026-06-25
Weight-only NF4/INT8 quantization shrinks memory, but for small models it can raise inference energy by 25–55%. Here's what the measurements say, and how to find your crossover point.
Read →