Notes & blog

Short, evidence-based notes on the energy cost of LLM inference — grounded in direct NVML power measurements across NVIDIA GPUs. Honesty over hype.

2026-07-25

We ran our own container on a rented RTX 4090: NF4 crosses over at 7B, INT8 never wins

15 of 15 configurations measured with direct NVML on one Ada card, using the open MLCube container anyone can run. NF4 goes from a +36% penalty at 0.5B to a −29% saving at 7B; INT8 costs more energy at every size, because throughput collapses.

Read →

2026-06-25

When not to quantize: the small-model energy penalty

Weight-only NF4/INT8 quantization shrinks memory, but for small models it can raise inference energy by 25–55%. Here's what the measurements say, and how to find your crossover point.

Read →