"Abstractions leak, and temperature sets the speed" — research brief for Sun 20 Sep 2026 (week 38)

Generated Mon 21 Sep 2026 (week 39) 08:19 PDT on the Intel.

Answers the voice note of Sun 20 Sep 2026 13:32 PDT: "Here's an idea for a blog post. I can write something like abstractions leak, and I can show that the temperature actually determines the speed of some of the operations."

The headline, and it is measured, in your own JSON, and no page of yours mentions it yet.

On an A100-SXM4-40GB, your own 8192³ FP16 tensor-core GEMM runs at a sustained 1,216 MHz, not the 1,410 MHz boost clock — 13.7% below the number every "an instruction takes N cycles" calculation silently assumes. In wall-clock terms the same matmul takes 15.9% longer than the boost-clock model predicts.

Sustained clock is measured (nvidia-smi samples inside matmul/a100_8192_dtype_energy_results.json, run 2026-09-02). The 15.9% is derived from it by t ∝ 1/f.

And this appears to be new. The only public A100-under-load clock measurement found says the opposite — "core clocks went to 1410 MHz under load and stay there" past 350 W, no throttling (Chips and Cheese) — and no published A100 clock-vs-temperature or HGEMM clock-droop curve turned up at all (§2.1). So the post is not re-explaining a known effect; it is reporting a measurement that contradicts the one number in circulation. That raises the bar: §4's confirming run is mandatory, not optional.

And then the twist that makes it a post rather than a benchmark table: on the same chip, in the same run, at the same temperatures, two of the four kernels did not lose a single megahertz. Temperature is not the variable. Power is. Temperature only gets a vote because it spends part of the power budget — and your data says that vote is worth about 1%, not 14%.

1. What is already in your own trees

Everything in this section is from ~/git/sutro-problems on the Intel (the copy the doctrine says to use) and ~/git/nekko/et-soc1-prototyping — which is present on this Mac, contrary to the memory note that calls it M5-only. Nothing here is from the literature.

1.1 The clock ladder — measured, unpublished

~/git/sutro-problems/matmul/a100_8192_dtype_energy_results.json (dated 2026-09-02T04:41:15Z, A100-SXM4-40GB, 400 W cap, driver 580.95.05) runs the same 8192×8192×8192 GEMM in four datatypes, 5 trials × 5 s each, with nvidia-smi sampled every 100 ms. Its per-trial clocks_sm_MHz block has never been read by any report. Here it is:

Source: his own matmul/a100_8192_dtype_energy_results.json, extracted by reports/auto/2026-09/assets-temperature-leak/analyze_8192.py.

Read that table twice. The fp32 and b1 kernels hit exactly 1,410 MHz on every steady-state sample, at 54–60 °C. The fp16 and int8 kernels, at the same temperatures on the same board, lost 10–14% of their clock. The variable that separates them is board power: 270–340 W versus 370–397 W against a 400 W cap. Trial 1 of each stage reads low only because its 5 s window includes the ramp out of the 1,095 MHz idle state — that is the first-iteration-vs-steady-state artifact, worth 1.8% on fp32 and worth naming in the post as its own small leak.

Steady-state means, trials 2–5 (his data), and the wall-clock consequence (derived, t ∝ 1/f):

The arithmetic closes, which is the best evidence the clock number is real. 2·8192³ = 1.0995×10¹² FLOP. At the measured 4.3621 ms that is 252.1 TFLOP/s = 80.8% of the A100's 312 TFLOP/s dense FP16 datasheet peak — a gap everyone blames on cuBLAS. Scale the datasheet peak to the measured 1,216 MHz and it becomes 269.1 TFLOP/s, and the kernel is at 93.7% of it. Roughly two thirds of the "cuBLAS inefficiency" everyone talks about is not inefficiency at all; it is that the chip was never running at the clock the datasheet quotes. (Derived; arithmetic in reports/auto/2026-09/assets-temperature-leak/derive.py.)

1.2 The leakage slope — measured, four independent times

The same file brackets every stage with a 5 s idle window: 0% utilization, SM clock pinned at exactly 1,095 MHz, identical 2,351 MiB resident. Literally nothing changes between the two windows except how hot the die is. Power:

Source: his own matmul/a100_8192_dtype_energy_results.json. This is a leakage-current curve measured by accident, four times, in a benchmark that was not looking for it — same instructions (none), same clock, same voltage rail, +0.4 to +0.6 W per °C, about +5% of the idle floor for +7 °C.

The second board confirms it independently. mnist/submissions/small-qda-20260915/results/energy-netherlands.json has a 4,166-row columnar trace; its idle samples, also at a pinned 1,095 MHz, go 37.00 W @ 27 °C → 37.34 @ 28 → 37.58 @ 29 → 37.97 @ 30 → 39.27 @ 31, and then a second monotone run 39.05 @ 34 → 39.43 @ 35 → 39.69 @ 36. Two independent monotone ramps, same sign, same order of magnitude.

1.3 The board-to-board number that should scare him

Two A100-SXM4-40GB running the same small-QDA workload (mnist/submissions/small-qda-20260915/results/):

Two chips with the same part number differ by 55% in idle power (57.7 vs 37.5 W), and per his own reversible-networks pass >99% of a short MNIST run's joules are the idle floor — so the part of the abstraction that leaks catastrophically here is not seconds, it is joules. Meanwhile the speed difference is nearly nil: the cooler, 320 W-capped board was 1.19% slower on the PTX round and 0.11% faster on the matmul. And Canada's own round-to-round spread on the identical PTX round was 0.016797→0.017497 ms = 4.2%, which swamps the 1.19% between-board difference. His harness's noise floor is bigger than the effect he would be claiming at these power levels. Say so in the post; it is the sentence that makes readers trust the 15.9%.

1.4 The negative result, stated plainly

Across 4,123 busy NVML samples on three different A100 boards (1,597 + 1,288 + 1,238), spanning 29–47 °C and 70–220 W, the SM clock was 1,410 MHz in every single sample. Zero throttling events. Sources: mnist/submissions/medium-pca-qda-20260915/energy-audit/results-sxm40/independent.json, .../small-qda-20260915/results/energy-{canada,netherlands}.json; extracted by assets-temperature-leak/analyze_clock_temp.py and analyze_trace.py.

A post whose thesis is "temperature determines speed" has to survive 4,123 samples that say it doesn't. It survives by being restated correctly — see §3.

1.5 The fourth layer: a chip with no DVFS loop at all

His ET-SoC-1 work (~/git/nekko/et-soc1-prototyping, measurements 18–19 Sep on the AI Foundry cards) is the control experiment. From his own report page, verbatim: "Power rose about 3 W within each 12 s run as the chip warmed, so longer runs would read slightly worse." And the minion clock was steady at exactly 600 MHz in every sample on every page — the card exposes no voltage/frequency control he has found, and scripts/mmbench-power.py only snapshots DM_CMD_GET_MODULE_CURRENT_TEMPERATURE before and after because there is no clock to watch.

So: same physics, opposite leak. Heat raised power by ~3 W on a 30.6 W idle floor / 57–62 W load (≈5%) and changed the speed by exactly nothing, because nothing was closing a loop around it. The leak is not a property of silicon. It is a property of the control system you bolted onto the silicon. That is the strongest single paragraph available for the post, and it is his own measurement on a chip almost no reader has touched.

2. The physics and the specs — four layers, and they do not agree on direction

2.1 Layer 1 — GPU DVFS and power capping (the layer his data lives in)

The clock numbers. A100-SXM4-40GB: idle graphics clock 420 MHz, default application clock 1,095 MHz, max boost 1,410 MHz, memory 1,215 MHz, power configurable 100–400 W (Microway, DGX A100 review). A100-PCIe's datasheet quotes base 765 MHz / boost 1,410 MHz (NVIDIA A100 PCIe Product Brief PB-10137-001_v03). ⚠️ These two "base" numbers are different quantities — NVIDIA's datasheet base (765) is not NVML's default application clock (1,095). The post must say which it means; his idle samples sit at exactly 1,095, so his chip's floor is the application clock. The clock grid is 210–1,410 MHz in 15 MHz steps (Afzal et al., GROMACS Unplugged, arXiv 2510.06902), which matches his own 2026-09-15 pass's "81 clock steps." TDPs: 40 GB PCIe 250 W, 80 GB PCIe 300 W, both SXM4 variants 400 W (NVIDIA A100 datasheet r4).

NVIDIA states the mechanism in its own brief: the GPU "automatically and dynamically adjusts the GPU clock during runtime to optimize performance within the power cap and thermal limits" (PCIe Product Brief). The leak is documented behaviour, not a defect — which is a much better post than "I found a bug."

🛑 The correction that matters most. The A100-SXM4-40GB's real thresholds, from an actual nvidia-smi -q dump: GPU Max Operating Temp 85 °C · GPU Slowdown Temp 89 °C · GPU Shutdown Temp 92 °C · Memory Max Operating Temp 95 °C (NVIDIA/k8s-device-plugin issue #343). 85 °C is the software throttle target, not the hardware slowdown. His hottest sample in any run is 66 °C — nineteen degrees below even the software target. No thermal mechanism was engaged in any of his data. Anything the post says about heat must route through the power path.

What the NVML flags actually mean (verbatim, NVML Clocks Event Reasons):

• SwPowerCap — "The clocks have been optimized to ensure not to exceed currently set power limits." This is the bit his fp16 run was almost certainly flying.
• SwThermalSlowdown — clocks reduced so temperature does not exceed GPU Max Operating (85 °C) or Memory Max Operating (95 °C).
• HwThermalSlowdown — "HW Thermal Slowdown (reducing the core clocks by a factor of 2 or more) is engaged." That sentence is his citation for "a real thermal event halves your clock" — a 2× leak, not a 14% one, and it fires at 89 °C.
• HwSlowdown / HwPowerBrakeSlowdown — same "factor of 2 or more" language.

⚠️ The premise check, and the good news. I could not find any published A100 clock-vs-temperature curve, or any published measurement of a cuBLAS HGEMM clocking down from 1,410 MHz. The only public A100 load measurement found says the opposite: on an SXM4 A100 the "core clocks went to 1410 MHz under load and stay there," past 350 W, with no throttling (Chips and Cheese — the same article shows an H100 PCIe at 350 W settling to 1,395 MHz, "just under 80% of its maximum boost clock"). Power-capping studies likewise find the A100 insensitive until caps go far below the board limit — performance holds down to ~150 W for small kernels and ~250 W for large ones (GROMACS Unplugged, arXiv 2510.06902), and a separate sweep puts the A100's minimum-energy point at 990 MHz with a transition frequency of 1,005 MHz (arXiv 2607.00819).

Do not read this as "his data is wrong." Read it as the reason to publish. The public claim is that an A100 holds boost; his own 100 ms samples say a 400 W-class FP16 tensor-core GEMM does not, and that fp32 and b1 on the same board do. Chips and Cheese did not run a full-cap HGEMM; he did, twice, five trials each, by accident. That gap is the entire novelty of the post — and it is also why §4's clocks_throttle_reasons run is not optional. One confirming run turns a blog post into a citable measurement; one disconfirming run saves him from being wrong in public.

2.2 Layer 2 — DRAM refresh doubles above 85 °C

The abstraction here is "a DRAM read costs N nanoseconds." It is temperature-dependent by specification.

The rule, verbatim from vendor datasheets: - DDR4 — "64ms, 8192-cycle refresh up to 85°C / 32ms, 8192-cycle refresh at >85°C to 95°C", giving tREFI 7.8125 µs → 3.90625 µs, and "2x refresh rate when TC exceeds 85 °C; 4X when TC exceeds 95 °C" (Micron 4 Gb DDR4 datasheet). - DDR5 — "JEDEC specifications require the refresh rate to double when TC exceeds 85 °C", tREFI 3.9 µs → 1.95 µs (Micron 16 Gb DDR5, die rev D). - Academic statement of the same rule and the 1x/2x/4x ladder: Bhati, Chang, Chishti, Lu, Jacob, DRAM Refresh Mechanisms, Penalties, and Trade-Offs, IEEE Trans. Computers 65(1), 2016 (PDF).

What it costs. Same paper, measured/simulated: refresh already costs up to 11.4% IPC in high-bandwidth workloads, >30% IPC and 20–30% of DRAM energy at 32 Gb density, and 13–23.5% DRAM latency degradation in low-bandwidth workloads. Derived (datasheet arithmetic, not a citation): at tRFC1 = 350 ns on 8 Gb DDR4, refresh occupies 350/7800 = 4.5% of rank time cold and 350/3900 = 9.0% hot; at 16 Gb (tRFC1 = 550 ns) it is 7.1% → 14.1%. So crossing 85 °C hands back roughly 4–7 points of memory bandwidth, silently, with no counter anywhere in the software stack reporting it.

⚠️ HBM2e is not verified. JESD235 is paywalled and no public HBM2e tREFI table turned up; only secondary sources assert the same behaviour. The post must not put a number on the A100's HBM2e refresh. State the DDR4/DDR5 rule, note that HBM2e's memory max-operating temp is 95 °C per the nvidia-smi dump above, and stop there.

This is also a better layer than it looks, because it is the one that cannot be hidden by a control loop: there is no frequency to lower, only time that disappears.

2.3 Layer 3 — the transistor, where the sign flips

Delay rises with temperature at nominal voltage. Propagation delay increases 19.6% at 180 nm and 54.5% at 65 nm going from 25 °C to 125 °C (Kumar & Kursun, ISCAS 2006). That is the folk model, and it is correct — at nominal V_dd.

Leakage rises with temperature, roughly exponentially, because threshold voltage is complementary-to-absolute-temperature (subthreshold conduction). ⚠️ "Doubles every 10 °C" has no strong primary source. The defensible phrasing is "roughly doubles every ~10 °C"; the hard, citable number is Borkar's: leakage was already ~60% of processor energy at 65 nm, with leakage current rising 7.5× per generation (Borkar, DAC 1999 / ACM). His own +0.49 W/°C over 35–47 °C (§1.2) is the same curve, measured on an A100 he did not pay for. Nobody publishes that; it would be the first original figure in the post after the clock table.

🛑 And then the sign flips. Below the zero-temperature-coefficient supply voltage, a hotter transistor is a faster transistor: threshold-voltage reduction beats mobility degradation. The ZTC crossover sits 45–53% below nominal at 180 nm and 68–69% below nominal at 65 nm — roughly 0.85–1.0 V and ≈0.31 V respectively (Kumar & Kursun, Microelectronics Journal 2007; the voltage conversion from the 1.8 V / 1.0 V nominals is arithmetic, not quoted).

This is not theory. It is measured on shipping silicon and already being exploited:

• Exynos 4412 (28 nm), 5433 (20 nm) and 7420 (14 nm), swept 10 → 80 °C: minimum operating voltage falls as junction temperature rises at under-drive frequencies, worth 12.5–37.5 mV at high frequency and 25–50 mV at low frequency at a 40 °C average (Park & Cha, T-DVS).
• On an AMD A10-8700P, temperature inversion yields >5% V_dd reduction, rising to >8% near threshold (Zu et al., Ti-states, MICRO-49 2016).
• At the architecture level on FinFET, TEI-aware DVFS buys 15.70% throughput or 31.26% energy over a TEI-agnostic governor (Cai & Marculescu, TEI-Turbo, ICCAD 2015).

⚠️ Could not find an open-access "at 0.6 V, delay improves X% per 10 °C" one-liner; the ZTC offsets above are the defensible substitute.

So the honest statement of the physics is: the derivative of speed with respect to temperature changes sign depending on your supply voltage. A post titled "hot means slow" is asserting a high-voltage special case as a law. A post that says "the sign of dSpeed/dT is a design choice you made when you picked V_dd" is saying something true at every node, and it is the sentence that will get it shared by circuit people.

2.4 Layer 4 — the CPU in front of him, where it is most naked

The cleanest consumer example is the machine this brief was generated on.

• 2019 8-core MacBook Pro (Core i9-9880H): bursts to 5.0 GHz, settles to 2.9–3.0 GHz over 10 back-to-back Cinebench R20 runs, at ~94 °C with peaks near 100 °C (AppleInsider). A 40% sustained clock loss — three times bigger than the A100's.
• And the best single fact in this whole brief: that chip's top turbo bin is explicitly gated on a thermometer. Intel Thermal Velocity Boost gives 4.8 GHz single-core only while the CPU is below 50 °C (base 2.3 GHz, all-core 4.1 GHz) (Notebookcheck). The ISA's instruction-cycle count is, by documented design, a function of a temperature sensor. That is the whole post in one sentence, and it is shipped in millions of laptops.

⚠️ Not verified: any AVX-offset sustained clock for this specific mobile part (AVX offsets are documented for Xeon and desktop SKUs). Do not claim an AVX-512 number for a MacBook.

3. Where "temperature determines speed" is wrong, or subtler than it sounds

This section is the post. Everything above is setup.

(i) On the A100, the binding constraint was the power cap, not the thermal limit. His hottest sample across every run is 66 °C (int8 trial 5), against a GPU Max Operating Temp of 85 °C, a hardware slowdown trip at 89 °C and shutdown at 92 °C (§2.1). He was 19 °C below the point where the software throttle even begins — the chip was never thermally throttled once, in any run, ever. What throttled was SW_POWER_CAP: the board asked for more than 400 W and the DVFS controller took clock away until it fit. Temperature enters only through the leakage term, i.e. through how much of the 400 W the die wastes on being hot. So the honest headline is not "temperature determines speed." It is:

The power budget determines speed, and temperature quietly spends part of the budget.

(ii) And his own data bounds how much. Measured leakage slope ≈ 0.49 W/°C (§1.2). The fp16 stage heated 55.6 → 63.8 °C across trials 2–5, i.e. +8.2 °C, i.e. ≈4.0 W of a 400 W budget = 1.0% — which, at fixed voltage, predicts a 14 MHz clock loss. Measured drift trial 2 → trial 5: −1.2 MHz (−0.10%), well inside the 100 ms sampler's noise. So the naive leakage-to-clock prediction is ~10× too big, and his data can only bound the temperature-driven part at under about 1% over 8 °C — it cannot resolve it. (Derived; assets-temperature-leak/derive.py.) The 13.7% clock loss is essentially all workload-power, and essentially none of it is the warm-up.

This is the number that keeps the post honest. Without it the post is "look, the clock moved"; with it the post is "here is exactly how much of the leak is temperature, and it is 1 part in 14."

(iii) Closed-loop control can hide the leak completely — or invert it. Once the controller is regulating power, it holds power constant by construction, and a hotter die simply buys you fewer megahertz for the same watts; the leak moves from the time axis to the frequency axis. Regulate frequency instead (nvidia-smi -lgc, or the ET-SoC-1's fixed 600 MHz) and the leak moves entirely to the energy axis — time becomes perfectly predictable and joules become the thing that varies. You cannot remove the leak, you can only choose which axis it comes out on. That is a much better thesis than "abstractions leak" and it is the one his three datasets actually support.

(iv) The sign can flip. Under inverted temperature dependence, at low enough supply voltage a hotter transistor is a faster transistor. A post that says "hot = slow" is stating a high-voltage special case as a law. Near-threshold and IoT-class silicon runs on the other side of the ZTC point. See §2.3.

(v) A constant-temperature environment erases the effect. A datacentre holding a fixed inlet temperature with a closed-loop fan curve makes the die temperature a controlled variable, and the benchmark becomes reproducible again. His laptop cannot do this; Modal's A100 fleet mostly can. That is why the two boards in §1.3 differ by 55% in idle power and by ~0% in speed.

(vi) The one place the abstraction is most broken is not speed at all. It is energy. Two identically-named A100s, 37.5 W vs 57.7 W idle; >99% of a short run's joules are that floor; the ET-SoC-1's floor is over half its load power. "An instruction costs N picojoules" leaks far worse than "an instruction takes N cycles." If he wants the strongest version of this post, that is the punchline.

4. The experiment — one afternoon, on hardware he already has

4.1 The one to run: a pinned-clock ladder on the Modal A100

His own 2026-09-15 binary-tensor-cores pass already established the knobs exist: "A100 SXM exposes 81 clock steps 210→1410 MHz and power caps 100–400 W." The experiment is to stop letting the controller choose, and measure t(f) directly.

# Inside the existing Modal A100-40GB image (the one in matmul/modal_a100_dtype_energy.py).
# Needs the container to be able to talk to nvidia-smi persistence/admin; if -lgc is refused,
# fall back to the -pl ladder in 4.2, which is the same chart with a different x-axis.
nvidia-smi -pm 1
for F in 1410 1305 1215 1095 900 705 510; do
  nvidia-smi -lgc $F,$F                     # pin both floor and ceiling
  python bench.py --dtype fp16 --m 8192 --n 8192 --k 8192 \
                  --seconds 20 --tag clk$F
done
nvidia-smi -rgc

bench.py is a 40-line edit of his existing harness — keep nvmlDeviceGetTotalEnergyConsumption (his 2026-09-15 pass cites arXiv 2312.02741: power.draw samples only ~25% of runtime on A100, so never integrate it for this) and add a per-iteration log:

Decode the throttle-reason bitmask (§2.1) and print which flag was set. This single field settles the whole post's causal claim in one run, and it costs one NVML call.

🛑 A one-line bug in his existing harness, worth fixing before anything else. Nine of his MNIST harnesses already call nvmlDeviceGetCurrentClocksThrottleReasons — but only inside the telemetry_before / telemetry_after snapshots. All 90 recorded masks across 19 result files decode to NONE, because every one of them was taken while the GPU was idle (assets-temperature-leak/throttle_reasons.py). The field he needs is being read at exactly the two moments it cannot say anything. Move that call into the 100 ms sampler loop and the 2 September run answers its own question retroactively — and so does every future run, for free.

Start each point from a cold board (60 s idle, confirm T ≤ 40 °C), then run 20 s hot. That gives, for free, the warm-up transient inside each point.

4.2 The fallback if the container refuses -lgc

Same script, ladder the cap instead: nvidia-smi -pl 400 350 300 250 200 150 at fixed fp16 8192³. The resulting curve is clock vs power budget — which is exactly the mechanism §1.1 caught by accident, now swept deliberately, and it also hands him the Zeus-style energy-vs-slowdown tradeoff curve his 2026-09-15 pass quoted from the literature but never measured himself.

4.3 The zero-cost version

The headline chart can be drawn today, from JSON he already owns, with no GPU time at all: 20 points (4 stages × 5 trials) of mean board power vs sustained SM clock, from matmul/a100_8192_dtype_energy_results.json. If the afternoon never happens, the post still ships.

4.4 Expected shape, and what falsifies the thesis

• Expected: per-iteration ms is a clean straight line in 1/f across the pinned-clock ladder (R² > 0.99 for a compute-bound GEMM — HBM stays at 1,215 MHz throughout, so nothing else moves). At the unpinned 1410 setting, the board will fail to hold 1410 under fp16 and settle near 1,216 MHz, reproducing §1.1. Within each 20 s point, temperature climbs ~8 °C and clock falls < 1%.
• Falsified if: with the clock pinned and the cap slack, per-iteration time still varies with die temperature by more than the ~0.3% sampling noise. Then something other than DVFS is temperature-coupled and the post has a real physics finding instead of a control-system finding.
• Also falsified — the version he should hope for: if clocks_throttle_reasons reports HW_THERMAL_SLOWDOWN rather than SW_POWER_CAP at any point, then the 13.7% is thermal and the original voice-note thesis is literally right. His §1 data says to expect SW_POWER_CAP.
• Null result to be ready for: the fp32 and b1 stages will sit at 1410 MHz forever. The post must explain why two kernels on one chip disagree — that is the interesting part, not a blemish.

5. The post

Title: "Your matmul is 14% slower than the datasheet, and the datasheet is not lying" (alternates: "The clock is not a constant"; "An instruction takes N cycles, and other lies"; "Temperature is a variable in your inner loop")

Outline (7 bullets):

1. The abstraction. Every roofline, every "FLOPs ÷ peak" estimate, every CUDA occupancy calculator assumes a fixed clock. State it as the load-bearing assumption it is: time = cycles ÷ f, and f is a number on a datasheet.
2. The receipt, and the disagreement. His own 8192³ GEMM: fp32 and b1 hold 1,410 MHz for five straight trials; fp16 and int8 sit at 1,216 and 1,274 MHz on the same board at the same temperature. One JSON file, four kernels, two answers — and the published claim is that an A100 holds boost under load. Lead with the conflict; it is the reason to read on.
3. The 80.8% → 93.7% reveal. The famous "cuBLAS only hits 80% of peak" gap mostly evaporates once you divide by the clock the chip actually ran at. This is the bullet people will quote.
4. So what is the real variable? Not temperature — power. Walk the 400 W cap, show the four kernels ordered by watts and the clock falling monotonically with them.
5. Then where does temperature come in? The accidental leakage curve: +0.49 W/°C at 0% utilization and a pinned 1,095 MHz, measured four separate times. Heat is a tax on the power budget, and he can price it: ~4 W out of 400, ≈1% — a fourteenth of the total leak.
6. Three more floors of the same building. DRAM refresh doubling above 85 °C (§2.2); leakage doubling ~every 10 °C (§2.3); and the sign flip — at near-threshold voltage, hot silicon is faster (§2.3). The abstraction leaks at every layer, and the leaks do not even agree on direction.
7. The control-system punchline. ET-SoC-1, running a fixed 600 MHz with no DVFS loop: warming it up moved power by ~3 W and speed by zero. You cannot delete the leak; you choose whether it exits through seconds, megahertz, or joules. Close on the one that actually costs him money: two identical A100s whose idle floors differ by 55%, in a workload where >99% of the joules are the floor.

The one chart the post needs:

• X axis: mean board power during the trial, W (100 ms nvidia-smi samples, 5 s window).
• Y axis: sustained SM clock, MHz. Fix the range 1,000–1,450 so 1,410 sits near the top.
• Series (4): one colour per datatype — fp32_strict, b1 AND+POPC, int8 tensor, fp16 tensor — 5 points each (one per trial). Marker fill encodes mean die temperature (a light→dark ramp, 50 → 66 °C), which is the visual argument: dark and light markers sit on top of each other at every power level. Colour says "the temperature axis does nothing here."
• Annotations: a dashed horizontal line at 1,410 MHz labelled "what your FLOPs ÷ peak calculation assumes"; a dashed vertical line at 400 W labelled "board power cap"; and one callout arrow on the fp16 cluster reading "−13.7% clock, +15.9% wall-clock time."
• Second panel (small, beneath, shares nothing): the leakage curve — X = die temperature 35–47 °C, Y = board power 65–72 W, four two-point segments (one per stage's idle-before/idle-after pair), with the fitted +0.49 W/°C slope drawn through them. Caption: 0% utilization, clock pinned at 1,095 MHz — this is the die doing nothing, more expensively.
• Build it with matplotlib in the existing report pipeline; system-theme HTML per repo doctrine.

6. Reproducing this brief

Scripts are durable, in reports/auto/2026-09/assets-temperature-leak/:

• analyze_8192.py — the clock/power/temperature tables in §1.1 and §1.2, from ~/git/sutro-problems/matmul/a100_8192_dtype_energy_results.json.
• analyze_trace.py — the Canada/Netherlands comparison in §1.3.
• analyze_clock_temp.py — the 4,123-sample negative result in §1.4.
• derive.py — every derived number in §1.1(iii) and §3(ii).
• throttle_reasons.py — decodes the 90 NVML throttle masks his MNIST harnesses already record, and shows all 90 are NONE because all 90 are idle-window snapshots (§4.1).

Verified vs derived. Measured, from his own files: every SM clock, board power, die temperature and millisecond in §1; the +0.39…+0.59 W/°C leakage slopes; the two boards' idle floors and caps; the ET-SoC-1 ~3 W warm-up and fixed 600 MHz. Derived by arithmetic, labelled as such: the 15.9% and 10.6% time penalties, the 3.7626 ms boost-clock model, the 80.8% / 93.7% peak fractions, and the 4.0 W / 1.0% / 14 MHz leakage-budget chain. From the literature, cited in §2: everything in that section and nothing outside it.

Total life satisfaction: the voice note was a hunch on a Sunday afternoon. The hunch was 86% right and the 14% that was wrong is the better post — and the whole thing was already sitting in a JSON file he generated on 2 September and never opened. The cheapest version of this ships today, from data he already owns, with no GPU time and no literature survey (§4.3).