10k-watt GPU meet 40-watt lump of meat

Fun title, I'll bite:

- not aware of any 10 kilowatt GPUs in this world, certainly not in the sense where a GPU = a single chip at least, which is I'd say what most everyone would associate to - the author probably thinks of the DGX H100 (obviously), which is an 8 GPU cluster

- an NVIDIA H100 pulls 400 or 700 watts maximum depending on the SKU [0] (in the DGX, the per-GPU "share" then works out to 1250 watts of course)

- a single H100 can handle many tens of concurrent chat sessions "in real time" using batched inference [1]

- therefore, the amortized instantaneous power per "virtual brain" is going to be strictly less than the human brain's 40 watts; even using the DGX numbers, this bar is cleared with "just" 32 concurrent sessions per GPU.

So as far as I'm concerned, maybe this is not a comparison people should be drawing up. These models may not be anywhere close to the human brain in many respects, but use more power, they do not. We're just scaling them out like there's no tomorrow.

Not even water use seems to be a good example by the way. Random paper by Google [2] that came up when searching for this suggests a water use of 1.15 liters per kWh. Apply this to the DGX case laid out above, that's 1.125 liter per day per "virtual brain". As everyone probably knows, a real person needs more like 2-3 liters of water a day in comparison (admittedly, serving more than just the brain though).

But then a fair comparison is usually not the goal of political tropes like this anyhow.

[0] https://www.nvidia.com/en-eu/data-center/h100/

[1] https://developer.nvidia.com/blog/unlock-massive-token-throu...

[2] https://services.google.com/fh/files/misc/measuring_the_envi...

This says something about why LLMs simultaneously feel like incredible productivity enhancers and at the same time don’t have as clear of an impact on productivity in the bigger picture. It probably does increase productivity, but doesn’t widen the bottlenecks or reduce the friction of tech debt. Instead it pushes more work through the existing bottlenecks, clogging them up even more, and makes code even harder to understand as no person ever necessarily had to understand it in the first place.

> The NVIDIA GB200/GB300 NVL72 rack-scale system consumes approximately 120 kW to 135 kW (TDP), with peak power potentially reaching 155 kW depending on the workload.

Widely acknowledged to be the baseline for running a frontier tier model at scale.

Pedantic corrections notwithstanding, Dave does a great job of highlighting the fact that faster code-gen without more robust review processes just leads to unfinished AI code that is too complex for a human reviewer to understand and confidently deploy.