Hacker News Reader

Comment by ftxbro | Hacker News Reader

I can't find how much it will cost or how much power it will use. I mean it will be a lot and maybe only Google and Microsoft and Facebook can afford it but I still want to know.

View on HN

The DGX A100 was $200k at launch. I found a DGX H100 in the mid $300k area. And those are 8 GPU systems. So you need 32 of those, and each one will definitely cost more plus networking. Super low estimate would be $500k each for $16M total. But considering its moving from 98GB to 480GB RAM per GPU. Might be more like $1.5M per 8, round it to say $50M.

And at 1/8th the power per GB, you have 700 Watts / 96GB / 8 * 480GB come to around 450 Watts per. And 115kw for the 256.

What does this mean for the AI race? For example what if a newish company (newer than Google/Facebook/Microsoft/etc.) like Anthropic, Scale, Perplexity, or Stability is able to scrape together $5B USD funding and spend their hardware budget on these things. Say that can buy $1B of them and spend the rest on hackers and operating expenses (idk if that's realistic). So maybe they could purchase and operate like 20 of them. Say that they spend six months doing experimental things and then the next six months training their Tsar Model. If they follow the Chinchilla scaling laws and normal architectures, how good will these models be?

For AI race means it opens the door to a competitor. Could be AMD, Google, or Amazon. All which have offerings in this space.

However while the hardware isn’t cheap, it’s still likely not a blocker. Costs do inhibitor more experimental research.

Startup which will rent this compute when needed will likely have more advantage on AI front.

Selling shovels is good business, but doesn't compete directly in AI area.

I have no expertise in GPU System used for AI Learning, but would It be possible to buy a bunch of consumer cards and get the same performance? Or is this not possible because consumer cards go to 40 ish GB RAM and Models would not fit or „swapping“ like crazy and be slow.

would It be possible

Not the same. Not all problems can be efficiently divided among NUMA nodes with low bandwidth interconnects.

Consumer cards only have PCIe 4.0, at most 24GB VRAM and the only recent model with NVLink, the RTX 3090, can only be connected to exactly one other card. It doesn't scale beyond that. So you are limited to PCIe 4.0 x16 speeds.

Technically the A6000 has 48 Gb of VRAM and works in 2-way NVLink.

The NVLink interconnect on all the GPUs is a huge part of it, and cannot come even remotely close to that bandwidth with consumer goods. Then the density of RAM to compute and power is huge. A single 4090 is 450 watts, for 24GB where this is 20x the memory for the same watts. 2.3Mw or so. If you say $0.14 / kwh, thats something like $325 / hour in power costs to run. Not counting additional cooling you are definitely going to need. And I am sure there is inefficiency this doesn't cover but 240v 10,000+ Amps for that?

With that much memory I could probably run Crysis 4 and an Electron app side by side!

It’s in interest of Nvidia, to try to make sure they do not end up in a situation where they have a small group of very big customer that buy a large slice of their production. For Nvidia a market of the same size, with many small to medium customer is a lot better as those customers will have a lot less power to force Nvidia to do something that isn't in it's interest or it does not want. I expect to see moves from Nvidia to help smaller players, open source or semi open models to not be crushed by the big players. Not because they are nice, but because it is in their best interest.

I wonder if you can even run this on a regular 20A circuit - I'm thinking no - 20 * 120 = 2400 Watts - I assume that will not be enough...

For a single Grace+Hopper node? I'd bet it fits in that budget, the Grace Hopper datasheet says the combo has a CPU + GPU + memory TDP of 450W - 1000W programmable, and that leaves more than half of the room for the rest of the node's power budget. For the DGX GH200? It's 18,432 CPU cores with 256 GPUs across 16 full racks of servers :p.

Here's an example of an 8x H100 machine - look at the tech specs: https://lambdalabs.com/deep-learning/servers/hyperplane

6x 3000W PSUs in a 3x3 redundant config. So 9000 watts total. So at least 240v x 50A. x2 for redundancy.

My hob in my kitchen is 7.3kw? Normal 32a * 240 circuit allows up to 7.68kw, and the 6mm^2 cable is rated to something like 45amps

This seems fairly common e.g. https://www.currys.co.uk/products/aeg-ikb64401fb-59-cm-elect...

I am sure data centers have larger circuit breakers and chunkier cables than my kitchen appliances!

Woah, I looked at the specs:

    Front left: 2.3 kW / 3.7 kW

Cripes. You can boil water extremely fast on that IH setup! I'm living with 1.5 kW, and it is painful...

When we first got our induction cooktop I was so excited about how ridiculously fast we could boil water. Which is definitely an odd thing to get excited about. It definitely isn't that powerful though. That's a lot of power.

I would guess about $100m and ~1.5-2MW

Reply on news.ycombinator.com