Hacker News Reader

Comment by sva_ | Hacker News Reader

A single 3090 will deliver more tflops than the m2 ultra.

The 3090s also give you access to all those CUDA inference and training frameworks used in industry. With Apple you're pretty much stuck with either MLX or llama.cpp.

Honestly though I'd be curious to see a cost analysis of Apple vs. Nvidia for commercial batched inference. The Nvidia system can obviously spit out more tokens/s but for the same price you could have multiple Mac Studios running the same model (and users would be dispatched to one of them).

The M2 Ultra doesn't require doing electrical work on your house like this 8x 3090 setup did though.

Yes, for 20x the FLOPS at about twice the price you might need to do some work.

Yeah cause it's doing like 50x less work.

You could power limit the 3090s to fit a standard 120V*20A = 2400W outlet if you really want to. The default power limit is 350W each so you'll only lose a little perf.

Also most rooms have multiple circuits. Just connect half the GPUs to each outlet. I already do this with my desktop PC because it has 2 PSUs.

Also most homes in the US have 30A*240V = 7200W dryer/stove outlets in the kitchen, laundry room, garage, etc. That's where you want to keep loud computers.

Brb replacing my stove with a bunch of GPUs

Inference Oven

Easy-bake cryptominer

You can cook on the GPUs from now on.

A standard outlet is 15A, and you're only allowed to use 80% continuously. So now you're dropping to 150W each.

Using two circuits is viable.

In the bay area every home I've lived in had 20A circuits for everything.

The NEC 80% rule is only relevant in data centers where someone enforces it. The time-current curve for any 20A breaker shows that it will take infinite time to trip when you draw 20A. This is correct behavior because 12awg wire can handle 20A indefinitely

Do they actually install the 20A outlets with the sideways part? I don't know if I've ever seen one in person.

I guess the outlet doesn't matter a whole lot, especially if you take multiple cables to the wall.

But I still don't want to be doing unenforced electrical violations.

Some outlets have the 20A sideways part, but from a safety/code perspective that's irrelevant.

Consider what happens when you plug two 10A devices into a power strip. In the US there are no true 15A rated outlets. All outlets must handle 20A because the electrical code allows any outlet to be on a 20A breaker.

> Consider what happens when you plug two 10A devices into a power strip.

I don't know about any argument that depends on such an action being safe. There's plenty of power strips I wouldn't trust with 20 amps.

That's just how the US electrical code works. I assure you that every "15A" outlet on the wall can handle 20A. That's why products like this don't have a fuse: https://www.homedepot.com/p/Woods-2-ft-12-3-SJTW-Multi-Outle...

Sure, neither does just using ChatGPT, but we don't do things because they are easy, we do them because they are difficult.

You can run 8 or even more 3090s without doing electrical work on a standard US residential power feed, you just need outlets connected to multiple circuits. This is not always convenient, which is why some may choose to do electrical work, but it’s absolutely not required.

Location specific issue. Higher voltage resolves this and a regular household socket will power this just fine. Then again, if you're spending that much money to get the job done, new wiring is probably one of the cheaper parts of the build.

1x 3090 would be faster than the mac, cheaper than the mac, and run on a normal residential circuit. How many breakers would you need for a cluster of 20 macs running at full throttle to match the perf of OP's build?

A single 3090 won’t even fit the model. OP is talking about running a 405B model, it needs close to 200gb of memory just to open it which is why we’re talking about mac studio and it’s 192gb unified memory.

This is exactly like when the AMD fanboys got a burr up their ass about the “$50k Mac Pro” with 2tb of memory… when you could the same thing with a threadripper with 256gb of memory for $5k, and it’s just as fast in Cinebench!

https://old.reddit.com/r/Amd/comments/f1a0qp/15000_mac_pro_v...

your gaming scores on your 3090 with 24gb are just as irrelevant to this 200gb workload as the threadripper is to those Mac Pro workloads lol

Yep. The 8x3090 setup should be at least 10 times faster than the Mac.

Yes though for inference memory bandwidth is more important than tflops. Not sure how Apple compares in that regard. The OP does train models too though which is more compute heavy.

Each of the 3090s has something like 15% higher memory bandwidth than the whole M2 Ultra system does.

Reply on news.ycombinator.com