Hacker News Reader

Comment by adastra22 | Hacker News Reader

So? The difference in cost can be as much as 10x. If you're building a startup, that matters.

What matters is support contracts and uptime.

If you have a dozen customers on a server that cannot access things because of an issue, then as a startup, without a whole customer support department, you're literally screwed.

I've been on HN long enough to have seen plenty of companies get complaints after growing too quickly and not being able to handle the issues they run into.

I'm building this business in a way to de-risk things as much as possible. From getting the best equipment I can buy today, to support contracts, to the best data center to just scaling with revenue growth. This isn't a cost issue, it is a long term viability issue.

Home lab... certainly cut as many corners as you want. Cloud service provider building top super computers for rent... not so much. There is a reason why not a lot of people start to do this... it is extremely capital intensive. That is a huge moat and getting the relationships and funding to do what I'm doing isn't easy and took me over 5 years to get to this point of just getting started. I'm not going to blow it all on cutting corners on some used equipment.

I'm building this business in a way to de-risk things as much as possible

Then why did you go with AMD and not Nvidia? Are you not interested in AI/ML customers?

In my eyes, it is less risky with AMD. When you're rooting for the underdog, they have every incentive to help you. This isn't a battle to "win" all of AI or have one beat the other, I just need to create a nice profitable business that solves customer needs.

If I go with Nvidia, then I'm just another one of the 500 other companies doing exactly the same thing.

I'm a firm believer that there should not be a single company that controls all of the compute for AI. It would be like having Cisco be the only company that provides routers for the internet.

Additionally, we are not just AMD. We will run any compute that our customers want us to deploy for them. We are the capex/opex for businesses that don't want to put up the millions, or figure out and deal with all the domain specific details of deploying this level of compute. The only criteria we have is that it is the best-in-class available today for each accelerator. For example, I wouldn't deploy H100's because they are essentially old tech now.

> Are you not interested in AI/ML customers?

Read these blog posts and tell me why you'd ask that question...

https://chipsandcheese.com/2024/06/25/testing-amds-giant-mi3...

https://www.nscale.com/blog/nscale-benchmarks-amd-mi300x-gpu...

OK, I just looked at the first blog post: “ROCm is nowhere near where it needs to be to truly compete with CUDA.”

That’s all I need to know as an AI/ML customer.

That is fine. Nobody is pretending that the software side is perfect. What we and AMD are looking for is the early adopters willing to bet on a new (just available in April) class of hardware that is better than previous generations. Which, given the general status of AI itself today, should be pretty easy to find.

The latest generation generally isn't available used in much quantity. Used 100G equipment is cheap because it's almost ten years old.

Depends on your use case. For AI it’s not much good. Other applications don’t require as much internode bandwidth.

What do you mean not much good for AI? I guess it depends, 10 year old equipment is not great but plenty of 7 year old stuff is excellent. SN2700 switches for example are EOL, but that doesn’t matter because you can run mainline Linux flawlessly, if you enable switchdev (I’ve managed to run FreeBSD too). CX5 and CX6 cards are used everywhere still. I don’t have much experience with Broadcom gear, but I hear there are good options there too, but they tend to require more to get the stack setup on Linux.

What I mean is that AI training can be bottlenecked by internode communication speed, as GPUs sit idle while weights are swapped around. Other applications (e.g. graphics rendering) are “embarrassingly parallel” and don’t even use the internode communication. Most applications lie somewhere in between.

This is one of the many reasons why systems used for mining crypto did not transition into an AI role. The hardware requirements were totally different.

I had a hard time convincing my rather non-technical bosses of this in my previous company.

Yeah those crypto cards were nerfed. Truly e-waste, I’d love for someone to figure out some innovative ways to use them, maybe with some unhinged hardware mods.

It isn't just the cards, it is the whole chassis. We actually had the cards specifically manufactured from older chips that were just sitting in a warehouse somewhere. They didn't have fans or display ports.

In reality it was everything about the system though... CPU, PSU, mobo, ram, disk, switches, cables. It was all focused on ROI, not quality or performance.

I spent years looking for alternative uses for them and came up empty handed. At one point, we had 20,000 PS5 APU chip blades in production (and another ~30k sitting in boxes). I found a professor scientist who could use them to needle in haystack searching for quasars. We did some small testing and if we had been able to find funding to power them for a couple months, it was Nobel worthy research.

Sadly, the company shut down, I was laid off, and I have no idea what happened to it all.

Wow, thanks for sharing. Truly tragic. I wonder where all the parts are sitting now, hoping it’s not just e-waste, but knowing that likely is the case. A shame about the quasar research, that would have been a wonderful project to read about.

100% e-waste, I'm sure. I also talked to a number of firms that will buy whatever you have, for pennies on the dollar, and then deal with it for you. Apparently things like memory chips can be unsoldered recycled pretty well.

This is another reason why I'm going with Dell these days, they have a program for recycling. Even if it isn't perfect, at least it is something...

https://www.dell.com/en-us/lp/dt/recovery-recycling-services

I think as a rule of thumb, latest generation hardware makes the most sense for cloud compute providers supporting AI. But while yes, training can be bottlenecked easily by internode training speed, it really depends on the model, model size, how you’re doing the training (DDP? FSDP? Custom sharding?). I’ve seen bottlenecks there, but usually it’s on the latency side of things, and you won’t really see an improvement there moving from Infiniband HDR to NDR. Preparing hardware for a generic cluster with changing or unknown workloads, yeah max it out and build the most flexible fabric you can. But if you know your model, you can optimize your hardware to it.

Reply on news.ycombinator.com