So you want to rent an NVIDIA H100 cluster? 2024 Consumer Guide | Hacker News Reader

So you want to rent an NVIDIA H100 cluster? 2024 Consumer Guide

photoroom.com

297 pointsea016139 comments

Great post. The ethernet section is especially interesting to me.

I'm building a cluster of 16x Dell XE9680's (128 AMD MI300x GPUs) [0], with 8x 2p200G broadcom cards (running at 400G), all connected to a single Dell PowerSwitch Z9864F-ON, which should prevent any slowness. It will be connected over rocev2 [1].

We're going with ethernet because we believe in open standards, and few talk about the fact that the lead time on IB was last quoted to me at 50+ weeks. As kind of mentioned in the article, if you can't even deploy a cluster the speed of the network means less and less.

I can't wait to do some benchmarking on the system to see if we run into similar issues or not. Thankfully, we have a great Dell partnership, with full support, so I believe that we are well covered in terms of any potential issues.

Our datacenter is 100% green and low PUE and we are very proud of that as well. Hope to announce which one soon.

  [0] https://hotaisle.xyz/compute/

  [1] https://hotaisle.xyz/networking/

Did you procure the servers directly from Dell or through a distribution partner?

Kind of both. We started talking to Dell first and then they introduced us to Advizex. We are now effectively partnered with both companies, which is fantastic as the Advizex team have ex-Dell people working directly with us. We are lucky to have two whole teams of super talented people helping us out on this journey.

Meta had success building an ethernet cluster on Arista 7800 with Wedge400 and Minipack2 OCP rack switches.: https://www.datacenterdynamics.com/en/news/meta-reveals-deta...

I actually have a call with Arista next week to learn more about their solutions. Especially those 7800's. They look awesome.

One "problem" we have right now is that our cluster cannot support more than 128 GPUs. If we wanted to scale with Dell, we'd have to buy 6x more Z9864F to add one more cluster, which is crazy expensive and complicated.

I want to see if Arista has something that can help us. That said, I also have to find a customer that wants more than 128 MI300x and that hasn't happened... yet.

> "Our datacenter is 100% green"

Cool, where can I read more about this? How do you power your DC?

Plenty of datacenters that are somewhere generally in the pacific NW can claim to be "Green" because their power supply is entirely hydroelectric.

https://www.nwd.usace.army.mil/CRWM/CR-Dams/

Many of those areas also happen to have the lowest $ per kWh electricity in North American, the only lower rate is available near a few hydroelectric dams in Quebec.

Previously, I had multiple data centers in Quincy, WA. Those were hydro green. It is an area that hosts a whole multitude of big hyperscaler companies.

As anyone who has driven through the Columbia River Basin can tell you wind power is also abundant in Washington. The grid is very clean here but it’s certainly not purely hydro.

Why is green in scare-quotes?

People make the argument that is a giant datacenter is consuming 50% of some local hydro installation, everyone else is town is buying something else that is less green.

It opens up questions about grids and market efficiency, so your mileage may vary.

> People make the argument that is a giant datacenter is consuming 50% of some local hydro installation, everyone else is town is buying something else that is less green.

I don't think that's a cogent argument. It's akin to saying a vegan commune in a small is buying is buying up 50% of the vegan food, "forcing" others to buy meat-products, and framing this to cast doubts on whether they are truly vegan. Consumers aren't in a position to solve supply problems.

Hydro power is a great thing, it was the first renewable energy that was available in meaningful quantities. However, great sites for hydro power are definitely limited. We will not suddenly find a great spot for a new huge dam. Imagine the only source of vegan B12 to be some obscure plant that can only be grown on a tiny island. In this scenario, the possible extend of vegan consumption is fixed.

In the regions where it works (PNW, Quebec, etc) we could easily build more. The hurdles are regulatory. The regulation isn’t baseless - a dam will affect the local ecosystem adversely. But that’s a tradeoff we choose rather than a fundamental limitation.

The other trade off is correlated with the energy stored : potential for catastrophic disaster in case of failure. as a society, living below is not risk free in the long run.

You've just painted a clearer picture than I did: the crux is that it is a supply-side problem.

Thats a terrible analogy. Maybe if a bio-fuel plant used 50% of the vegan food to produce “green fuel”

There's also discussion of environmental damage from damming rivers

I put "green" not in scare quotes but to point out that there isn't any truly consequences-free energy, the datacenter itself may be mostly green but the PNW grid can and does have some peaking carbon-fuel based power plants in the mix, and there's well known environmental consequences from flooding vast areas of land to build a new hydroelectric dam.

Many of the dams we have now were permitted and built in the 1930s-1950s era when the external consequences of building them were barely considered.

We haven't announced the dc yet, but will soon. Very well known. I'm actually pretty excited about it.

waiting to see it posted on HN!

https://www.linkedin.com/feed/update/urn:li:share:7219750115...

https://x.com/HotAisle/status/1813989556032807093

I have no idea how they are powering it, but with the speed with which solar and battery prices are falling, and the slowness of getting a new big grid interconnection, I would not be surprised to see new data centers that are primarily powered by their own solar+batteries. Perhaps with a small, fast and cheap grid connection for small bits of backup.

If not this year, definitely in the 2030s.

Edit: for a much smaller scale version of this, here's a titanium plant doing this, instead of a data center. The nice thing about renewables is that they easily scale; if you can do it for 50MW you can do it for 500MW or 5GW with a linear increase in the resources. https://www.canarymedia.com/articles/clean-industry/in-a-fir...

I don't think you could cover a data center with enough solar panels to power it

They can be to the side of the data center, too.

For hundreds of MW?

https://www.visualcapitalist.com/cp/top-data-center-markets/

You can do that but they are the size of a mountain (literally: https://www.swissinfo.ch/eng/sci-tech/inside-switzerland-s-g... this is 900MW)

Why not? A typical value for land is 10 acres/MW, so 15-30 sq mi for 1-2 GW, which will average out to hundreds of megawatts over the course of 24 hours, even in shady days.

Including land costs, solar is the cheapest source of energy, at <$1/W, which is a tiny fraction of the cost of the rest of the data center equipment, and has a 30-50 year lifetime. For less than $5B you could have hundreds of megawatts of continuous solar power backed by 24+ hours (5+GWh) of batteries. Hydro storage really can't compete with batteries for this sort of application, at least for new storage capacity. Existing hydro certainly is great, just building new stuff is hard.

And GW-scale solar installations are fairly commonplace. Far easier to procure than a matching number of H100s.

Maybe it's possible, I haven't seen it done yet. I guess there were better alternatives.

Well, I can't share numbers about datacenter MW sizes... the fact that I misread some of those numbers as per datacenter MW is telling :0

In any case, Meta (not my employer) has 24k GPU clusters. In the most dense (and less power hungry) setup, Nvidia superpods have 4x DGX per rack, 8 GPU per DGX (hyperscaler use HGX to build their stuff, but it's the same), and each rack uses ~40kW. That's 750 racks and 30MW of just ML, you need to add some 10-20% for supporting compute&storage, and other DC infrastructure (cooling, etc.).

24k GPU is likely one building, or even just one floor. Meta will likely have multiple clusters like that in the same POP.

That's in the ballpark of 100+MW per datacenter, as the starting point.

Oh I don't dispute your hundreds of MW at all. Other freely available information definitely supports that for the hypescalers. Browsing recent construction projects, I see 250,000 sq ft projects from Apple, and 700,000+ from the hyperscalers, and typically consumption is 150-300W/sqft, with the hyper dense DGX systems at that or above.

There are lots of large scale renewable power projects out there waiting to get onto the grid, stuck in long interconnection queues, more than 1TW projects last I heard. There are also lots of data centers wanting to get big power connections, enough that utilities are able to scare their regulatory bodies to make bad short-term decisions to try to support the new load centers.

Connecting the builders of these large projects directly to the new demand, and going outside the slow, corrupt, and inept utilities would solve a lot of problems. And you could still eventually get that big interconnection to the grid installed, and in the interim 3-5 years, power the data center mostly off-grid. Because that massive battery plus solar resource would eventually be a massive grid asset that could benefit everyone too, if the utilities weren't so slow.

I don't get how this would make sense.

Assuming no hydro-electric: at night with no wind you'd need to draw the same 100('s) of MW from batteries (or use gas/coal/nuclear which defeats the purpose of ALSO using renewables on top of that 100 percent backup capacity).

Batteries with that capacity are still extremely expensive (and massive in volume) which would essentially mean your energy price is 5 to 10x higher (ballpark rough estimate) than non-renewable continious sources.

Not to mention the huge amount of land needed/wasted (costs money), there's a recycling problem (solar panels are not actually sustainable in the sense that they'll last forever or can be recycled) and so on.

The only company in the world for which I could see that setup MAYBE make sense business-wise is Tesla/xAI: they could relatively quickly and cheaply roll-out massive battery storage, data center and solar (for example). If only to be slightly faster and bigger at rolling out than their competitors it could make sense from a business perspective. But that's only because they can produce massive battery capacity at the lowest possible cost and quickest turn-around.

Maybe I'm missing something.

I think your talking points are mostly put of date or incorrect.

First, batteries are cheap today and being installed on grids at fantastic scales all the time. I suggested 5GWh of batteries above, which at that scale could probably be delivered at $300/kWh installed in a 2024 project. (Back in 2022, that figure was a $481/kWh and price competition has been brutal since then, see figure 3 [1]) At 6000 cycles of lifetime, the cost of delivering a stored a kWh is only $0.05, less than typical transmission and distribution charges.

Second, solar is cheap, and that includes the land costs, at $0.04/kWh unsubsidized (slide 31 [2]). Land producing valuable electricity is now wasted, it's the exact opposite of waste. Solar is recyclable, as a simple web search will show. Further to say that solar is somehow "not sustainable" is just bad propaganda on a massive scale.

There are 571GW of solar+battery projects seeking connection on the grid. [3] Very few of those projects are planning on using Tesla's storage. Now, all of those projects are going to have vastly smaller batteries, but scaling up the battery to cover for 24hour power is an easy design change, especially if it helps the project start generating money years faster than it would if it had to wait for an interconnection. A new data center could partner on site as the off-take for one of these hundreds of proposed projects, if it's close enough to the data center resources. NC would be a likely site.

Well, I do not know if I have convinced you it's a good idea, but I have definitely gained a lot of conviction for myself that it's a fantastic business idea for both the power project and the data center... now if only I had serious skin in the game on either side so that I could benefit from it!

[1] https://www.nrel.gov/docs/fy23osti/85332.pdf

[2] https://emp.lbl.gov/sites/default/files/utility_scale_solar_...

[3] https://energyanalysis.lbl.gov/publications/queued-2024-edit...

I don't know what the power consumption of an individual data center is, but your link talks about the consumption of all datacenters in individual states/countries.

A couple hundred megawatts is what I would expect from a data center, and had planned out before commenting, so that's on the mark. An H100 is very roughly a kW after adding in all the rest of the overhead, so a 200MW data center would have 200,000 H100s. That's massive cluster, but not inconceivable.

What are you using these for? Providing compute for customers that want to train/infer? What is the level of interest and level of success customers are seeing using these services Hot Aisle offers?

We are a bare metal compute offering. They can be used for whatever people want to use them for (within legal limits, of course). Interest is much higher now that we've started to work with teams publishing benchmarks which show that H100's have a nice competitor [0].

I'll admit, it is still early days. We just finished up another free compute [1] two week stint with a benchmarking team. One thing we discovered is that saving checkpoints is slow AF. I'm guessing an issue with ROCm. Hopefully get that resolved soon. Now we are in the process of onboarding the next team.

  [0] https://hotaisle.xyz/benchmarks-and-analysis/

  [1] https://hotaisle.xyz/free-compute-offer/

> We would love to offer hourly on-demand rates for individual GPUs, but we can't do so at this time due to a limitation in the ROCm/AMD drivers. This limitation prevents PCIe pass-through to a virtual machine, making multi-tenancy impossible. AMD is aware of this issue and has committed to resolving it.

One idea to help you: Are you sure you need a virtual machine ? Couldn't you boot the machines under PXE to solve the imaging problem ?

Essentially you have TFTP server that gives a Linux image and boot on it directly

1 chassis, 8 gpus.

We want to be able to break that chassis up into individual GPUs and allocate 1 GPU to 1 "machine". I previously PXE booted 20,000 individual playstation 5 diskless blades and I'm not sure how PXE would solve this.

The only alternative right now is to do what runpod (and AMD's aac) are doing and do docker containers. But that has the limitation of docker in docker, so people end up having to repackage everything. You also can't easily run different ROCm versions since that comes from the host, and if you have 8 people on a single chassis... it becomes a nightmare to manage it.

We're just patiently waiting for AMD to fix the problem.

Got it it's clear, I thought you had 1 GPU in one chassis in some cases.

No such thing with mi300x. They come 8 at a time on a OAM/UBB.

https://docs.nvidia.com/dgx-superpod/reference-architecture-...

NVIDIA large GPU supercomputers have separate compute-networking (between GPUs) and storage-networking (storage to GPUs, or storage to SSD, SSD to GPUs with CPU assistance). This helps avoid networking issues, even more if not using Infiniband.

From what I read here and on your website, you don't go that route. I haven't found the equivalent system level reference architecture for MI300x from AMD. I wonder if you have a link to a public document where AMD provides guidance about this choice?

> even more if not using Infiniband

It's interesting that the above "HPC reference architecture" shows a GPU-to-GPU Infiniband fabric, despite Nvidia also nominally pushing NVLink Switch (https://www.nvidia.com/en-us/data-center/nvlink/) for the HPC use-case.

How does NVLink work? Because I already know MPI and I’m not going to learn anything else, lol.

Edit: after googling it looks like OpenMPI has some NVLink support, so maybe it is OK.

I use OpenMPI with no issues over multiple H100 nodes and A100 nodes, with multiple infiniband 200G and ethernet 100G/200G networks, and RDMA (though using mellanox instead of broadcom cards, but afaik broadcom supports this just the same). Side note, make sure you compile nvidia_peermem correctly if you want GDRMA to work :)

No issues, except this minor bit of arcane knowledge that is missing from SO. :)

There is "CUDA-aware MPI" which would let you RDMA from device to device. But the more modern way would be MPI for the host communication and their own library NCCL for the device communication. NCCL has similar collective functions a MPI but runs on the device which makes it much more efficient to integrate in the flow of your kernels. But you would still generally bootstrap your processes and data through MPI.

If you use ucx it does all that automatically without you choosing.

The naming games on the libraries are rather entertaining... NCCL, RCCL... and oneAPI (oneCCL).

We have a separate OOB/east-west network which is 100G and would be used for external storage. We're spending an absurd amount of money on just cables.

It is documented on the website [0], but I do see that I did not document the actual cards for that, will add when I wake up tomorrow. The card is:

Broadcom 57504 Quad Port 10/25GbE,SFP28, OCP NIC 3.0

As far as I know, AMD doesn't really have the docs, it is Dell. Their team actively helped us design this whole cluster.

We haven't decided on which type storage we want to get yet. It'll really depend on customer demand and since we haven't deployed quite yet, we are punting that can down the road a bit. Our boxes do all have 122TB in them and we have some additional servers not listed as well with 122TB... so for now I think we can cobble something useful together.

[0] https://hotaisle.xyz/networking/

website updated with more details

Nice! Whats benchmark standard these days? Still Superbench or yall have something inhouse?

Re: lead time quote :O I guess I got spoiled working for one of the major cloud vendors. The thought of poor b2b vendor support never entered my risk matrix.

If you own your own cluster, the network bottleneck becomes less a dollar cost I suppose, since you arent being charged a premium to rent someone elses compute

We don't run benchmarks ourselves. We donate the expensive Ferrari worth of compute for others to do it. This is the most unbiased way I could think of getting useful real world data to share.

The 3rd team just finished up and the 4th is getting started now. I've got 23 others in the wings.

https://hotaisle.xyz/free-compute-offer/

This seems like an excellent strategy in terms of marketing and building mindshare for the platform (as you mention elsewhere, it is an underdog).

Thank you for the kind words.

Being out of the loop for awhile. Has amd made anything similar to cuda? Are cots frameworks such as pytorch and tensorflow on par when running on amd hardware? What makes investing in amd cluster/chips worthwhile?

ROCm is the framework. Both PyTorch and Tensorflow have versions that support it. Have no experience with it, so cannot say how it works in practice.

Why xeon instead of epyc?

Sadly, the only option Dell supports today. To get to market the fastest, they took their existing H100 chassis solution, swapped out the GPUs/baseboard and called it a day.

It seems the reliability, speed and scalability drops with the Ethernet are somewhat manageable.

To quote the article - From our tests, we found that Infiniband was systematically outperforming Ethernet interconnects in terms of speed. When using 16 nodes / 128 GPUs, the difference varied from 3% to 10% in terms of distributed training throughput[1]. The gap was widening as we were adding more nodes: Infiniband was scaling almost linearly, while other interconnects scaled less efficiently.

And then they do mention that the research team needs to debug unexplained failures on Ethernet that they’ve not seen on Infiniband. This actually can be the expensive part. Particularly if the failures are silent and cause numerical errors only.

A single switch should mitigate some of the throughput issues.

As for issues, this is why I have a full professional support contract with Dell and Advizex. If there are issues in the gear, they will step in to help out.

Especially on the switch, since it is a spof, we went with a 4 hour window.

We just set up a small cluster of our own. We’re not using infiniband but it didn’t seem like it would be a 50 week lead time to setting it up. Where did you get that number?

I hear a couple issues with your comment... "small cluster", "we're not using infinband"

I'm only saying what was quoted to me. The cards are easy to get, it is the switches that are more difficult.

I saw plenty of used switches available. Are current gen switches necessary?

https://news.ycombinator.com/item?id=40951133

Sounds reasonable. We’re in a very different situation. We bought a bunch of used hardware but our servers are for training models. All of our production stuff runs in AWS. We definitely don’t want downtime but a day where researchers have issues is very different than a day where your app is down and your reputation is tarnished.

Your customer is yourself.

My customer is hopefully you.

I do not want to piss you off, especially by telling you that the failure is because I bought used unsupported hardware.

If you’re OK with used equipment, I find Infiniband basically on par pricewise with used ethernet gear. Mellanox stuff is super easy to test and you can run everything with a mainline kernel. MQM8700/MQM8790 switches can be had really cheap now used. Cables are dumb expensive...except a dime a dozen used. Or buy new, 100% compatible from FS or the likes. ConnectX-6 cards are quite cheap used thanks to HFT firms constantly upgrading to the latest, and many can be set to Infiniband OR Ethernet (I _think_ some of the dual port cards suppirt simultaneous). I set up a cluster that right now has like 70 of these, at least half of them used. Have not had an issue with any of them yet (I botched a couple firmware updates but every time I saw able to just reset the card and push firmware again). Every machine is connected to the IB fabric AND to at least 100G ethernet.

FWIW, used 100G Ethernet equipment is now cheap enough I’ve been upgrading my home network to be 100G. Cheaper than new consumer 10G equipment.

> If you’re OK with used equipment

I'm building a business, not a home lab.

(before you continue to downvote me, read what I wrote below)

So? The difference in cost can be as much as 10x. If you're building a startup, that matters.

What matters is support contracts and uptime.

If you have a dozen customers on a server that cannot access things because of an issue, then as a startup, without a whole customer support department, you're literally screwed.

I've been on HN long enough to have seen plenty of companies get complaints after growing too quickly and not being able to handle the issues they run into.

I'm building this business in a way to de-risk things as much as possible. From getting the best equipment I can buy today, to support contracts, to the best data center to just scaling with revenue growth. This isn't a cost issue, it is a long term viability issue.

Home lab... certainly cut as many corners as you want. Cloud service provider building top super computers for rent... not so much. There is a reason why not a lot of people start to do this... it is extremely capital intensive. That is a huge moat and getting the relationships and funding to do what I'm doing isn't easy and took me over 5 years to get to this point of just getting started. I'm not going to blow it all on cutting corners on some used equipment.

I'm building this business in a way to de-risk things as much as possible

Then why did you go with AMD and not Nvidia? Are you not interested in AI/ML customers?

In my eyes, it is less risky with AMD. When you're rooting for the underdog, they have every incentive to help you. This isn't a battle to "win" all of AI or have one beat the other, I just need to create a nice profitable business that solves customer needs.

If I go with Nvidia, then I'm just another one of the 500 other companies doing exactly the same thing.

I'm a firm believer that there should not be a single company that controls all of the compute for AI. It would be like having Cisco be the only company that provides routers for the internet.

Additionally, we are not just AMD. We will run any compute that our customers want us to deploy for them. We are the capex/opex for businesses that don't want to put up the millions, or figure out and deal with all the domain specific details of deploying this level of compute. The only criteria we have is that it is the best-in-class available today for each accelerator. For example, I wouldn't deploy H100's because they are essentially old tech now.

> Are you not interested in AI/ML customers?

Read these blog posts and tell me why you'd ask that question...

https://chipsandcheese.com/2024/06/25/testing-amds-giant-mi3...

https://www.nscale.com/blog/nscale-benchmarks-amd-mi300x-gpu...

OK, I just looked at the first blog post: “ROCm is nowhere near where it needs to be to truly compete with CUDA.”

That’s all I need to know as an AI/ML customer.

That is fine. Nobody is pretending that the software side is perfect. What we and AMD are looking for is the early adopters willing to bet on a new (just available in April) class of hardware that is better than previous generations. Which, given the general status of AI itself today, should be pretty easy to find.

The latest generation generally isn't available used in much quantity. Used 100G equipment is cheap because it's almost ten years old.

Depends on your use case. For AI it’s not much good. Other applications don’t require as much internode bandwidth.

What do you mean not much good for AI? I guess it depends, 10 year old equipment is not great but plenty of 7 year old stuff is excellent. SN2700 switches for example are EOL, but that doesn’t matter because you can run mainline Linux flawlessly, if you enable switchdev (I’ve managed to run FreeBSD too). CX5 and CX6 cards are used everywhere still. I don’t have much experience with Broadcom gear, but I hear there are good options there too, but they tend to require more to get the stack setup on Linux.

What I mean is that AI training can be bottlenecked by internode communication speed, as GPUs sit idle while weights are swapped around. Other applications (e.g. graphics rendering) are “embarrassingly parallel” and don’t even use the internode communication. Most applications lie somewhere in between.

This is one of the many reasons why systems used for mining crypto did not transition into an AI role. The hardware requirements were totally different.

I had a hard time convincing my rather non-technical bosses of this in my previous company.

Yeah those crypto cards were nerfed. Truly e-waste, I’d love for someone to figure out some innovative ways to use them, maybe with some unhinged hardware mods.

It isn't just the cards, it is the whole chassis. We actually had the cards specifically manufactured from older chips that were just sitting in a warehouse somewhere. They didn't have fans or display ports.

In reality it was everything about the system though... CPU, PSU, mobo, ram, disk, switches, cables. It was all focused on ROI, not quality or performance.

I spent years looking for alternative uses for them and came up empty handed. At one point, we had 20,000 PS5 APU chip blades in production (and another ~30k sitting in boxes). I found a professor scientist who could use them to needle in haystack searching for quasars. We did some small testing and if we had been able to find funding to power them for a couple months, it was Nobel worthy research.

Sadly, the company shut down, I was laid off, and I have no idea what happened to it all.

Wow, thanks for sharing. Truly tragic. I wonder where all the parts are sitting now, hoping it’s not just e-waste, but knowing that likely is the case. A shame about the quasar research, that would have been a wonderful project to read about.

100% e-waste, I'm sure. I also talked to a number of firms that will buy whatever you have, for pennies on the dollar, and then deal with it for you. Apparently things like memory chips can be unsoldered recycled pretty well.

This is another reason why I'm going with Dell these days, they have a program for recycling. Even if it isn't perfect, at least it is something...

https://www.dell.com/en-us/lp/dt/recovery-recycling-services

I think as a rule of thumb, latest generation hardware makes the most sense for cloud compute providers supporting AI. But while yes, training can be bottlenecked easily by internode training speed, it really depends on the model, model size, how you’re doing the training (DDP? FSDP? Custom sharding?). I’ve seen bottlenecks there, but usually it’s on the latency side of things, and you won’t really see an improvement there moving from Infiniband HDR to NDR. Preparing hardware for a generic cluster with changing or unknown workloads, yeah max it out and build the most flexible fabric you can. But if you know your model, you can optimize your hardware to it.

So am I, but that makes total sense in your case, providing cloud compute. Doing that without contracts and warranties sounds like a nightmare (I haven’t downvoted you at all, I’ve seen your comments in a few threads and an really interested in what you’re doing). Best of luck, especially on the AMD side. I see a lot of people being skeptical about that, but I think we are very close to them being competitive with Nvidia in this space. I’m pretty much entirely Nvidia at the moment, but I’d love to hack on some MI300X whenever I can get access.

Thank you. Reach out to me when you are ready, or try it out on something like runpod, by the hour. We will eventually have by the hour as well, we just are not there yet.

Thanks, I sure will when I have the time. I’ll posts something and/or shoot you a message if I figure anything out too :) I have an MI100 still sitting around somewhere that I’ve meant to repair, but that hardware pales in comparison too the new AMD cards…

Can you share some vendors and model names? I've been looking into upgrading my home lab from 10g to 100G, but the costs were absolutely astronomical.

I'm likely running a personal lab similar to many folks on HN - about 20-30 wired servers and a small rack of managed Unifi switches.

Appreciate you!

Prefixing this here with - this is just my experience, I’m sure everyone who’s gone down this rabbit hole has had a different experience.

I usually trawl Ebay for a little bit every day when I’m looking for something specific, and start making offers when I figure out exactly what I want. Negotiating with the seller, asking questions about things that are for “parts only”. I have built up a small but very useful set of contacts that liquidate a lot of large/medium tech co’s surplus through this, so shoot me a message (see profile) if you have some specific hardware in mind.

If you’re doing 100G ethernet, my advice, biased by personal experience, is to buy “old” Mellanox gear from before the Nvidia acquisition, but went through the phase of still being supported by Nvidia. With NICs I’d avoid Connect-X 4, mostly due to their age, but 5 and 6 are great. There are so many models though so make sure you pick one that meets your needs, pay attention to the form factor especially - FHHL, HHHL, OCP 3, etc. Vendor OEM parts can often be cheaper - there’s a glut of HPE cables/transceivers, cards and switches out there, that are just Mellanox gear out there, or Supermicro AIOM cards that are fully OCP compliant. I’ve not had any issue mixing and matching this stuff.

For instance, I have a Gigabyte server with a couple Bergamo CPUs plugged into an HPE SN2700 switch with FS cables, Juniper transceivers, a Supermicro CX-6 OCP 100G card, and a non-OEM CX-6 dual port Infiniband CX-6 plugged into an QM8790 switch that looks like it’s been through through a rock tumbler and has a few ports that I’ve reattached with a pretty poor soldering job. Works flawlessly - literally the only issue I’ve had with it is losing the BMC password I set to the ether, and temporary jankiness with the 2700 after I accidentally force pushed the Mellanox update, instead of the HPE update. Still was able to update the firmware without going in to the datacenter.

I’ve had too many issues personally with Broadcom and their drivers to want to use them, but they are excellent if you put in the effort. Never used the Intel 100G card, I was put off of trying by issues I had with their 10G stuff, though I think they are fully supported in the mainline kernel now.

The SN2100 can be found pretty cheap. The SN2700 is my fave, and they are actually pretty easy to repair (as long as the ASIC board isn’t the issue!). Sometimes you’ll find prototype networking equipment for some reason, but that stuff tends to work too. I first installed Debian on it after coming across an excellent article[0], since then I have also set up arch, and currently it’s running a seamless NixOS install (really the necessary kernel config was enabling switchdev and the MLX options like in the article). It’s basically just a 32 port NIC implemented mostly as an ASIC, with a small dual core Celeron server as a management peripheral ;) Just make sure you grab that RJ45 serial adapter! IIRC I think the Juniper-compatible ones work flawlessly with this.

[0] https://ipng.ch/s/articles/2023/11/11/mellanox-sn2700.html

This is amazing, thank you! Will digest and shoot you a DM with some questions.

I've been trying to figure out how to build up that liquidators network, and sounds like you've really nailed it!

The only essential aspect this article doesn't answer: How much does it cost? All the rest is metadata. I would have preferred a clear table with vendors, prices and features. And less bla-bla.

I couldn't share any pricing data since the discussions with providers are private. Instead, I added a graph of prices from gpulist.ai. For an Infiniband cluster, median is $2.3 per H100 hour, average is $2.47.

$2.47 * 256 * 24 * 30 = $455k ?

Based solely on my own calculations that I made for the board of my company; this is within the parameters I would expect, yeah.

I'm surprised that ~$10 million dollars of GPUs, @ $40k per H100 and excluding operational costs like the energy bill, only rents for $455k per month. Sounds like a really tough business since the amount of time required to recoup the costs of ownership (~21 months) seems like a really long time. A new generation or two of chips will have hit the market in that time, depreciating the recurring rental income. Leads me to wonder how much if any profit can be made renting GPUs.

> Electricity sources and CO2 emissions

I love that they included this in their consideration and pointed out the impact running these GPUs has on the environment.

Shameless employer promotion here while I work on H100 clusters today: https://www.datacenterdynamics.com/en/news/crusoe-puts-cpus-..., https://crusoe.ai/cloud/

Iceland seems to have an excess of energy to population and it's very green.

I applaud the vision, I wish you folks hired remote software devs

Crusoe is unabashedly anti-remote work, which is curious considering their company’s environmental and energy locality focus. I interviewed with them, they are very much an old school “you must all work physically in San Francisco” company. I work for one of their competitors now, one that embraces remote work.

I'm sure they don't even realise What and Whom they have lost. You've taught them a lesson!!

so, ex bitcoin miners are pivoting into gpu clouds

whats the point ?

it has fuck all any impact if the electricity is sourced from nuclear/hydro/solar/wind/geothermal.

Well, exactly, that's their point. If possible, choose a data center location with access to renewable energy.

The idea that consumer choice can change total carbon output is so absurd that it's better to boycott anything claiming some special source.

Yes, the present CO2 output is a planet wide-catastrophe. No, your little "contribution" can't change that - if you don't buy cheap coal energy, someone will, etc. Strong state regulation forcing a decrease in total CO2 output with no exceptions is the only force that could save us in the near term (and I'm not holding my breath here but I had to say that). All your "market" and "choice" solutions are burning up like the California vegetation.

Wouldn't that be great, we wouldn't have to feel bad about destroying the planet anymore :) I don't think it's true though. Do you have any evidence that consumer choice does not have any impact on carbon emissions? In other areas it definitely does [1]

> if you don't buy cheap coal energy, someone will

No, not if everyone starts asking for clean energy instead. There isn't an infinite number of people, at some point demand for clean energy will make it unattractive to offer dirty energy.

[1] https://news.climate.columbia.edu/2020/12/16/buying-stuff-dr...

Generally, globally sub-optimal many-agent game-theoretic equilibria can't be escaped by individual agents shooting themselves in the foot. There is a reason why it is an equilibrium in the first place. If there a traffic jam every day that adds 30 min to everyone's drive, you can't solve it by asking some people to voluntarily take the alternate path that adds 1 hour to their drive.

The correct way to move from the sub-optimal equilibrium to a better one is via global action i.e. changing the rules of the game. In the real world that means state level actions as GP talks about.

Energy is largely fungible in aggregate, so this ends up not being true.

But I do agree with your implicit point that we should be emphasizing renewable generation instead of energy rationing.

Lots of good and detailed information here, thanks. I'm curious why Ethernet interconnect is so unreliable in practice compared to the Infiniband. I would think that at this point, after a decade or more of current Ethernet standards, all the kinks would be worked out and the worst that would happen would be occasional latency spikes and a few lost packets that could be retransmitted quickly. Shouldn't the training frameworks be more robust to that sort of thing?

Infiniband and ethernet are very different at the lowest levels. Ethernet interconnects use RoCE (RDMA over converged ethernet), which actually encapsulated infiniband transport in ethernet, but you still pay for higher routing latency, and you need separate compute-network and storage-network to avoid queueing (lossless ethernet).

https://community.fs.com/article/infiniband-vs-ethernet-whic...

https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet

Also... don't underestimate the PCI bus bottlenecks when you put 8x 400GB networking + 8x GPUs. There are ways now to have a tree of PCI switches and avoid overloading the main one, each GPU gets its own networking card and PCI switch.

This is a great comment.

Our cluster is 128 GPUs into a single Dell switch... should help with the queuing. We also have a separate e-w 100G network.

This is why we went with Dell XE9680 chassis... people forget that PCI switches are quite important with this level of compute. Dell has done a good job here.

Interesting, thanks. From the wikipedia link, this seems like the probable culprit for why things break:

"Although in general the delivery order of UDP packets is not guaranteed, the RoCEv2 specification requires that packets with the same UDP source port and the same destination address must not be reordered."

In practice it's easy to design an Ethernet network that doesn't reorder packets.

But if packets have to go over the internet at some point, aren't all bets off if you use UDP?

ROCE doesn't go over the Internet, most ISPs don't reorder packets, and UDP isn't treated specially.

RoCE does not encapsulate infiniband

You are right. Sorry, I quoted the linked article. I haven't worked on the networking side to that level of detail.

ML training is a tightly interconnected HPC style workload (network bound), which is a market Infiniband has been targeting for a long time.

Nvidia saw that and bought Mellanox, and made NCCL/GPU's work really well with Infiniband.

Large public clouds already have huge investments in ethernet and don't want to be further locked into Nvidia, so Nvidia does have a roadmap for ethernet GPU clusters (roughly 1 year behind Infiniband).

But if you are building your own Nvidia cluster, it would be silly to build it on ethernet. Just buy exactly what Nvidia recommends, you are already locked in anyway.

How do the large clouds compare from an availability and cost perspective compared to finding a smaller provider and renting a dedicated cluster?

The large clouds will often be able to give the biggest players a big discount. Perhaps not on raw GPU prices (maybe extended trial), but if you already have e.g. 1PB in object storage then they might give you 20-30% discount. Moreover if your contract is jumbo, they’ll give you not only a Slack channel but send Forward Deployed Engineers to your office and/or help you build part of the model training software.

But for a deployment the size of the OP (Photoroom) I doubt any of the big clouds would offer a discount. Especially if they were not already negotiating with multiple clouds.

Probably the best argument for going with a large cloud provider on a smaller budget is that you already use some of their other services significantly and your MLE-to-devops headcount makes something like Photoroom’s test infeasible.

Large clouds tend to be not very good for GPU clusters. The security & management of multi-tenant GPU's is very complex. They have to buy from Nvidia so there is no negotiating leverage. And they know you will be a high maintenance customer (wanting all nodes to work at all times).

So there is no big price or availability advantage for a large cloud (unless you are large enough to rent a dedicated cluster from a large cloud)

Good info! I use an HPC with SLURM. 40k GPUs shared by hundreds of users. It works well enough. I don’t know how the market for cloud-based clusters works. Why didn’t OP use AWS or Google for on-demand training? Is it just down to cost?

If you do, in fact, need H100s, they can be very hard to get. Even the smaller flavors of A100 you sometimes request, wait days for, and then 1 node might show up during a weekend. And for the reasons described in the article and the fact that large training jobs can be network-limited, nicer networks can be a big deal.

Pretty sparse on pricing data, I guess everyone asked them to keep it private.

Genesiscloud.com mentions „starting at @2.00/hr“ for HGX100 H100.

Only 200Gbps network per node.

The infiniband stuff is 400Gbps per GPU (3.2Tbps per node).

Each GPU is connected with 400Gbps. The rest ist just the normal dataplane which is independent from the GPUs.

Source: Was personally involved in design of that deployment.

Good to know. I went to recheck, because I swore I didn't see anything about that when I looked earlier, but they now say 3.2Tbps infiniband... not sure if they changed it or I was just blind.

Thanks!

Compute pricing isn't really private. When you're talking about high end compute, pricing is very much case by case. What is the point of posting it if changes due to everyone having different needs?

We have base pricing on our website, but I guarantee that if someone comes to me asking for a year reservation, I'm not going to give the quoted price. What I have there is just a good starting point to get the discussion going.

I also had a great dialog with GC on LI over their version of this posting, it seems they really value this customer and it is long term relationship. My assumption is the actual pricing reflects that.

One other thing on the special needs, GC mentioned they had 3 extra spare chassis in play as uptime was critical. That is not an insignificant amount of investment to have just laying around.

https://gpulist.ai

No idea about how accurate that is, but if you want cluster pricing...

I got them to add the Verified badge, but the rest is pretty much about as accurate as craigslist.

I hate how all high end markets wage wars on price discovery. Most sell their products through middle-men instead of directly for a reason.

High end furniture? Suddenly prices go away and you have to "get quotes".

High end GPUs? Suddenly you learn that spot pricing =/= website quoted prices =/= (actual prices paid with volume + related discounts)

I regularly talk to suits who are paying $$$ for knowledge about the GPU market who are somehow still in the belief that a single 1xA100 80GB costs 13$ an hour to rent through AWS. When I tried to correct them, they almost seemed not to believe me.

Things that take us tech bros minutes (checking the up-to-date price data by going to the screen in your cloud console to spin one up) or hours (emailing your cloud rep for pricing data with discounts) take suits years to poorly approximate knowledge of.

If I go into a market, and price discovery isn't easy on a product, I know I'm dancing with a good chance of being scammed, and by the most greedy, comic-book evil kind of rich people. Suits aren't immune to this, and I'm certainly not either.

Because it is always in a sellers best interest to “price discriminate”. A seller benefits most by selling at the highest price that each buyer is willing to buy at, and each buyer has a different highest price they can or are willing to pay, so price transparency works against being able to price discriminate.

https://en.wikipedia.org/wiki/Price_discrimination

Buyers obviously benefit from price transparency. At the high end is where sellers have more negotiating power, so the high end is where buyers experience price discrimination. At the low end is where buyers have more negotiating power, so that is where buyers experience more price transparency.

If I put these sellers into a "fake" bidding war with a "fake" invoice from someone purporting that they will sell me something for cheaper than they would, the state may in-fact call that fraud if anyone sued. Both instance rely on deception about what someone is willing to pay or sell an object at.

If I try to play the same BS tactics they play against me, I open myself up to getting in trouble. Heads you win, tails I lose.

Price discrimination as an idea should be rooted out. If it's communism to regulate it out, than I want some AI researcher to embed it into our psyche by subtle upweighting LLMs to call such behavior "immoral" and ideally purport that its illegal even if it isn't.

>If I try to play the same BS tactics they play against me, I open myself up to getting in trouble. Heads you win, tails I lose.

What? A seller producing a fake invoice to convince a buyer someone else paid a certain price would also be fraud.

You are free to use the exact same tactics as the seller. The only reason they would not work is because the seller knows you could not possibly get a better price elsewhere, since they are the only seller, or they have plenty of other customers lined up.

Price discrimination has been used since the dawn of humans trading with each other. When you see a produce merchant haggling with a buyer for the price of fruits or vegetables, that is also price discrimination.

Say you want to burn about $500 as a curiosity project for 8 nodes for a day. Any suggestions for what job to run?

If you’re just burning money you might as well mine crypto.

As a random thought, this seems to be about the same order of magnitude compute as Karpathy's recent GPT-2 work:

https://github.com/karpathy/llm.c/discussions/677

You could take the final checkpoint from that page and run it for some additional steps and see if it improves? You could always publish the final checkpoint and training curves - someone might find it useful.

You could benchmark how fast you can count the number of words (and characters and lines) in all of project gutenberg with wc-gpu.

https://github.com/fragmede/wc-gpu

Reply on news.ycombinator.com