Hacker News Reader

Comment by PJones2000 | Hacker News Reader

PJones2000

We got AWS credits too, but when we wanted to do some serious ML there were never any machines available. Our contact person at AWS just stopped replying to emails when we raised this. While one shouldn't look a gift horse in the mouth, it was just a waste of time and we had more success elsewhere.

View on HN

Our experience exactly. AWS credits are advertised everywhere, but seem difficult to utilise.

https://scientistseessquirrel.wordpress.com/2019/04/16/for-t...

We’ve not had issues getting T4 GPU instances from AWS. We faced difficulties provisioning A100 and AWS is annoying that the 2 tiers are either T4 or directly A100’s I think.

We use AWS GPU machines for our CI but for serious ML training workloads we use GCP L4 GPU instances. Even in GCP we couldn’t provision or A or H100 (our quota itself is just 1 GPU of these instances) but we’ve never had issues provisioning L4 GPU’s and I think that’s enough for smaller not LLM Scale Models. For LLM scale startups, it’s tough provisioning GPU’s even if you have money.

(We’re based in Bay Area btw)

Did you try this? We've had good luck using capacity blocks, they always seem to have some A100s and H100s available in the next few days. You can just get them yourself, don't have to wait for someone from AWS to help you.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-capa...

This seems promising, I'll be trying this out. Thank you for the pointer.

What kind of machine weren't you able to find? Were they specific H100 or A100 setups? Were your credits bound to one particular region?

No, credits weren't tied to a region but even switching to a few other regions we couldn't get hold of machines. We asked if there was a way to search across all regions so we could switch to where the machines were located. No reply. No A100s or H100s at all. We were just looking for more than 24GB VRAM which is not a big ask. I don't understand the AWS credits; however I noticed they asked a lot of questions about our business model, technology and customers. Draw from that what you will.

Credits or no credits, aren't you looking for those machines just like any other AWS customer? Credits are just to reimburse the expenses after you booked something, no? Means that if I go right now and pay in cash i would expect the same luck.

Possibly. We've tried at monthly intervals and every time, no machine available. The message we're getting from this is don't built on AWS.

I'm forced to operate in AWS GovCloud for my work and it's the same thing, but even worse. Old instance types and there's barely any of them in there. Mind you, there's two GovCloud regions and the East one is even worse! There's basically nothing outside general compute instance types, so you're really stuck with just the West region. P4's are officially available in only 1 AZ, come in exactly 1 size, and are one of the most expensive instances in the region. P3's (initially released in 2018!) are so hard to come by it's infuriating. Meanwhile we have a horde of AWS reps and they all claim there's availability.

Really feels like if you need accelerated compute GCP is the better option these days. At least there you can rewrite in Jax if it comes down to it and opt for TPU's.

Same. But in time, I realise, you don't see 90% of the things that block companies. Startups lose their main marketing channel, their plan for scaling is thwarted by quotas or random API limits that were unexpected, core team members leave.

The trick is your strategy and what's in place to survive and turn into those waves when they come in the end. Seems like you did this as you had more success elsewhere.

If your business is dependent on big Tech credits and quotas, you’re going to had a bad time. Maybe even lose your business entirely due to an ELUA change.

The problem is, what other option is there sometimes?

Quotas in this case refers to needing to request access to GPUs. Those reviews takes minutes or days depending on your relationship, and are often final.

Its possible that none of the cloud providers offering GPUs will ever allow you quotas high enough to scale to profitable margins, and there's nothing you can do about it except try and host in house (which would be mad). But the GPUs being on cloud makes that very uneconomical and they are in high demand.

This was all as true in 2019 or 2022 (more so with TPUs in 2020 I should say) as it is now. It's not a ChatGPT thing.

2013 is approximately when university and national computing units became relatively useless compared to GPU cloud compute. This stopped a wave of would-be university spin offs from having access to sufficient compute to compete.

>elsewhere

Where?

Reply on news.ycombinator.com