Ask HN: Google Colab alternatives for large lang models? | Hacker News Reader

Ask HN: Google Colab alternatives for large lang models?

1 pointlovelearning5 comments

I just want to run some LLM experiments on GPUs. Since these are 10-20GB models, I need something like an A100 40GB GPU at a minimum.

But I'm finding that most of the paid Google Colab options don't work. Instead of working on the LLM experiments, I'm wasting considerable time on infrastructure.

First, I purchased pay-as-you-go credits[1]. But about 80% of the time, I get a "Selected GPU unavailable" for A100.

Next, I tried the custom GCE VM option. But the Colab deployment marketplace app[2] keeps hitting me with quota denials for A2-CPUs and A100-GPUs.

Requesting quota increases in GCP is equally frustrating. They auto-approved 12 A2-CPUs for a zone but denied one A100-GPU for that zone!

There are some 50 or so zones and apparently I have to individually request quota increases each of them but they get denied.

-----

QUESTION 1:

How are you people testing LLMs like GPT-NeoX, LLaMA 66B, etc?

------

QUESTION 2:

Is there a simple, non-time-wasting notebook alternative for Colab where I can get A100 or better GPUs easily?

Is there some arrangement of using Colab with local runtime which somehow proxies to a better GPU service provider?

Kaggle NB is also frustrating in other ways. Their GPUs don't seem to work with these models. Also difficult to connect to GDrive, difficult to transfer files in and out, etc.

-----

QUESTION 3:

If there's no alternative to Colab, anybody here has some practical tips to avoid these quota denials?

------

[1]: https://colab.research.google.com/signup/pricing

[2]: https://research.google.com/colaboratory/marketplace.html

Getting access to A100s seems to be a general problem for everyone, not limited to you or GCP.

The denial is probably because they don't have any to rent you. Many clouds facing this issue.

When submitting a GCP quota increase, be wordy and explain the details of your need for the new quota. Also, only ask for what you actually need and make sure they have them for the zone. Not all GPUs are in all zones.

Did you ask for more than one? Are you spending actual money on GCP or just using credits?

Also, just a heads up, even with quota, you can still get a no GPU available error if they are all rented at the time

Spending actual money on GCP. I don't have any GCP credits.

I asked for just 1 GPU and explained that I need to run 20GB+ models. Didn't realize they expect a doctorate application in that 3-line textbox.

All of it is so bureaucratic. I have to look up their zone-GPU mapping. I have to test availability indirectly via the quota workflow route. It reminds me of my country's shitty bureaucracy!

I tried 3 zones. In one zone, a GPU is granted in the quota but still shows quota error on the main UI. The other 2 zones were denied.

If they don't have any to rent, I don't know why they don't just say that. Instead, they expect an onerous workflow of requesting quota increases with pathetic user experience, only to deny them and notify the denials via email.

Any suggestions for Colab alternatives?

I have had no issues with GCP quotas, you are exaggerating the situation out of frustration, understandable. The turn around is typically a few minutes, and if you haven't tried this on other clouds, you don't know how good that is.

There is a single page list of GPU availability by zone: https://cloud.google.com/compute/docs/gpus/gpu-regions-zones

You do not have to write a "doctorate" but you do need to explain your need, not the what will you do.

again, A100s are in very high demand, so everyone is preferring the highest value customers first.

Did you get a A100 in one zone? The error messages for "you are out of quota" vs "(google) resource not available" look similar. We have devs getting them confused all the time

I don't think I'm exaggerating - I wasted hours today on their terrible workflow and UI. All of that can be simplified and automated by Google via their frontend instead of a workflow that expects users to visit multiple pages.

I have a quota of 1 GPU in one zone and zero allocated GPUs. I keep seeing only "exceeded quota by 1 GPU" error on their UI when trying to create a VM for that single GPU in the correct zone.

Thank you for the replies. Looks like most users are fine with all this. I'll probably explore non-GCP alternatives.

you're welcome

For GPUs (among others), there are two quotas, regional and zonal. We have N regional T4s and unlimited zonal. Maybe you have 1 zonal and 0 regional?

If you just need a GPU, there may indeed be better or cheaper providers

Reply on news.ycombinator.com