Hacker News Reader

Comment by UncleOxidant | Hacker News Reader

I don't want to deploy an app, I just want to play around with LLMs and don't want to go out and buy an expensive PC with a highend GPU just now. Is Fly.io a good way to go? What about alternatives?

View on HN

Use https://vast.ai and rent a machine for as long as you need (minutes, hours, days). You pick the OS image, and you get a root shell to play with. An RTX 4090 currently costs $0.50 per hour. It literally took me less than 15 minutes to sign up for the first time a few weeks ago.

For comparison, the first time experience on Amazon EC2 is much worse. I had tried to get a GPU instance on EC2 but couldn't reserve it (cryptic error message). Then I realized as a first-time EC2 user my default quota simply doesn't allow any GPU instances. After contacting support and waiting 4-5 days I eventually got a response my quota was increased, but I still can't launch a GPU instance... apparently my quota is still zero. At this point I gave up and found vast.ai. I don't know if Amazon realizes how FRUSTRATING their useless default quotas are for first-time EC2 users.

Pretty much had the same experience with EC2 GPUs. No permission, had to contact support. Got permission a day later. I wanted to run on A100 ($30/hour, 8GPU minimum) but they were out of them that night. I tried again next day, same thing. So I gave up and used RunPod.io.

You might actually be better off building a gaming rig and using that. The datacenter GPUs are silly expensive, because this is how NVIDIA price discriminates. The consumer, game GPUs work really well and you can buy them for almost as cheap as you can lease datacenter ones.

I can recommend runpod.io after a few months of usage - very easy to spin up different GPU configurations for testing and the pricing is simple and transparent. Using TheBloke docker images you can get most local models up and running in a few minutes.

Paperspace is a great way to go for this. You can start by just using their notebook product (similar to Collab), and you get to pick which type of machine/GPU it runs on. Once you have the code you want to run, you can rent machines on demand:

https://www.paperspace.com/notebooks

I used paperspace for a while. Pretty cheap for mid tier gpu access (A6000 for example). There were a few things that annoyed me though. For one, I couldn’t access free GPUs with my team account. So I ended up quitting and buying a 4090 lol.

> What about alternatives?

Custom models? Apart from the Big 3 (in no particular order):

- https://together.ai/

- https://replicate.com/

- https://anyscale.com/

- https://baseten.co/

- https://modal.com/

- https://banana.dev/

- https://runpod.io/

- https://bentoml.com/

- https://brev.dev/

- https://octo.ai/