Hacker News Reader

Comment by girvo | Hacker News Reader

I didn't spend that much, only $6500 AUD for a GB10 based Asus GX10 which is even slower than OPs, but I spent that because it makes for a great learning platform. Theres not much else that lets me fiddle with 128GB of RAM for my graphics processor, and it's quite lovely to be able to run things as long as I like without worrying about my cloud instance being shut down.

It's not financially a good idea: renting really does beat owning, and cloud beats both if you're only running inference on these machines. But I'm not just doing inference, and as a thing I can do silly stuff on to learn, it's hard to beat!

View on HN

When you say you are not just doing inference, you mean you are also training your own llms? I am curious what other things can be done.

Fine tuning, and yeah training my own, experimenting with architectures and learning how it all works. Been a lot of fun

$6500 AUD can get you a good chunk of B200 time on any of the GPU neoclouds :)

Less than I expected, though! And I get to run this all through the night

I do still use Vast and Runpod for things too, but it’s much nicer to test a fine tuning run here to make sure I’m in the ballpark

I also did literally say “It's not financially a good idea, renting is better than owning” so I’m confused why I have two people telling me that

Also it’s just far more fun to play with something tangible to me :)

You could just rent a bare metal server with those specs

Yes I could, but that is annoying because of spot pricing and having my instance shut down, and it has fluctuating prices

It’s also annoying because then I need to make sure my little “lab” setup is well automated, and I’m lazy :)

Also, I literally said “ It's not financially a good idea” so I’m confused why you think I don’t know that.

Spot pricing and instance availability don’t apply to on metal hosting. You’d have your own machine dedicated to your own use only, at a locked in price.

> renting really does beat owning, and cloud beats both

Wait, I went and looked far and wide today and I can’t find anything with ~100GB of VRAM that isn’t $20k a year, what am I missing

That's too small. I was quoted a machine with ~1.5TB of vram, for $10k/mo. This was the minimum node size in the AI data center I was talking with -- they don't make smaller nodes that you can lease as bare metal. There is no public pricing though, and you had to know people to get in.

Reply on news.ycombinator.com