Now, I have absolutely no clue how long this situation is going to last! But the economics don't really work out for local models while it does.
So if you can spend $30K and immediately start mining $1500/month out of thin air, that's a pretty nice investment even if the electricity costs $200/month. Two years later the cards will have paid for themselves entirely and (I suspect) will still be pretty useful.
It does argue in favor of just paying OpenAI or Anthropic for tokens, though.
At the same time, when I need to use the hardware for something, whoever is renting it from me at the moment is going to get unceremoniously booted, and I imagine they are not going to be happy about that. I assume that vast.ai's providers get uptime ratings that drive their work allocation, right?
What I do is… rent on the same platform when I actually need to use a card. A benefit is that if I need a burst of more power, that’s not an issue since it’s available from other hosts. But obviously there’s an inflection point of first party utilization where it makes more sense to own.
And there's a lot of labour and effort involved in setting these things up and maintaining them. People don't even run their own email servers, even though the hardware side of that is trivial.
Not $400 a month, it doesn't. At least not around here (we average $0.12/kWh and I don't personally run my cards over 300W.)
And there's a lot of labour and effort involved...
Theoretically, the people who hang around HN are more likely than most to be capable of the labour and effort of setting these things up and maintaining them.
People don't even run their own email servers...
People don't run their own email servers because a convenient coalition of spammers, standards bodies, and large email providers have done their best to make running one's own email server almost impossible.
I did the math and decided it’d better to pay for tokens than to buy the hardware and generate them myself.
'Capable' doesn't mean your labour has no opportunity costs.
I think my argument is easier to attack by noting that you can use AI to substitute for much of that labour.
It's worth it, knowing that there are no rugs Sam or Dario or anyone else can pull.
At the moment, it's still very easy to switch from one open weights model to another, and even between the closed models. So the 'no rug pull' property is nice to have, but not as big of a deal.
Theoretically, the people who hang around HN have immense opportunity cost when doing this. Being capable does not mean it takes no time. Time they could be using for something more profitable (and fun?).
The AI models we currently have still maintain their working state with a finite and laughably-small context window, but that is already starting to change. My .claude directory contains over 300 .md files that I didn't put there myself. Their contents are eye-opening. The question of who owns, stores, maintains, and can access that data is going to become insanely important over the next couple of years.
If you thought LLMs themselves were disruptive and contentious, just wait until the fight over object permanence gets under way. That's when owning your own box full of graphics cards is going to become important. My own bet is that I won't care too much about the electric bill or my opportunity cost when we all find out what the AI labs really have in mind, and what they're going to have to do in order to justify the valuations they're seeking.
TL,DR: it's not about the tokens, IMHO.