If you have ~100k to spend, aren't there options to buy a gpu rather than just blow it all on cloud? How much is an 8xA100 machine?
4xA100 is 75k, 8 is 140k https://shop.lambdalabs.com/deep-learning/servers/hyperplane...
4xA100 is 75k, 8 is 140k https://shop.lambdalabs.com/deep-learning/servers/hyperplane...
Ignoring the operational costs of on-prem hardware is pretty common, but those costs are significant and can greatly change the calculation.
My thinking about pricing doesn't include that option because I wouldn't just hook a server like that up to a regular outlet in an office and use it for production work. If that works for you- you can happily ignore my comments. But if you go ahead and build such a thing and operate it for a year, please let us know if there were any costs- either dollar or in suffering- associated with your decision
[edit: adding in that the value of this machine also suggests it cannot live unattended in an insecure location, like an office]
signed, person who used to build closet clusters at universities
Of course that's still a very small system when talking LLM training, the only reason why i would not put that in a regular office is it's extreme price. Do you really want something worth 80k in a form factor that could be casually carried through the door?
Most people who rent cloud servers are not doing this type of workload.
In my experience, physical hardware has a management overhead over cloud resources. Backups, large disk storage for big models, etc.
Sure, if you're planning to service a large number of users, building your infrastructure in-house might be a bit overkill, as you'll need a infrastructure team to service it as well.
If you're just want to buy 4 GPUs to put in one server to run some training yourself, I don't think it's that much overkill. Especially considering you can recover much of the cost even after a year by selling much of the equipment you bought. Most of your losses will be costs for electricity and internet connection.
Buying and selling hardware isn't free; it comes with its own cost. I would not want to be in the position of selling a $100K box of computer equipment- ever.
True, but some things are harder to sell than others. A100's in today's market would be easy to sell. Harder to buy, because the supply is so low unless you're Google or another big name, but if you're trying to sell them, I'm sure you can get rid of them quickly.
One is buying capital that produces models, the other is buying a single model.
Cloud pricing is pretty steep and obviously has a fat profit margin but building your own data centers isn't cheap either. Doing this at scale is not something most companies would be very good at either. Which means it probably is quite a bit more expensive relative to what the big cloud providers are doing.