Hacker News Reader

Comment by amelius | Hacker News Reader

> Of course building is cheaper than AWS...

Why "of course"? Last time I checked baking my own cookies is more expensive than just buying a bag at Walmart.

If you need 10k cookies every day for multiple years, you're better off building your own bakery. If you need multiple years of continuous DL work, you're better off building your own DL system.

If you need 5 cookies today (and maybe the next batch next week), go to Walmart. Same for an one-off DL job.

According to the article you can break even after only two weeks which is far from "10k cookies every day for multiple years".

Tech employees and maintenance is more expensive on the margin than cookie bakers

I think the analogy holds. The numbers in the article assume constant usage (there is a nod to the cost of static storage, but all the math is done on $3/hr gpu usage, not $0.1/mo/GB storage).

So if your usage is 4 hours per week, 50 weeks a year, you hit $600/year of gpu. If your usage is 24 hours a day, you hit $27k.

So it's not a clear-cut "buying metal is cheaper" - you have to decide how your usage amortizes. As pgeorgi says, if your demand is a pack of cookies a week, buying a bakery isn't the most cost-efficient way to satisfy that demand.

For DL time is measured in years*GPU instead of hours.

Only for niche problems run into by huge companies and research groups. Most 'normal' problems encountered by 'normal' companies don't need anywhere near that much training.

Depends on your use case. The more accurate models these days require lots of data with huge models. Machine Translation, Image Generation, Segmentation, etc. Each experiment takes a couple days/weeks to train, so we were encountering long stretches of constant usage.

It's fine for exactly what the article describes: a student doing projects at home.

For anything bigger than that, you'll quickly run into issues. Try talking to your ops team and telling them you want to set up a mid-range desktop PC with extra RAM and a high end graphics card as a DL workstation. I think you'll very quickly find some friction, especially once you want to go into production with it.

Google went to production with Pentium desktops on shelves.

IIRC their original hardware plan was based around commodity hardware and getting the most performance per dollar since they built the crawler to be distributed pretty early on.

Point being, consumer grade hardware has come a long way since then, and if you're doing something cutting edge like DL it's not outlandish to expect that rolling your own might be worthwhile and totally justified.

That is a horrible comparison.

Google's start up between 1996 and 1998 was also over twenty years ago. There were fewer than 200 million Internet users on the Internet and less than 2.5 million web sites [0] in 1998. Google was also started by two college grad students, meaning gasp it was initially just a research project. I'd also point out that while, yes, the initial production servers were cheap and used commodity hardware, this [1] is what they looked like, which is hardly the type of setup that the article is suggesting.

[0]: http://www.internetlivestats.com/total-number-of-websites/

[1]: https://en.wikipedia.org/wiki/History_of_Google#/media/File:...

And that's just unnecessary friction, which is a problem of politics, not technology.

I do recall a time as recently as 4 years ago where it was difficult to find, for example, rack-mountable servers that could hold a large number of these GPU cards, but substituting for a mid-range PC with 2 or 4 of them would not have been an issue, even then.

Cost would be higher but not necessarily outrageously so, since large server chasses often have startlingly large PSUs. It might be an extra 50% on top of that $3k, but that's still competitive, even with the pre-builds the article mentions.

Really? Do you count the time needed to make them? Butter, sugar, and flour aren't expensive.

This is usually the case for this kind of thing, yes. Cheap biscuits are a really cheap bulk product while the margins on the fresh ingredients are higher, and people making their own are doing so as a hobby luxury.

And I wouldn't think those cheap cookies are made from the same quality ingredients you'd made it from.

> Last time I checked baking my own cookies is more expensive than just buying a bag at Walmart.

Yes, but most likely they aren't of comparable quality

Quite likely, the Walmart ones are better, at least if it comes to taste and looks. Unless you're an expert baker. And then your cookies probably still aren't better, you will just claim they are, and it's polite to agree with that assessment. In fact, it's polite in our society to agree that any half-good DIY attempt has better quality than commercial equivalent, even though it doesn't stand up to scrutiny.

> the Walmart ones are better, at least if it comes to taste and looks

And awful if you want anything resembling "real food". (Not blaming anyone for buying them, I buy pre-made cookies as well.)

But yes, getting a nice looking/tasting cookie might require a few tries.

Only with taste and looks. Maybe.

> Even Walmart’s “Great Value 100% Whole Wheat Bread” contains seven ingredients that Whole Foods considers “unacceptable”: high fructose corn syrup, sodium stearoyl lactylate, ethoxylated diglycerides, DATEM, azodicarbonamide, ammonium chloride, and calcium propionate.

http://www.slate.com/articles/life/culturebox/2014/02/whole_...

If you had picked another example of pre-made food, I might agree, but cookies are really easy. Just follow the recipe on the back of the chocolate chip bag, you'll get something that is far better than anything out of a bag that's engineered to sit around in a store for months.

Indeed, off-topic, but I find it fascinating that I can buy a year of Kosher, Vegan, Gluten Free Survival Food on Amazon from ~$1469.

https://www.amazon.com/NorthWest-Fork-Gluten-Free-Emergency-...

That drops to $1149 if I don't require Vegan, non-GMO, Kosher:

https://www.amazon.com/Healthy-Survival-Food-Emergency-Prepa...

SNAP benefits at anywhere from $134 to $192 a month are both more than sufficient for this. So why are ~42 million people starving in America? Answer (IMO): Cloud providers reduce the friction of GPU adoption the same way grocery and convenience stores reduce the friction of food acquisition, both in exchange for profiting off of it.

Normal food can be even cheaper. You could probably indefinitely avoid starvation with dried rice and beans, oil, and a vitamin supplement for $2/day (in NA), without a big upfront cost. Poverty is complicated.

You've ignored the cost of the home required to cook those things in, which is really the whole point.

(not directing at you, more explaining to those who didn't already know)

That recent Boston(?)-based completely automated rice bowl startup plus drone delivery equals solved hunger

https://chiefmomofficer.org/2016/11/18/cost-of-cookies-is-it...

The answer is not clear cut!

Is it really, in America? Amazing.

This is not the normal human experience at all. Raw food are usually cheaper than processed ones, anywhere in the world.

Not necessarily. E.g. making Ramen noodles, much cheaper to go to a ramen shop than to attempt to make it yourself. Economy of scale.

Where in the world are you? It's my normal human experience from both the UK and Sweden. Of course the stuff you make from scratch will often have a higher quality (or at least be made from higher quality ingredients) and if you want to buy similar or better quality then it will cost more. But the cheapest chocolate chip cookies you can find in the food store will be cheaper than buying ingredients and baking.

So, whatever bakery paid for ingredients, wages and owner's profit is still less that what you'd just pay for just the ingredients? What are they putting into these cookies, sawdust?

The trick is that the bulk discount (or "small package premium") on the ingredients is huge. Maybe $300 for a tonne versus $1.50 for a kilo of flour?

Note that this works for electronics as well. You could buy one resistor for a cent or a thousand of them for a dollar.

What are they putting into these cookies

Cheap palm oil and other vegetable fats instead of butter, corn syrup instead of cane sugar, much lower quality chocolate than is generally available in grocery stores, various chemical stabilizers instead of eggs, artificial vanilla flavoring instead of real vanilla and so on.

Plus they obviously get everything at a massive discount since they're buying everything by the ton instead of by the kg.

You might want to ask why those cheap ingredients are cheap and you will find a bunch of farmers getting huge subsidies from the taxpayers to grow otherwise unprofitable crops.

Kind of like how if you remove all the subsidies given to fossil fuel producers, some forms of renewable energy suddenly become much more competitive.

So when you add a whole cost center for IT, the inefficiencies of the way they operate suddenly make the cloud efficient.

I also think processed products usually require less effort to store and transport than some fresh ingredients.

Wages and profit is amortized over thousands of cookies per day. You pay less than 1 cent per cookie. If you don't bake all the dough in your mixing bowl you already have spend more on ingredients than the labor at the factory.

The comments on cheap low quality ingredients and bulk pricing are true as well.

It will be cheaper if you buy in bulk the local Tescos to me in the UK sells rice in 10Kg bags which is several times cheaper than boil in the bag

And I am sure the local Asian cash and carry's sell even bigger bags even cheaper.

> Why "of course"?

Cause AWS is _really_ expensive...

Reply on news.ycombinator.com