Hacker News Reader

Comment by conistonwater | Hacker News Reader

Are they using the on-demand price instead of the preemptible price? It seems like the sort of job that can run on preemptible machines, just because it's a batch job. Also, should the cost really be calculated using public market prices at all, as opposed to the running costs of the TPUs? It is not guaranteed at all that the opportunity cost to Google of using all those TPUs is equal to the price that you or I would pay Google to use them. I understand it cost a lot, but I'm not convinced by the headline figure of $36M.

View on HN

The article actually addresses that. The precise number is not the point but the ballpark is:

"In terms of actual cost to DeepMind (a subsidiary of Google’s parent company) to run the experiment, there are other factors that need to be taken into account, such as researcher salaries, or that the quoted TPU rate probably includes a healthy amount of margin. But for someone outside Google, this number is a good ballpark estimate of how much it would cost to replicate this experiment."

True, but "giving a ballpark estimate to replicate the experiment" is quite different to "How much did it cost"

KataGo and Leela Zero and all the other AIs certainly didn't cost that much (the people running them wouldn't have had that much money and resources) and are probably stronger than Alpha Go Zero. I don't think it's at all fair to say this number is a good ballpark estimate of how much it would cost to replicate this experiment. It's wrong as a calculation of Google's costs, it's wrong per the title How much did AlphaGo Zero cost, and it's also wrong as an estimate of the cost of replication.

> KataGo and Leela Zero and all the other AIs certainly didn't cost that much

And you base that claim on what exactly? Leela Go is trained by the community, which donates self-play resources. Just because you outsource your cost to volunteers doesn't mean it's free!

In order to get to a realistic estimation, you'd need to get the average cost for electricity, hardware cost (proportionate to use), and of course opportunity costs.

Since you cannot do that, I'd argue that you have no clue what the true training cost of these projects compared to on-demand/cloud costs really are.

Go is a small community, and there's just no chance they managed to donate $36M worth of anything, whatever the electricity and hardware costs, that's just too much money. KataGo's page says it was developed using resources donated by Jane Street, the company that its developer worked for; the general magnitude of numbers is also way off: sure they're a quantitative trading firm, but it's implausible that they'd donate $36M to develop a go AI.

The cost estimate was based on what what you and I would have to pay if we were to train AlphaGo Zero in 40 days on the given hardware using the reported number of games and resources.

Just replace the self-play TPU resources with commodity hardware or even just cheaper GPU compute providers and you'd reduce cost 10-fold by just not using TPUs. Same goes for the number of self-play games.

That still doesn't change the estimate itself. IF the other projects would've used Google TPUs, they'd well have been around the same cost as the estimate.

I really don't understand what you're trying to argue against here.

Leela Zero used some commercial providers for training. Mostly, we used the free offers for new GCP/AWS/etc members, so it's only a certain fraction of course.

To provide an order of magnitude: There are about 20m training games. Iirc a V100 could complete one game in something like a minute. So that's 300k-ish hours of value. Obviously, while V100 were the fastest, other GPUs were more cost efficient.

Replicating the experiment would normally mean running the same code for the same length of time, which is what the article measures. If you're using different approaches like KataGo, it's not a replication any more.

How is Leela Zero stronger if it's the same calculations and less compute time?

One could imagine an 'ideal' allocation of TPU's, where compute time is allocated to the project that earns the most dollars per FLOP.

Minor improvements to Google Books OCR might not be worth much, whereas better search result scoring would be worth lots. An automated system would decide where it was most efficient to spend the TPU's. Management would set how many dollars a 10% performance improvement was worth.

I'm sure the reality is a bunch of middle managers arguing over why their team deserves them more than another.

You mean better ad revenue, because search results are getting worse and worse. So search results can't really be google's primary focus anymore.

> One could imagine an 'ideal' allocation of TPU's, where compute time is allocated to the project that earns the most dollars per FLOP.

That's a short sighted, immediate benefit or bust mentality. Not to mention that projects have a ramp-up time where they are not profitable yet, but still very valuable strategically.

Like you, I suspect there are substantial bulk discounts available if you're using tens of millions of dollars worth of these.

I don't know how much less, but if you were to do fully pre-emptible at this scale I wouldn't be surprised if you could get it down to one-tenth the price. I wouldn't suspect the same of other more generic resources like CPUs that have a much lower price point to begin with, but the TPU sticker price seems very high with lots of headroom.

For me I think it's very interesting just to get a ballpark order-of-magnitude estimate. I'm sure the cost isn't much less than $3.6M and so the underlying story doesn't change that much, i.e., this is not something accessible to hobbyists.

Reply on news.ycombinator.com