Hacker News Reader

Comment by jonas21 | Hacker News Reader

> DeepMind claimed that they used “128 TPUv3 cores or roughly equivalent to ~100-200 GPUs”. Although this amount of compute seems beyond the wildest dreams of most academic researchers...

So, we're talking like what? Maybe $100K to $300K of hardware? Wet biology labs often have multiple pieces of $100K+ equipment at their disposal. Why shouldn't computational labs too?

View on HN

You can rent a 32 TPUv3 pod from Google Cloud at $32 per hour. So 128 pod would be roughly 150 per hour. $1K gives you 8 hours of training time.

https://cloud.google.com/tpu/pricing#pod-pricing

Yeah but it's also putting it together and properly utlizing it, which takes specialist knowledge.

This was never really a bottleneck for science. One needs to realize first that something can be done, then it will be done.

The cost of computing will also go down in the future and for government funds this sounds like a drop in the sea when they are building multi billion dollar particle accelerators.

Imagine building a multi billion dollar computing cluster solely for research.

I guess the main reason it hasn't been done is that the deprecation is still huge due to chip advancements.

There is something like this: https://bigscience.huggingface.co/en/#!index.md

Is that not just for the final training run though?

All the experimentation and fine tuning probably meant thousands of trials which may have been significantly bigger scale before they got the model optimised ...

Reply on news.ycombinator.com