Hacker News Reader

Comment by Game_Ender | Hacker News Reader

To me the most interesting part of the article is the cometary on where basic research is going to happen in the future. The fear is that if it only happens in large companies, then the unbiased pool of experts society relies on will be smaller and less informed. Along with the issue of nobody being around for the slog of defining a field, setting up databases, competitions and standards. These are what allow well funded corporate labs to apply their skills and compute and blow a problem out of the water. The problem is, would they do the work to define an unknown problem in the first place?

View on HN

> DeepMind claimed that they used “128 TPUv3 cores or roughly equivalent to ~100-200 GPUs”. Although this amount of compute seems beyond the wildest dreams of most academic researchers...

So, we're talking like what? Maybe $100K to $300K of hardware? Wet biology labs often have multiple pieces of $100K+ equipment at their disposal. Why shouldn't computational labs too?

Yeah but it's also putting it together and properly utlizing it, which takes specialist knowledge.

This was never really a bottleneck for science. One needs to realize first that something can be done, then it will be done.

The cost of computing will also go down in the future and for government funds this sounds like a drop in the sea when they are building multi billion dollar particle accelerators.

Imagine building a multi billion dollar computing cluster solely for research.

I guess the main reason it hasn't been done is that the deprecation is still huge due to chip advancements.

There is something like this: https://bigscience.huggingface.co/en/#!index.md

You can rent a 32 TPUv3 pod from Google Cloud at $32 per hour. So 128 pod would be roughly 150 per hour. $1K gives you 8 hours of training time.

https://cloud.google.com/tpu/pricing#pod-pricing

Is that not just for the final training run though?

All the experimentation and fine tuning probably meant thousands of trials which may have been significantly bigger scale before they got the model optimised ...

> unbiased pool of experts

Sounds like an oxymoron these days.

Always has been

Reply on news.ycombinator.com