ML training compute has been doubling every 6 months since 2010

I think the Large-Scale Era does point to a new phenomenon that emerged pretty discontinuously, which is that there are now 'two lanes' in ML scaling. Prior to 2015, academic and industry would train roughly similarly compute intensive models. Since then, a small number of industry players frequently train models with 10-100x more compute than what the typical researcher uses.

Something about environmental problems causes people to focus on vary narrow types of harm rather than the major ones like habitat destruction. Domestic cats kill ~5 orders of magnitude more birds than windmills, yet somehow bird strikes of windmills is in the public continuous. I think it comes down to the inability to really grasp the difference between large numbers combined with specific harms used as distractions.

We use multiple GPUs at 100% capacity (all cores) for ML training.

When need be, we use parallelization/concurrency to use every core in computer for our application.

Almost no computer application today buffers while processing (thanks to parallelization / concurrency)

Software can improve a lot faster than hardware can (due to low capital required)

This hypothetical GPT-4 would be big enough that it shouldn't have the context window or BPE token problems of GPT-3. It would be superhumanly good at predicting token sequences-- generating text. What that would look like, exactly, is not entirely clear. (What does it mean to be twice as good as a human at writing an essay?) If it had the same architecture as previous GPTs then it wouldn't be "conscious" or be goal directed, but it would be able to correlate more information and draw conclusions that we couldn't. (Would we understand these conclusions is another open question.)

It would also be quite expensive to run in a capital-amortized sense, at first, maybe hundreds of dollars per word. The commercial use of such a thing would be limited. Large tech companies seem to be building AI because of how self-evidently useful it will be... eventually.

The economic case for high-cost NNs is the opposite of most automation, which started from the bottom up. If a net is expensive to train and run then you have to pursue the Tesla strategy-- start from the top down. So you need to target high-price knowledge work in terms of dollars per hour, but is tolerant of small variance, since the results are still probabilistic. In the near future you won't see NNs designing jet engines, producing complete software systems from requirements documents, or even standard grunt-level software engineering work of wiring two systems together, since you can never be quite sure GPT-programmer is producing the right work.

Similarly, GPT-lawyer or GPT-CEO would be tough to do. GPT-hollywood would be interesting, if you could get it to crank out a complete Marvel movie with superhumanly good CGI. More terrifyingly, GPT-advertiser, if it can produce super-appealing ads, would be able to pay for its own runtime at the cost of a machine hijacking human minds to plug money into gacha games or the equivalent in the 2030s.

Terminator, Terminator 2

The vast majority of AI research (over all time) has not been seriously intended or expected to arrive at a general intelligence.

Sure there are some true believers there in some sort of handwavy construction argument ("and then a miracle occurs") and there have been serious research attempts to understand intelligence, of limited scope an resources. But to be clear, whether this wave or previous ones, the money washes in and the army of PhD's happens when there is a sniff of practical, usable algorithmic results from machine learning. It's not the same thing.

Parallel application of even relatively stupid algorithms targeted at the right tasks will rapidly outproduce any reasonable number of organic intelligences of the type you describe :)

Or maybe your newborn is more capable than mine!

"Why try to beat nature?"

If we wouldn't try "beat nature" we would be still in caves or would be extinct. Nature is "beating" itself with evolution and we are part of that.

oscar wilde's the soul of man under socialism probably better describes the idea.

which I thought was brilliant, ha!

2. Because ai doesn't have biological scaling limits. There's nothing particularly special about human level of intelligence.

However, I've replaced the submitted title ('ML training compute has been growing by a factor of 10B since 2010') with what the tweet says now.

edit: s/parameters/FLOPs