Hacker News Reader

Comment by kelseyfrog | Hacker News Reader

Diminishing returns is not only way an AI explosion might never happen; Liebig's law of the minimum[1] is just as important.

We often talk about innovation driving innovation in discussions about AI explosions. However, we must honestly ask ourselves, "Even though it looks like innovation is the limiting factor in AI now, what other factors will be limiting in the future?"

In other words is our model wrong because it could be too simplistic? If innovation was uncapped, what would be our next limiting factor, and how much of a difference would this represent?

1. https://en.wikipedia.org/wiki/Liebig's_law_of_the_minimum

View on HN

The most limiting factor, by far, is processed silicon. And that is a rising limit.

Algorithm limit? Progress is accelerating noticeably, year-to-year.

Demand? No bottleneck there. Tech demos have become accidental products, with dozens of users rising to hundreds of millions in months.

Which, for those of us who grew up in the 17th century, is pretty damn fast!! Let me tell you!

New applications? All kinds of brilliant (and pathetic!) uses springing up. Just on my Mac alone.

Commercial deep learning (neural network) software hit the road in the early 90's. People wondering when the explosion will happen apparently missed the long burning fuse phase. It's happening.

A bigger limit for the successful cases right now is the data limit. Even if silicon wasn't a limit, there's only a good few hundred years of data, and frankly less than 30 years of high volume (if low quality) data. Science as a discipline isn't that old in the grand scheme of things, and the collective data set won't end up training something that can go beyond the input.

It's quite possible "good enough" AI will trap us in a local maximum of mediocrity where the near-term ROI on thinking hard is too low.

The majority of data was created in the last 10 years, and the rate of data creation is accelerating. I don't think we'll run out.

AI will also enable the collection of more data. For example, robots could do the labwork for science experiments in far greater numbers than humans could.

Lab work has an unbelievable number of gates in front of it that mean the units of work pert unit time isn't high.

A massive amount of gibberish isn't _useful_ data.

Our use of data is WILDY inefficient currently.

Current training systems extract information on everything from every input. We don't read a cookbook to learn about space, but these models are twiddling weights all over these massive spaghetti balls of logic.

Looking at the data we have, the literal collection of all human knowledge and lamenting not having more is like standing on a ladder sideways.

But it's really starting to feel like pile of ladders is getting tall enough to bootstrap.

And these massive inefficient systems filled with data and churning matrixs will be set in order. And all the ladders laid end to end.

Surely there are limits. But it matters a LOT how far away they are. Anyone betting on humanity hitting limits to its exponential growth would have been unpleasantly surprised many times. And if you talk about how much we're changing the earth as opposed to how many people there are, we're still going strong.

Reply on news.ycombinator.com