Hacker News Reader

Comment by aab0 | Hacker News Reader

Seems consistent to me. Just because you have hardware doesn't mean you know how to use it well. Consider everyone accidentally writing an O(n^2) algorithm on their multi-gigahertz/gigabyte/terabyte (CPU/RAM/HDD) machines. At the time, there was no reasonable algorithm which even on the top supercomputer could hope to beat the world champion. Then MCTS came along a few years later, and at that point maybe you could've scraped together enough hardware to beat him; then another few years, and AlphaGo... We are in the same position now for AI: the bottleneck is not the hardware but knowing what to run. (Some technical examples of this: model compression, binary networks, and residual networks, show that what we can run far outstrips what we're able to train.)

View on HN

It's not clear we have enough hardware. I've been trying to get a handle on the processing power needed for human level AI. If you take the naive approach and assume it's just a matter of getting models with the same number of parameters as the brain has synapses, our biggest models are about a million times too small. If you assume Moore's law will remain dead, then it may be a long while. Then again if modern deep learning is amenable to some sort of ASIC it could happen much quicker than one would expect, depending on how much investment is put into such things. And who knows how algorithmic advances will change things. If we ever get quantum computers, they're likely to be very useful in the design of conventional computers and materials technology in general, so another age of exponential advancement isn't out of the question.

It's very, very hard to be certain about these things.

As I've come to understand it when implementing deep learning machine learning the hardware requirement for running a trained neural network model efficiently is much lower than the one required to train it. I've also read somewhere that if an instance of AlphaGo run on a modern powerful desktop machine would compete with the cluster used in Lee See-dol match which consisted of something close to 2000 of CPUs and more than 100 gpus then the ratio in wins would be only be 3:1 in favor of the cluster. Much less than if you would compare it to the ratio of computational power so that gives you hint of how much of the intelligence in it is about previous knowledge gained vs brute computing power. To me that also implies that a lot of the innovation done at DeepMind is more about improving the speed of which a neural net learns and aquires knowledge and less about the running time of applying the model. And that's probably an important distinction in the discussion of this technology.

> If you take the naive approach and assume it's just a matter of getting models with the same number of parameters as the brain has synapses, our biggest models are about a million times too small.

And if you had that hardware, what would you run on it, exactly? There is no off the shelf algorithm which we could plop on that futuristic GPU and say, yes, this will definitely yield a general human-level intelligence. That is my point. We may or may not have the necessary hardware, but we sure don't have the necessary software right now.

I agree. But I often here the claim that we have enough hardware and now it's just a software problem. I don't think that's at all clear.

Reply on news.ycombinator.com