The Google Brain Team – Looking Back on 2016
research.googleblog.com
research.googleblog.com
"The Great AI Awakening"
> The new incarnation, to the pleasant surprise of Google’s own engineers, had been completed in only nine months. The A.I. system had demonstrated overnight improvements roughly equal to the total gains the old one had accrued over its entire lifetime.
https://www.nytimes.com/2016/12/14/magazine/the-great-ai-awa...
(I wrote a longer-form answer to a very similar question a few months ago: http://da-data.blogspot.com/2016/05/dont-quit-that-programmi... )
I don't see how ML change the scenario, instead of querying some db, you are now querying some trained models. Until we have an algorithm that could write correct code given requirement specified in natural language, the demand or prospect of software engineer as a profession will not be affected by ML/AI too much. However it is hard to predict when such technique will emerge, 10 years? 20 years? maybe 50 years.
So at the very least they are working on it!
But so far all are just like toy examples. The big barrier here is natural language understanding, so far we have limited success even with deep learning at hand.
If eventually natural language understanding is solved, it is going to be a much bigger deal than programmer losing their jobs.
http://www.primaryobjects.com/2013/01/27/using-artificial-in...
This is the branch of IA/ML that turns me on the most. When we get there, it will redefine many jobs (as well as create an existential thread as a side effect).
I think it is sad that the terms AI and Machine learning do not make a clear difference between narrow and strong AI, this is the source of much confusion.
> Your job is to ask questions and to find edge cases that the product people didn’t think of. Your job is to help operations define processes well enough to be automated.
https://medium.com/swizec-s-nightowls/what-should-you-learn-...
Right now it takes 5 to 10 years to train a human who already knows how to code to do this. Some never learn.
I think we're safe for a while.
Engineers don't "write code", they build solutions. We just happen to use code to do it. And we happen to manually type that code. Even if an AI could type it for you, you'd still need someone to build the solution.
Even with currently available programming techniques, simply getting many coworkers to move from "WET" to "DRY" can be a daunting challenge.
Many people just hate abstractions. ML / AI tools seem to often just boil down to larger aggregations of abstractions (which require some training and/or experience to learn). I guess the "we enjoy [lots of] typing" crowd will go the way of the dodo as the tooling moves up abstraction levels. I hope they don't suck me down with them :-(
(yes, I'm glossing over the requirements gathering aspect of the comment to carp about the construction aspect, but the same strategy of abstraction/investigation, or not, applies)
But, the majority of developers I know, including myself, are ultimately writing software to be used by humans, typically presenting data that is comprehensible to humans, so that they can make decisions.
If AI advances to the point where similar decisions can be made by machines, looking at the same data, then I fear that entire classes of applications will disappear, and with them, the need for developers who build them.
Obviously, we'll still need developers, though perhaps far fewer than are needed today. Perhaps software development will go through the same revolution that agriculture did, where from we went from > 90% to < 3% of the population being involved.
Did FORTRAN reduce the need for programmers by offering a compiled language?
What part of the decisions involved in that process have "magic" attributes that makes them uniquely tied to the meat-based data processor in your head?
1 - introduction of automation vastly increases throughput of industry, either eroding a traditional
2 - industry vastly increases demand of technically and scientifically literate workers
3 - folks compete for contemporary educational/employment opportunities in industry
4 - an engineer acclimates to their niche
If you're already in a stage 4 situation, it may seem like there's always a new blockbuster industry in stage 1. But, the transition into stage 3 for a new technology may seem insurmountable. That said, technologies rarely meet their hype..
Either way, engineering is a subset of problem solving, and people will always have problems. It would be great to assimilate all kinds of problem solving skills that you can.
So we better hope that the BCIs develop quickly as well.
I haven't read much of the literature around deep learning, mostly only what has been applied to computer vision. But from what I understand, the general consensus is that the current crop of state-of-the-art deep learning models are very good at performing specific tasks (machine translation, object recognition in images, etc.), but are not so good at generalising across multiple fields. This seems more relevant to the domain of reinforcement learning (which does indeed include deep learning), which has proven to be a very difficult problem to solve.
I am working on a webpage to try to break down why I think this is coming so fast.
https://www.youtube.com/watch?v=BP7vhBaBDyk&t=4589s General Reinforcement Learning
https://www.youtube.com/watch?v=T9eSVYLSSrs The Emotional Mechanisms in NARS
https://www.youtube.com/watch?v=eVrflIw6sGg&t=4056s AGI-15 Keynote by Jürgen Schmidhuber - The Deep Learning RNNaissance
AGI(-16) is a cognitive science / philosophy conference (and not particularly high impact). Similarly, NIPS is not really about neural information processing.
AGI is in fact a developed field with key insights into general intelligence. You should study it.
There aren't many active researchers/experts commenting on HN (better things to do), which is IMO a big issue with the ML-related discussion quality on HN (it's basically 90% futurism/speculation).
The human brain is full of meta upon meta levels. Networks on top of other networks, and so on. Abstractization. Correlation between different domains. Feedback loops. There will be no AGI until we start building that kind of architectures.
We're currently only making individual Lego bricks. We haven't started to assemble them yet.
Networks on top of other networks -- about any recursive neural network could be in that category. Also there are multiple projects exploring metalearning on deep NNs where it learns the network topology, i.e. learning how to learn.
Abstraction in single domains is very commonly achieved with various types of hierarchy in NN and non-NN systems. Correlation between different domains has fewer examples simply because most AI systems do not deal with multiple domains, but they do exist.
Feedback loops are common and fundamental to AI and simpler non-AI control systems.
I don't know when, but I can guarantee it won't be within 5 years. The reason: digital hardware improvements are slowing down. Today's best hardware can only run a model with the number of nodes in a human brain at 1/2400th speed, and probably with an oversimplified model: http://www.telegraph.co.uk/technology/10567942/Supercomputer...
I don't think we will reach human level AI on digital hardware; it just doesn't really make sense with the Von Neumann bottleneck. Parallel, analog computation is the way forward and that has yet to be commercialized, so we probably have a ways to go.
That being said, AI is getting very impressive on the software side, and with the use of TPU's, GPU's, and FPGA's, we can definitely look forward to compelling use cases where these algorithms have apparently creative intelligence, but only within an extremely specific category (when compared to the generalities of biological analog brains).
It's definitely worth every software engineer's time to learn how computers work from all the way down at the laws of electricity, basic signal processing/analog circuits, up to semiconductor physics and microarchitecture.
I definitely see how my language would convey that whereas I mean to refer to digital computers, not all hardware, when I mentioned the Von Neumann bottleneck. There are different arguments against digital hardware in general, even though FPGA's can be about an order of magnitude more efficient than microprocessors. Modern analog goes much further, although we're still not very good at 'programming' it, which is why I think AGI is a ways out.
Also, perhaps I'm misreading their paper[1], but that DARPA project is still using digital logic. In fact, they specifically say that they decided against using analog computation due to the increased density of digitial (one of the main problems with analog computing, as I mentioned). Regardless, it's pretty clear that the project doesn't have the goal of replacing general purpose computing, so I'm curious why you consider this evidence that analog computing will replace digital computing?
Finally, in what way are FPGAs much closer to analog computing than microprocessors? The advantages of FPGAs (configurability) are orthogonal to analog vs digital.
The topic is AI, where efficient, large scale neural nets will require fundamentally different hardware, like the programmable mixed signal Samsung boards used by Synapse. Those are best described as specialized FPAA's and they certainly do rely on digital for programming, and special (although still solid state circuits) for pseudo-analog properties. However, the most interesting parts are yet to come, and are being developed by partners/founders of the program: http://knowm.org/about/
The next generation boards are likely to contain memristors and self organizing memory, with a structure somewhere between the organization level of FPGA LUT blocks and biological nervous systems. This will obviously need to interface with/through digital circuits to be of any use.
However, I still take issue with your overall point.
> Efficient, large scale neural nets will require fundamentally different hardware, like the programmable mixed signal Samsung boards used by Synapse.
I think it's at least plausible that we'll see gains from fundamentally different architectures (namely combining memory and computation) but where's the evidence that we need to switch from digital to analog? Where's the evidence that analog is even better for an architecture like that? Again, the DARPA project you linked actually gave evidence of the opposite.
> The next generation boards are likely to contain memristors and self organizing memory, with a structure somewhere between the organization level of FPGA LUT blocks and biological nervous systems.
You keep talking as if there's some obvious reason that analog is more efficient/faster/better for the types of architectures you're talking about. I'm not seeing that reason.
Digital is more efficient for some problems in the sense that it's the only way to achieve precise computational results, but even when it comes to mathematical arithmetic, the circuitry to add two analog signals is much faster and simpler (two wires and a resistor), as is the design of a multiplier (one transistor).
It's definitely a complex subject, but I don't see why you would find it so difficult to believe -- the most intelligent machines we know of are analog and decentralized (like human brains).
That's not necessary true; again, see the DARPA project you listed.
> Digital hardware adds an additional abstraction layer built out of components that do not behave like the desired structures (neurons) and are basically running simulations.
The analog hardware doesn't necessarily behave like the desired components either (for example, due to manufacturing errors). With digital you get to choose your accuracy and with much greater control.
> Digital is more efficient for some problems in the sense that it's the only way to achieve precise computational results, but even when it comes to mathematical arithmetic, the circuitry to add two analog signals is much faster and simpler (two wires and a resistor), as is the design of a multiplier (one transistor).
Unless you don't care about noise at all, your design will be much more complicated than a single transistor. Either way, it's actually much easier to do this sort of "approximate computing" in the digital domain.
> It's definitely a complex subject, but I don't see why you would find it so difficult to believe -- the most intelligent machines we know of are analog and decentralized (like human brains).
A bird has feathers yet our most efficient airplanes don't use them; there's no reason at all to believe that just because the brain is analog, our design should be as well.
The reason I make this analogy is that programming isn't fun†. It's maddeningly difficult. As far as I'm concerned, plugging ML nodes into client-side interfaces can't get here fast enough.
† I don't care if you "think" you like it - just like someone might think they like sweeping excrement all day.
Disclosure: I work for Google on the Brain team.
Secondly, a single AI brain is composed of many small AI brains in a society. Also those brains, recursively down forever recursive.
I do not need the functionality personally but I want to see it in my lifetime.