Our Machines Now Have Knowledge We’ll Never Understand
backchannel.com
backchannel.com
>As a consequence, if you, with your puny human brain, want to understand why AlphaGo chose a particular move, the “explanation” may well consist of the networks of weighted connections that then pass their outcomes to the next layer of the neural network. Your brain can’t remember all those weights, and even if it could, it couldn’t then perform the calculation that resulted in the next state of the neural network.
Why did you reduce the neural network to the connections and layers but maintained the “brain” at the level of the whole? If you want to compare apples to apples you should also reduce the brain to neurons and connections. When a ball is about to hit your face your brain “can’t remember how to calculate the velocity and acceleration of the ball and predict that in 100ms it will impact your face and thus you need to send a signal for your hand to move to cover your face”. Yet we do it, how? Well, it’s encoded in the complex system of nerves and networks just like the the information about the various potential Go boards and moves is encoded in the neural network.
>And even if it could, you would have learned nothing about how to play Go, or, in truth, how AlphaGo plays Go — just as internalizing a schematic of the neural states of a human player would not constitute understanding how she came to make any particular move.
Similarly if you were to study the neurons, nerves, and networks and their firing patterns that are involved in protecting your face from the ball you wouldn’t learn anything about balls, acceleration or even pain.
It's likely doing optical flow calculations (much simpler), rather than integrating an ODE. Richard Dawkins apparently makes the same mistake about "catching a ball" in his book.
If so, that seems like it would be insufficient since a thrown object doesn't move linearly, let alone its projection.
If not, then I'm curious what you're describing, and how it's much simpler than integrating an ODE?
https://youtu.be/eKaYnXQUb2g?t=6m10s
https://www.youtube.com/watch?v=iz9UVIo_ZUo&list=PL0AknDL1Vt...
Camera homography preserves straight-lines, but it also gives a heuristic to avoid obstacles. Apparently this is also how bees and birds navigate.
That is pretty clearly not a useful definition of understanding.
A brain 2x the size does not mean it must learn twice as much, or can understand 2x more complicated things. But may just mean it can learn twice as fast.
There was a machine learning system specifically designed to produce interpretable models. It's called Eureqa, and it does symbolic regression. It finds the simplest possible mathematical expression that fits the data. Instead of millions of neural weights, you get a nice simple equation. How could not not be interpretable?
But in any nontrivial case, it's still impossible to decipher. "Why is there a sine there? Why on Earth is it using mod? What could this constant possibly mean?" One biologist used it on some data he had been struggling with. And it found a simple equation that fit the data perfectly, which is what he had been looking for. But he couldn't publish it, because he couldn't possibly explain why it worked or how to derive it. You can't just publish a random equation with no understanding.
I think the best method of understanding our models, is not going to come from making simpler models. Instead I think we should take advantage of our own neural networks. Try to train humans to predict what inputs will activate a node in a neural network. We will learn that function ourselves, and then it's purpose will make sense to us. I suspect most of what NNs do isn't complicated in an absolute sense. Its just time consuming to go through the data and work it out. (Maybe we could outsource it to something like mechanical turk... "What do these images have in common?")
There is a huge amount of effort put into making more accurate models, but much less into trying to interpret them. I think this is a huge mistake. Understanding a model lets you see it's weaknesses. The things that it can't learn. The mistakes it makes.
This has happened a lot in the past iirc: https://en.wikipedia.org/wiki/Unsolved_problems_in_mathemati... Imo if you can 'proof' it's really true, publish it, maybe some else some day can actually proof/explain it. Meanwhile we can use the formula where it works (at your own risk).
This kind of breathless "Oh we CAN'T understand!" editorializing gets clicks because machine learning is so strange to most people, but it's terrible. We should instead be spreading knowledge about the very simple principles that are underlying the complicated behavior.
Our brains are far, far more complicated than even our most advanced machine learning right now. Yet we are constantly making progress on understanding the workings of our brains. This article is just terrible.
> Dr. Thompson peered inside his perfect offspring to gain insight into its methods, but what he found inside was baffling. The plucky chip was utilizing only thirty-seven of its one hundred logic gates, and most of them were arranged in a curious collection of feedback loops. Five individual logic cells were functionally disconnected from the rest— with no pathways that would allow them to influence the output— yet when the researcher disabled any one of them the chip lost its ability to discriminate the tones.
(from https://www.damninteresting.com/on-the-origin-of-circuits/)
Granted, the results were probably due to the quirks of the FPGA gates on that particular chip (and that particular area of the chip) but without further (potentially destructive) investigation, Dr. Thompson (or us for that matter) may never know the actual reason that circuit worked the way it did.
If you don't understand how a thing works, how the hell do you plan to fix it when it doesn't?
There's also a fringe benefit here. If "we can't understand" what it's doing, then that assumption can appear to weaken the "auditability" argument for deploying FLOSS in life-critical AI systems. (I.e., freedom 1 from the GPL.)
I'll bet the argument will be put forward with more frequency over the next few years, especially as self-driving cars become more prominent. Perhaps "appear to weaken" is too strong an assertion. But I can imagine a consumer safety advocate arguing for code to be auditable by outside parties just like Signal can be, and a PR spokesperson deflecting that AI is "beyond our understanding" and therefore not subject to the same standard.
Aren't we constantly iterating over these AI algorithms specifically to give us ones that can help us understand things "We'll Never Understand"?
And to answer a probable objection, "It's BigProblem, all the way out."
Learning is not some abstract mystery, it is a process of "evolving" and maintaining a structure (of neurons or of bits - does not really matter) which reflects some particular aspects of reality (presumably not socially constructed) and could be used in a meaningful way to have advantage in the similar environment. This is how birds know how to make nests or babies read facial expressions of strangers.
Knowledge is not some abstract philosophical crap, it is a "trained" structure with reflects aspects of what is.
BTW, the structure of the sand floor of a lake, which reflects all the past waves could be called "knowledge", but it is totally useless. Most of flawed, socially constructed models are of this kind.
(Is anyone excited about the implications AI has for philosophy ?).
They then introduced several different ML add ons, like Vince, Panda, Penguin, etc. They know roughly what the algorithms are doing. But, as I understand it, it's still opaque enough that they can't explain exactly why a specific set of results is what it is. Especially since they don't all work together. They run one after the other, like a pipeline.
That is, there are some amount of false positives or negatives they can't explain. If you report something, they can guess and experiment around a bit.
For some spaces, they are basically king makers or breakers. So ML decisions are driving real world consequences. To the degree, though, that the results are "good enough", it won't ever get looked into. The acceptable error rate is the one that doesn't affect Google too much. An actual flaw could go unnoticed for quite some time.
The fact that we can't observe specifically how a brain works doesn't mean we can't work out some structure. The fact an ant doesn't explain to us the individual rules to us doesn't mean we can't divine them via observation.
These articles irritate me, because they act like NNs are magic and they are anything but.
This is alarmist and completely untrue. The scientific method specifically involves forming a hypothesis, trial, and error. We don't know much until we experiment.
That deep learning makes things possible before we fully understand them is nothing new.
I can understand this fearful attitude coming from non-scientists who fear the black-box nature of machines. However, technologists should understand that the functionality of this tech can be reverse engineered, or verified, with additional legwork.
We should proceed cautiously and with vigilance. Saying we'll "never" understand the inner-workings of such algorithms is too much.
Our brains are fallible, and so only approximately equivalent to Turing machines; and even ignoring that can only compute every computable thing given both infinite error-free storage capacity (which they don't have internally or externally) and sufficient time to execute the necessary steps of the computation.
Realistically, there are computable functions which no human brain will be ever be capable of computing.
Whether being able to compute a thing is sufficient, necessary, or tangential to understanding it is also a question.
So are actual computers. They're just "less fallible" and closer to a Universal Turing Machine than human brains in certain ways. A Commodore 64 is also incapable, in practice, of computing certain computable functions, due to memory limitations and what have you, but nobody would really claim a Commodore 64 is not Turing complete.
Sure, but no one was making claims about actual computers other than the human brain that requires pointing that out.
> but nobody would really claim a Commodore 64 is not Turing complete.
Actually, it's a rather common observation that real-world computers are not Turing complete (languages, considered independent of the limitations of concrete machines, may be) and particularly that concrete machines with limited storage space and operating with finite time constraints may not be able to compute all computable results, even though the abstract model they approximate, without those limitations, can.
I'm not sure a human can compute any algorithm beyond a certain complexity. But maybe if that human never aged or got bored and had an endless supply of paper. But then you're not dealing with a real human being anymore. Even if you had a super patient, super long lived human, what's to say they wouldn't get indefinitely stuck with really involved calculations?
i) Tell a person how a Turing machine works,
ii) Give her reams of paper
iii) Make her miserable by running some instruction.
iv) ?
v) Profit!
Seriously though. I don't think Turing completeness means anything at all; we're increasingly beginning to understand how important representations are. As hackers I'm sure we understand what this means in terms of PLs, but this is no coincidence IMO.
But given enough time and memory, shouldn't it be possible?
Although maybe that would require more than the universe has available, in which case I suppose it would be impossible in that sense I guess.
Darpa is not pumping money into safety on NN Nets research because confetti ran out.