Hacker News Reader

Comment by dlsym | Hacker News Reader

"What does a neural network actually do?"

This is a fundamental Question: Can we really say and predict what a neural network does?

Contrary to an engineered / constructed algorithm a neural network is 'trained'.

Whenever we will present a 'known' input pattern it will reposnd with a 'learned' response.

This however introduces interesting problems: How can we _debug_ a neural network? How can we debug a correlation? Sure we can tune its paramters, we can train it some more to again show the desired response. But reaching this point we just abandoned knowing how the intrinsic algorithm works in favor to just focussing on the result.

Okay - now if we follow this argument - this would lead to: If we simulate the whole brain by simulating the neural network, we won't gain any knowlege about the intrinsic workings of the brain. We won't find any enlightenment about the innermost algorithm _represented_ by the neural network we call our brain.

View on HN

I think you have hit upon a problem of present day AI.

Neural networks, Support Vector Machines, Hidden Markov Models and other stuff (Markov Networks, etc) do something like linear regression on some huge space - they draw a curve/plane between groups of things on this feature space. The tendency is for this division to make sense and to correspond to our common sense categorization of these things.

The problem is that once that happens, you can't really reason about the division you've drawn. It's just there. You can tweak for various purposes but that's a manual process.

You categorize animals by shape or particular adaptation or by genetic makeup. You can teach one of these algorithm each of these categorizations. But you can't do something like have the thing categorize for one purpose and then tell it to "change it's outlook" and categorize for a different purpose.

In this sense, despite seeming impressive, the products of these processes are dead-ends that we can't reason about, that lack the flexible intelligence of a human being.

Your criticism is fair, but you fail to explain how an NN or SVM is any worse than how a human mind actually operates.

In other words, the incomprehensibility of a modern AI model is not a failing of AI, it is a failing of (AI) psychology and (AI) neuroscience.

The artificially constructed intelligence works whether or not we understand how. The frontier of AI science is now open to AI Psychology. Psychologists and Neuroscientists will replace the data scientists! Such fun!

I could hardly contrast the operations of a NN to those of the human mind because we most definitely don't understand the later. I have already described apparent properties of human minds that NN and SVN definitely don't have. But I'll repeat and expand:

The artificially constructed intelligence works whether or not we understand how

The operation of human intelligence is very dependent on the fact that we human have operational understanding of each other's mental processes. Moreover, it is well known that human beings process language and can re-evaluate past experience in light of present understand.

Oppositely, we know enough about the properties of the various non-linear recognizers to know that constructs can't do and won't ever be able to do these things.

I can't find an example on Google right now, but I've seen demonstrations that it's possible to visualize the intermediate layers of a neural network - for example you can see how an image recognition network is first breaking down an image into horizontal and vertical lines, then combining those into more complex shapes, etc.

These are called filters. This is from my deep learning framework: Debugging a net visually: http://deeplearning4j.org/debug.html

An example of doing facial reconstruction: http://deeplearning4j.org/facial-reconstruction-tutorial.htm...

But visualizing is quite a ways from debugging.

To debug a program you actually verify that it's logic is correct (at least the good kind of debugging).

Consider a spectrum:

1. Natural language - we humans combine fragments of natural language easily and on an ad-hoc basis. We can get a fairly amount of use from reusing Shakespeare quotes and neologisms while spending rather little effort.

2. Trained programmers can reuse and combine general purpose libraries - with difficulty and often after considerable debugging.

3. AI algorithms like Neural Networks. These are just plopped in and tweaked, not combining seems possible.

It seem like "intelligent behavior" should be going more towards #1 but the process of Machine Learning seems to move things more towards #3. The "learn once, understand never" approach means that for each significant case, you'll need to do a re-tweaking and re-learning. The potential to get harder rather than easier over time might well be there.

Can you debug a human brain? I can't.

Is a human brain intelligent? I believe so.

Well,

Admittedly, all this is in manner of speaking but still, I would claim that most if not all the times you debug a program you are also debugging your mental concept of what the program does. By that fact that we can change our concepts, our minds are very "debuggable."

There's an interesting example in one of the coursera courses (Neural Networks for Machine Learning) - you just need to watch through the intro video to see it in action.

https://www.coursera.org/course/neuralnets

Reply on news.ycombinator.com