Introduction to Artificial Neural Networks – Part 1
theprojectspot.com
theprojectspot.com
Think about them like circuits. It is easy to see that a circuit full of NAND, AND, OR, etc can carry out computations. An AND gate is a gate that requires all of its inputs to be true and then it goes high. An OR gate requires at least 1 of its inputs to be true and it goes high.
Think of a neuron as being somewhere in between that (I understand that in the article he shows how to simulate gates, but doesn't quite describe neurons as being fancy gates (which is where my breakthrough happening)). It requires N of its inputs to be true and it goes high. Then imagine that the "wires" in our neural network don't carry information in the form of bits, but instead in the form of real numbers. Each wire has a "weight" associated with it, such that when the wire is turned on, it outputs that value, rather than a simple binary 1.
Now imagine that the neurons take all of these real numbered inputs, and apply some function to them to decide if the neuron wants to turn on or not. It might simply sum them, multiply them, or something more complex, but based on its inputs, it turns on. Its on signal then gets sent to all of the neurons that it points to, etc. The same way you can extract answers from a circuit of logic gates by seeing the output of the gates, you can extract answers from an NN by examining the output of certain neurons.
This description is quite simplified and doesn't go into the architecture of ANNs, but if you are really having a lot of trouble grasping how ANNs work, this description should give you some intuition. The hardcore ML people will probably dislike it, but you have to start somewhere. After understanding it like this, I branched out quite a lot and now all of my academic research involves machine learning. But it took that initial breakthrough!
This is probably just a personal preference though.
I find if you're tackling neural networks in the first place, it's great to have your fundamentals down in the first place though. I think a lot of the problem with people learning machine learning is that they just dive in to it without going back and understanding the moving parts that make up the whole.
I made that mistake personally when I was starting out a few years ago. I've seen that with others as well.
The final layer is just a standard logistic regression classifier in the new (usually higher dimensional) space.
Douglas Hofstadter, "Analogy as the Core of Cognition" http://www.youtube.com/watch?v=n8m7lFQ3njk
[1] http://books.google.no/books/about/Fundamentals_of_Neural_Ne...
Standard feed forward deep net: like the ones you used at university but with a few important features. One, you can stack layers on top of each other then train the whole network with back propagation. The nonlinearity you use can be important (rectified nonlinear units--Max(0,x) are popular now). Regularization can be important (with dropout being a popular method). Pre-processing your data can also be important (eg scaling the inputs, subtracting the mean, ...). How you initialize your weights is important. Using tricks like momentum and learning rate decay are important too.
Autoencoder: A standard feed forward deep net that tries to output an "uncorrupted" sample of a "corrupted" input. So basically, take an image, add noise, ask it to output the denoised image. Why you ask? Well, in doing this, the hidden units (the layers in between the input and output) tend to discover new features which can then be used elsewhere in machine learning pipelines. The big win here is automated feature engineering.
RBM: Restricted boltzman machine, sort of like an autoencoder but not. I'm not convinced one is better than the other, but there are definitely differences.
Recurrent neural networks: Like standard feed forward deep nets but extended to time series problems. The hidden units at each timestep feed in as additional input to the next timestep (along with the new input). So basically, each step it gets this as input: [whatever came before described as hidden units] + [input at T]. These are my personal favorite and currently hold state of the art in speech recognition, at least academically.
Convolutional neural networks: State of the art in vision, this one is kind of hard to explain quickly, but you can think of it kind of like this: I take some image where I want to recognize objects in it. Then I train a bunch of neural nets that represent different aspects of an image (one net might look for vertical lines, another might look for horizontal. It figures out what it's looking for automatically and distributes what it thinks are important across these little networks. You can think of this as "feature detection") You then take a window on the image (400x400 image, maybe you've got a 20x20 window) which you then slide across the image and check each section for the presence of whatever the net is looking for (vertical/horizontal lines, etc.) There's several layers of this (specifically a convolution operation followed by a pooling operation) before the result is fed to a standard fully connected feed forward net which then outputs a prediction.
While lower layers look for low level aspects of an image, each layer progressively looks for higher and higher level phenomena--for instance, when you hear about "the cat neuron" it comes from probing a neuron in the higher levels of a convolutional net and finding that pictures of cats happen to turn this neuron "on" while pictures of anything else don't. What is "high level" exactly? In this case, it really just means composed from "lower level" components... Some also say "higher level = more abstract" but I don't know this is quite the case--abstract means something different to me.
An important consideration is that the conv net architecture allows you to cut down on the total number of parameters (good when you've got something high dimensional like image data) and it also takes advantage of the fact that you can make certain assumptions about images and how objects move/appear in space basically.
Conv nets are probably the most difficult architecture to grasp and my explanation is extremely high level.
Ha, that was a bit longer than I had hoped, but I hope it's helpful to someone...
I might be totally off-base, but is Numenta's HTM (Hierarchical Temporal Memory) model a conv net? If so, there are some really good introductory/TED-talk-like explanations out there for the idea.
Personally, I think there are two big issues with what numenta was doing: they diverged too far from mainline neural net research (which, when they started in ~2005 was just before things started getting interesting--it was also a time when "mainline neural net research" was widely assumed to be at a dead end) and they tried too hard to come up with something that was biologically plausible rather than mathematically expedient. Sort of how airplanes need wings but don't need feathers. As far as I am aware, HTMs really just don't work well in practice (and by that, I mean they are not anywhere near competitive with any of the architectures I listed above).
You could, in other words, see a recurrent conv net as an opmitization of an HTM given a von Neumann architecture, or the reverse -- an HTM as an optimization of a recurrent conv net given a biological substrate (where it's much less costly to build tons of crummy processors and link them into an arbitrary graph, than it is to build a single fast processor.)
Again, though, I'm not an ML person, so I might be way off.
- Use the tanh activation function.
- Use many hidden nodes, e.g. 100 or 1000 for complex tasks.
- Tune the learning rate.
- Add a penalty of k*w^2 to each hidden node. Tune the parameter k to minimize the held-out error.
- If this isn't working, plot the error on the training set as you train. It should decrease to 0.
Read Yann LeCun's efficient backprop work for more tips.
Numpy based neural networks are a lot more digestable as far as understanding all of the moving parts due to the easy syntax.
Neural nets are usually verbose if you use an object oriented representation. Things like feedforward and backpropagation don't require loops if you know how to think about individual neurons and weight matrices.
Also, for working with them, here's something that cleared up a lot for me: when initializing neuron weights, it's randomly initialized. It seemed like magic to me at first and kept tricking me up when getting started with them.
For those who do the JVM, I'm working on finishing up a library for myself to use in various bits of work I do.
Java based, but you get access to fortran matrix routines, and it's interopable with scala and the like as well.
https://github.com/agibsonccc/sda-jblas
For those of you who need hadoop or want to scale out neural networks, I'm going to suggest a friend of mine's iterative reduce project for neural nets on hadoop.
https://github.com/jpatanooga/Metronome
I just finished up a parallel training setup I'm going to release based on this for neural nets as well. When working with them, you'll discover crazy long training times. Multicore tends to make things easy if you know how to balance out the weights though.
I'd be happy to answer any other questions as well.
TL:DR Neural networks are super powerful, but really hard to use properly for anything slightly harder than simple functions.
If you're interested I wrote a paper in Nature Systems Biology that shows this for artificial gene networks (just another branch of ANNs): Survival of the Sparsest: Robust Gene Networks are Parsimonious
In the most simple model, this sum is the output. In more sofisticated models the result of the sum is modified by a function with a sigmoid shape. One of these sigmoid function is infinitly derivable. This allows to fit any continuous function with a sum of these sigmoids. Different strategies have been suggested to do the fit.
But all these ANN don't explain the biological neural networks found in our cortex.
It is a fascinating problem which is still lacking an answer.
People do interpret ANNs. Normally by visualising the weight matrices on the input layer, which have a 1-1 mapping to input vector attributes (so you can label them). It gets a bit hard on subsequent layers though.
decision trees are an order of magnitude easier to interpret though, they are very compact and consist of a number of binary decision in the input space. For a given classification you only need to look at log(d) decisions to work out how it got to its conclusion. Its not usually that hard to how it arrived at the tree from training.
The relationship between the Jacobian and the training is fairly convoluted, given back propagation and cross validation.
PS so back-prorogation uses weights differentiated against outputs. Jacobian is input differentiated against outputs. You can do smart things with Jacobian aware neural architectures. See "Forward models: Supervised learning with a distal teacher" which trains two networks in parallel, one a forward model, one an inverse model, and uses the Jacobian to circumnavigate the problematic non-convexity of the inverse model.
Yeah it doesn't seem like it would be easy to gain much insight since the partial derivatives are a function of the (other) inputs.
> People do interpret ANNs. Normally by visualising the weight matrices on the input layer, which have a 1-1 mapping to input vector attributes (so you can label them).
What do the first-layer weights tell me?
If the neural network is processing images, the weights form and image too. You can tell what a lot of the units are "looking" for by plotting the weights as an image.
page 14, Figure 3: jmlr.org/papers/volume11/erhan10a/erhan10a.pdf
You can clearly see various digit deformations in the weights. (white is positive weight, black is negative weight typically)
Lots of papers do this, this one was happened to be the first one I managed to google.
So the first layer is normally readable because weights are in the same space as the input feature space. The second layer is normally a jumble as its randomly initialised before convergence. But in the paper's example you can imagine the 2nd layer output unit representing a final classification of a 4, is probably summing up all the 4 deformation detectors in layer 1 and negatively summing up everything else.
"For visualizing what units do on the 2nd and 3rd layer, we used the activation maxi- mization technique described by Erhan et al. (2009)"[1]
So there you have it! You can visualise all of a neural network! But you have to implement some specific numerics to get the data out. Reading the first layer is trivial, and subsequent layers need some analysis.
[1] Erhan 2009 Visualizing higher layer features of a deep network
(good find, thanks for asking the right questions!)
But that doesn't tell you much does it? It's effect could be reversed or minimized in subsequent layers.
Haven't read it yet but thanks for the paper.
Because your final layer is positively encoded, that polarity trickles down through back propogation. I also think the positive weights thing is to do with the way features normally work. It makes sense to look for corners, not anti corners for object detection.