Neural Networks from Scratch
aegeorge42.github.io
aegeorge42.github.io
The explanation of the calculus in this tool is equally fantastic. And the art is very cute.
There are many ways to skin a cat, of course, but this is as good a tutorial as I've seen for getting you through backprop as fast as possible.
It starts off with some tutorials using the Keras library, and then gets into the math later on.
By the end of the book, you create multiple different types of neural networks for identifying images, text, and more! I highly recommend it.
Fastai is top down: learn to use practical ML with abstractions, and then dig deeper and explain as needed.
I preferred fastai's approach, even though I enjoyed both. Ng's could be a bit too low level and fundamental for what I wanted to learn.
I tried Fast AI, but it seems to be trying too hard to take out the math, which oddly for me (as a STEM grad) makes it much more difficult to understand.
Had to stop when I saw him using Excel spreadsheets to explain convolution.
While backprop has major historic significance, in the end it's essentially just a pure calculation which no longer needs to be done by hand.
Don't get me wrong, I still believe that understanding the gradient is hugely important, and conceptually it will always be essential to understand that one is optimizing a neural network by taking the derivative of the loss function, but backprop is not necessary nor is it particularly useful for modern neural networks (nobody is computing gradients by hand for transformers).
IMHO a better approach is to focus on a tool like JAX where taking a derivative is abstracted away cleanly enough, but at the same time you remain fully aware of all the calculus that is being done.
Especially for programmers, it's better to look at Neural Networks as just a specific application of Differentiable Programing. This makes them both easier to understand and also enables the learner to open a much broader class of problems they can solve with the same tools.
I'm not entirely sure what you're referring to by "this" but assuming you mean my comment, I think what I'm saying is very much up for debate and not an "obvious and commonly accepted fact". Karpathy has a very reasonably argument that directly disagrees with what I'm suggesting [0]. Of course he also agrees that in practice nobody will every use backprop directly.
Whether it's JAX, TF, PyTorch, etc the chain rule will be applied for you. I'm arguing that I think it's helpful to not have to worry about the details of how your derivative is being computed, and rather build an intuition about using derivatives as an abstraction. To be fair I think Karpathy is correct for people who are going to be learning to explicitly be experts in Neural Networks.
My point is more that given how powerful our tools today are for computing derivatives (I think JAX/Autograd have improved since Karpathy wrote that article), it's better to teach programmers to learn think of derivatives, gradients, hessians etc as high level abstractions. Worrying less about how to compute them and more about how to use them. In this way thinking about modeling doesn't need to be restricted to strictly NNs, but rather use NNs and example and then demonstrate to the student that they are free to build any model by defining how the model predicts, scoring the prediction and using the tools of calculus to answer other common questions you might have.
edit: a good analogy is logic programming and backtracking/unification. The entire point of logic programming is to abstract away backtracking. Sure experts in Prolog do need to understand backtracking, but it's more helpful to get beginners understanding how Prolog behaves than understand the details of backtracking.
[0] https://karpathy.medium.com/yes-you-should-understand-backpr...
i am saying that if you want to work with ML algorithms on a more deeper level you must learn backprop
if you want to implement some models on the other hand, you can just follow a recipe approach
Why should I, though? I remember the concept from calculus. I know pytorch keeps track of the various stuff I do to a vector and calculates a gradient based on it. What more do I need to know when all I want to do is to play with applications, not implement backprop myself?
Here I disagree with you pretty strongly. Once someone is comfortable with differentiable programming it's much more obvious how to build and optimize any type of model.
People should be more concerned about when to use derivatives, gradients, hessians, Laplace approximation etc rather than worry about the implementation details of these tools.
Abstraction can also aid depth of understanding. I know plenty of people who can implement backprop, but then don't understand how to estimate parameter uncertainty from the Hessian. The latter is much more important for general model building.
EDIT: also uncertainty estimation is the stuff of probabalistic approach to ML. i would say that people who do probabalistic ML are quite mathematically capable (at least to my experience)
Are you sure about this?
It doesn't have to be that way. The hessian is an abstract idea and the chain rule and more specifically backpropagation are methods of computing the results for an abstract idea. When I want the hessian I want a matrix of second order partial derivatives, I'm not interested in how those are computed.
For a more concrete example, would you say that using the quantile function for the normal distribution requires you to be able to implement it from scratch?
There are many, very smart, very knowledgeable people that correctly use the normal quantile function (inverse CDF) every day for essential quantitative computation that have absolutely no idea how to implement the inverse error function (an essential part of the normal quantile). Would you say that you don't really know statistics if you can't do this? That a beginner must understand the implementation details of the inverse error function before making any claims about normal quantiles? I myself would absolutely need to pull up a copy of Numerical Recipes to do this. It would be, in my opinion, ludicrous to say that anyone wanting to write statistical code should understand and be able to implement the normal quantile function. Maybe in 1970 that was true, but we have software to abstract that out for us.
The same is becoming true of backprop. I can simply call jax.grad on my implementation of loss of the forward pass of the NN I'm interested in and get the gradient of that function, the same way I can call scipy.stats.norm.ppf to get that quantile for a normal. All that is important is that you understand what the quantile function of the normal distribution means for you to use it correctly, and again I suspect there are many practicing statisticians that don't know how to implement this.
And to give you a bit of context, my view on this has developed from working with many people who can pass a calculus exam and perform the necessarily steps to compute a derivative, but yet have almost no intuition about what a derivative means and how to use it and reason about it. Calculus historically focused on computation over intuition because that was what was needed to do practical work with calculus. Today the computation can take second place to the intuition because we have powerful tools that can take care of all the computation for you.
and that tool is backprop. if you do not understand what the chain rule is and what it is doing, that tool will be magic to you and you are blindly trusting its correctness. seeing that alot of risk is involved in using AI models in real life, blindly trusting your model is not a good approach
i agree that simply regurgitating rules of calculus is pointless to understanding. but thats definitely not what i mean when i talk about the need to understand the chain rule
ML is a mathematically intensive subject. there is no going around this fact
Pearlmutter, B.A. and Siskind, J.M., `Reverse-Mode AD in a Functional Framework: Lambda the Ultimate Backpropagator,'
I think you can accept that the chain rule is a thing, without understanding it, and then go further to understand its application to backprop.
I blame the common mathematical notation for that.
First he explains the necessary concepts:
1) Chain Rule
2) Computational Graph
Then he explains backpropagation in these three steps (first in general and then with examples):
1) Forward pass: Compute loss
2) Compute local gradients
3) Backward pass: Compute dLoss/dWeights using the Chain Rule
It is more useful to understand them as a series of transforms that bend and fold the input space, in order to place pairs of similar items close to each other. I would like to see people trying to illustrate that instead.
It also has the benefit of making the connection with linear algebra much easier to understand.
[0] https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/
I think it’s useful to understand it in both ways, but our intuitions about transforms are largely useless when the number of dimensions is high enough.
As you said, it's a sequence of transformations.
NB. If it's a graph: write out the edge list (etc.) .
NNs are diagrammed as graphs, but this is highly misleading.
They are also very much modeled after the brain, more specifically they originate from a 1943 paper by neurophysiologist Warren McCulloch and mathematician Walter Pitts who described how neurons in the brain might work by modeling a simple neural network.
Of course it’s not an accurate model, but it very much is based on early understanding of biological neurons.
However, it is difficult to see how a neural network that includes operations like sum, multiply and tanh, might be modeled this way. How do you describe a dropout as a graph?
I think the argument is that a graph is not sufficient to describe a NN, so technically speaking a NN is not a graph. It is more. It has edges between x and f(x), but we also need to specify what f(x) is. The mathematical definition of a graph doesn't do that.
As to the rest separating the NN from everything needed to generate it is a useful distinction. You’re not going to generate a different f(x) by slightly changing the training set etc. It’s however a somewhat arbitrary distinction.
What does any modern NN compu-sci model correspond to?
You're making a genetic fallacy here: that since the origin of X had property P, so must X. Eg., marriage was historically a legal contract of ownership, so today it is a legal contract of ownership.
The brain has nothing akin to back-propagation, nor supervised training. "Neurones" are not "data about house prices". According to any even highly simplified neurological model, a compu sci model of this, would compute with activations in a radically different manner than a dot-product over domain-specific data.
They are also not a weighted directed graph. In a sense almost any mathematical function is a weighted graph if one takes edges to be, eg., general function composition; and nodes to be input/output domains of those functions.
However to say a "NN is a network" is presumably meant to be more significant than this trivial observation that essentially "all functions" are.
In particular everything which distinguishes a NN from any mathematical functions are operations such as supervised training, backprop, regularization, pooling, etc. And these are not expressible as functions within this graph, but they are functions over the graph.
So any actual "network diagram" presented isn't a network. Its a bad diagram of a mathematical function which also includes functions over this diagram.
In any case, a neural network -- in the ML sense -- is neither neural nor a network. It is a function of a mathematical function that has no isomorphism to anything in the brain.
Also, the brain does actually use a form of supervised learning. Pain for example is effectively a form of input labeling and lack of pain the wanted output.
And it's a pretty crappy mathematical function, with zero homology to the brain. Nothing in the brain is piece-wise linear.
> Back propagation etc isn’t a property of a NN it’s a means to generate a NN.
You are the one claiming it isnt
Feed forward NN means a NN without loops. Back propagation is a method of generating a new NN that only works on a subset of NN.
I can generate a random number from a rolling a size sided dice once, just not any possible number.
I was taking you to be flip-flopping on backwards/forwards feeding; rather than to be talking about recurrent loops.
What is meant to be meaningfully a graph? What definition of "neural network" are you using?
The NN model itself, even recurrent, is just a regression function. In the sense that a Turing machine is just an implementation of a function from the Naturals to Naturals -- yes, a regression function, is likewise, a program.
The thing which is "Turing complete" cannot possibly be any particular regression function, as that is simply one program. So it has to be "the function of the function" -- which isnt a graph.
So again: what exactly is a graph here? What is the definition of NN you're using?
The NN family of algorithms are just means of interpolating through compressed feature spaces; ie., they are just kernel machines which remember compressions of their training data and compute distances for predictions.
That process, like essentially any function-generator, can without much work be made Turing complete. This, like being a "universal funciton approximiator" isnt an interesting property.
But either way, it still doest explain how any of this has anything to do with networks, brains, or anything like it.
I don't understand what issue you are referring to.
For a dense network, each pair of adjacent layers forms a complete bipartite graph. In other words, edges are all pairs with one node in layer N and another in layer N+1.
CNNs and RNNs take a little more work, but still easy to describe the graph structure.
But can you describe things like concat, max-pooling, attention etc. without changing the meaning of the edges? Or do you have to annotate edges to now mean "apply function here"? If so, op probably wants to say that you are describing more than a graph. There's a graph there, but you need more, you need elaborate descriptions of what edges do. In that case, op could be correct to say that technically, NN are not graphs.
Or, perhaps NN can generally be represented by vertices and edge lists. It certainly isn't the usual way to draw them, though.
Impressive. I think the first time I used Javascript I made a button.
Is it overfitting to say the same is true for humans, where a brain's starting weights and early experiences may make it much more difficult to achieve an accurate model?