w1 ← w1 − η ∂w|∂L
η is a constant called the learning rate that controls how fast we train. All we’re doing is subtracting η ∂w|∂L from w1:
- If ∂L | ∂w1 is positive, w1 will decrease, which makes L decrease
- If ∂L | ∂w1 is positive, w1 will increase, which makes L decrease"
Simple indeed.
Andrew Ng’s course on machine learning from Stanford (on Coursera, which you should be able to audit for free) gives great intuitive explanations of these topics, if you’re curious enough to invest a couple solid weekends to get the foundations. The foundations alone will get you very far indeed.
I took it in 2011 (ML Class) before Coursera existed, and it finally opened my eyes on not only what and how backprop worked, but also how everything in a NN could be represented and calculated using vectors and matrices (linear algebra), and how that process was "parallelizable".
The course uses Octave as it's programming environment, which is essentially an open-source and (mostly) compatible implementation of Matlab.
My first thought was "Finally! A use case for a home Beowulf cluster that is somewhat practical!"
It really opened my eyes and mind to a number concepts that I had looked into before, but couldn't quite wrap my brain around completely.
In case you were being sarcastic, it is simple if you don't take it out of context. It requires a certain understanding of derivatives, but a reader who doesn't understand them won't make it to the point you're quoting.