NanoNeuron – simple JavaScript functions that explain how machines learn

Thanks! Glad that example was useful!

Sure. Done.

Vote up. I think it'd be a great primer for more complex concepts.

It is possible. But don't have strict plans yet.

Thanks!

I believe OP's mother tongue is not english. So, I think it is fine

You can always open a pull request.

Sadly, no. This OP's writeup is just about the simplest exposition that doesn't miss the point, and it misses the major transition when you go to layers with nonlinearities between them.

There is no royal road to mathematics.

What this particular model shows is probably about as simple as you can get; the activation function (what is called "predict" in the code) is a simple linear model (very similar to RELU - which has other important properties valuable for multiple layer models vs other activation functions). You probably couldn't make it more simple unless you made it a binary function (ie - set it to "1" if the value is anything over zero, otherwise set it to "0").

Everything else basically stays the same, though, and it can't get much simpler.

Here's the thing though: As far as anyone knows - and research is ongoing on this, so this may change at any time - heck, it may be different now than the last time I looked into it - but anyhow, as far as anyone knows, in nature (that is, actual neurons in a brain), backpropagation is not done, nor is there an apparent mechanism to allow for it.

This has led Geoffrey Hinton to think maybe things should be rethought (and he has gone down the road of something called "capsule networks" which I have yet to understand - I can't tell if they are actually simpler or more complex from what I have read):

https://medium.com/intuitionmachine/the-deeply-suspicious-na...

This doesn't mean that backpropagation doesn't exist in nature, just that we haven't found a mechanism for it (although there are some competing theories).

I have also read research papers on other ways of making a neural network learn - one was something akin to, instead of backprop, randomly picking neurons in the layers and re-assigning them with random weights - and amazingly, the network would still learn.

All I can really say is to dig in and think about it, try to learn all you can about both artificial neurons and network implementations, along with the history of the field.

For instance, many don't know that Alan Turing played around on paper with neural networks a few years after the Mcculloch-Pitts model was released; I tend to wonder if there is anything in his notes that is worthwhile or has been overlooked - his network models, from what I recall, were not structured, and I believe his neurons were simple binary functions - likely, nothing is there, as people had to have studied this work, as obscure as it is.

Also study and learn what we know about actual neurons and the networks they form in "brains" (from simple things like C.Elegans to humans); some researchers believe that so-called "spiking neural networks" more closely model actual neurons than the commonly used style of mathematical model of the sort shown by this tutorial/github.

And keep in mind that all of this is "models" - not the real thing, and shouldn't be expected to behave the same as the real thing. But - maybe something simpler can be done that comes close - much like an airplane is a simpler machine than a bird, and looks and works nothing like a bird. Perhaps there is something simpler for a model.

But then again, nature seems to have come across a better solution than what we currently use - after all, our brains use no where near the amount of energy a GPU uses - and that GPU is used to simulate much simpler models than what the brain is. So what is the secret?

Nobody as of yet knows - but the first to crack it may become a very wealthy person (that, or depending on what happens, they may inaugurate the technological singularity with an advanced GAI that runs roughshod over mankind, viewing us as inconsequential as we do ants and insects on a construction site today - and as such, that person would become the most reviled human ever, for as long as the human race remains in the face of the changes)...