And yet there are basic problems with our method of representation. If you take error surfaces and project them into 3 dimensions ... they look smooth. And then you look at the polynomials (because vector multiplication yields polynomials) that we use to approximate those polynomials ... they look anything but smooth. They look like those scenes you got in Doom when you go out of bounds in a level. Very,
very different from nature.
It's a miracle it works at all. Making ML models is like making smooth and comfortable sofas by arranging spiked rocks. Now by arranging the right rock juuuuuust right you can get any shape, and indeed you can, but the shapes we're trying to make would be a hell of a lot easier to make using pillows. It's a miracle it ever succeeds at all, and the problem definitely is the basic building blocks.
Polynomials are famous for 2 reasons:
1) they're "spikey". When making more accurate matches what you're doing is adding every "higher frequency components" that do their own spiking. They fix something at one place, and screw lots of stuff up everywhere else
2) at their ends they almost always go off into infinities. Technically zeroes are possible, but you just never see that.
And Neural Networks have challenges that humans don't seem to have:
1) they have sudden, very "weird" ideas. All koalas are koala's, except if you modify pixel 381 down 10%, then it's an elephant.
2) they have ridiculous predictions outside of their training range. You might say "of course they do", but humans don't, and neither do animals ... what does a reasonable human/animal do when confronted with an impossible situation? They respond the same way they respond to the closest reasonable situation they know. A human seeing a tidal wave come for him runs away from the wave. That may be stupid, if they've got little chance to outrun it, but a neural network would just stand there, start shaking, and drop uncontrolled to the floor.
These problems seem related to using polynomials (as opposed to, say, sum-of-10-gaussians) for prediction. Those error surfaces we started with ... they don't quite look like 10 gaussians either ... but they kind of look a lot closer to those gaussians than to polynomials. They look like smooth, slowly sloping curves.