Neural networks as non-leaky mathematical abstraction
blog.cerebralab.com
blog.cerebralab.com
It very much is not. If you tried to learn calculus the way Newton learned it (with fluxions, or worse, awkwardly stated in the style of Euclid’s elements) or the way Leibniz learned it (ill-defined infinitesimals), you’re gonna be in for a bad time. We synthesise, reformulate, and reteach all the time. There’s a lot more that has changed since Euclid’s time than merely the notation.
I skimmed the rest of the article, but the philosophical mathematical preface is questionable.
If you are dealing with polynomials/algebraic geometry only then there is a very simple, totally rigorous approach using nilpotents. And there is a newer approach called "alpha calculus" that I know nothing about but here is an example: https://arxiv.org/pdf/0807.3477.pdf
The reasoning for author's premise that sigma summation and integrals are leaky abstractions is unclear.
The two are their own independent abstractions; while perhaps complicated to fully understand, they will work exactly as designed and, provably so, from a logical standpoint. If they were leaky, they would display some kind of shortcoming that didn't allow them to convey parts of the core idea they were created to represent.
On the other hand, an NN implementation of any sufficiently complex concept is almost guaranteed to be leaky as it is too complicated to be provably correct, and will have too great a test surface to verify exhaustively. There are likely to exist edge cases where the NN fails outright that will be never discovered until used in a specific scenario. That seems leaky.
Can someone set me straight on what the author was trying to convey?
Yep, someone definitely needs to take a course in Abstract Algebra.
This is absolutely how I was first taught integral calculus.
∫x^n = x^{n+1}/(n+1) + C
∫e^x = e^x + C
∫(f+g) = ∫f + ∫g
I used to think this was a bad thing, but now I’m not so sure. The word “calculus” does mean a system for calculation.
It actually means that the abstraction is not actually abstracting the concept it is representing, and thus requires you to understand lower level concepts in order to grasp it.
An example would be that the C programming language is a leaky abstraction because sometimes you need to understand and write Assembly code.
That doesn't make C bad or incorrect, it just means that at some point, you'll need to break through the abstraction and learn what it was trying to simplify.
Also, how does this definition make the Fourier transform not leaky? From the article:
> A leaky abstraction is the rule rather than the norm in mathematics. The only popular counterexample that comes to mind are integral transform (think Laplace and Fourier Transforms).
Yes you can pull Tensorflow and start creating layers without much knowledge in Algebra, just like you can use cryptography software without a PhD in Number theory, or apply Photoshop filters without understanding convolution.
But this is only possible because _other_ people do have that knowledge and distilled it down to a form you can use from the get go.
For example, positional notation decreases the size of natural number representations exponentially compared to straightforward application of Peano's axioms or the like, but the numbers themselves remain exactly the same.
> an autoencoder (AE) is basically one of the easiest ones me to explain, implement and understand
Then a list describing it. But that list does not constitute an explanation unless you have a pretty good intuitive understanding of linear algebra and function composition.
> So essentially, neural networks become a sort of mathematical abstraction that isn’t very leaky.
I think what the author really means by "leaky" is "not opaque." Mathematical structures that you can analyze in a well defined way (not opaque/"leaky") like integrals vs mathematical structures that the tools to dig into in a deep way don't exist yet (opaque/"not leaky").
Except that such understanding of neural nets is a major current research program.
So to all the youngsters who might be looking at this, this article is the ravings of a crank who doesn't know what he's talking about.
What is this supposed to mean? I honestly don't understand. They most certainly realized that although tedious to compute it is just as well-defined as 2*3=6. What is he implying here?
When I prove, say, Cauchy's theorem about groups, it doesn't matter which concrete group I'm operating on to prove it. If I write a sigma to sum some vectors, it doesn't matter if the vectors are coordinates in 3D space, functions defined on an interval, or just numbers. Mathematical abstractions are pretty much the only ones that don't leak.
What, that you have to understand summation to understand what the sigma refers to? Well, duh. That doesn't make the sigma a leaky abstraction, because the sigma is not an abstraction at all! Not even a trivial one. The sigma is notation. And it's hardly news that you have to understand the concepts behind a notation to understand notation itself.
The author seems to misunderstand what an abstraction actually is.
All signs refer directly to abstractions in the mind that can be compiled into a variety of useful forms. Yours does not take precedent in this case, as it is plain to see that sigma is under specified due to the huge range of concepts it’s practical usage covers. Your attempt to prescribe that which should be prescribed is nothing but authoritarianism.
This, I think, is the main problem with the article.
I also found the opening line curious:
> > I find it rather weird that mathematics is usually taught to people on a historical basis.
I think this is much more common in philosophy than in maths.
you're wrong. every "weird" technical hypothesis in a theorem is a leak. even in your super simple example there's a caveat: the sum has to be finite (that mean no you can't perform any sum of any set of vectors).
edit: i got downvoted so here's a better example that i just ran across: manifolds with boundary can have empty boundaries. where does this come up? exact 1-forms on S^1 integrate to zero. why? because by Stokes. how? S^1 has a boundary but it's empty.
You can use SVMs without fully understanding the specific QP optimization algorithm underneath, you can understand the kernel trick without all RKHS math etc.
Or in classical vision, you can use keypoint extraction without caring about the details.
Or in linear algebra you can understand what SVD is and does without knowing the detailed steps of computing it.
I don't see neural nets as very special in this.
Try explaining the intrinsic meaning of written language in written language. You can't close the loop without making the result trivial.