Hacker News Reader

Comment by FlyingAvatar | Hacker News Reader

This entire concept seems backwards to me.

The reasoning for author's premise that sigma summation and integrals are leaky abstractions is unclear.

The two are their own independent abstractions; while perhaps complicated to fully understand, they will work exactly as designed and, provably so, from a logical standpoint. If they were leaky, they would display some kind of shortcoming that didn't allow them to convey parts of the core idea they were created to represent.

On the other hand, an NN implementation of any sufficiently complex concept is almost guaranteed to be leaky as it is too complicated to be provably correct, and will have too great a test surface to verify exhaustively. There are likely to exist edge cases where the NN fails outright that will be never discovered until used in a specific scenario. That seems leaky.

Can someone set me straight on what the author was trying to convey?

View on HN

The neural network counterpoint feels odd, but with respect to summations and integrals as leaky abstractions I think the author's point is that deploying them effectively as tools typically depends on a mountain of hierarchical knowledge on top of which elementary calculus is built. I don't know that this is necessarily a requirement -- one could conceive of a world where integrals are taught purely in terms of methods for moving in and out of that concept domain (analogous to the train/predict interface common in ML), and where people are simply taught algebraic rules for manipulating integrals entirely in the realm of integrals without relying on their pedagogical grounding in limits (sort of like how we have high-level descriptions of layers, drop-out, and other concepts that are at the same level of abstraction as neural networks themselves). I think the author claims that this latter approach is rare in mathematics, and it seems they make the stronger claim that mathematical concepts aren't generally amenable to that kind of strategy with integrals and summations as specific examples of where it's impossible. FWIW, I disagree with that assertion, but nevertheless it seems to be what the author is trying to say.

> FWIW, I disagree with that assertion.

Yep, someone definitely needs to take a course in Abstract Algebra.

one could conceive of a world where integrals are taught purely in terms of ... algebraic rules for manipulating integrals entirely in the realm of integrals without relying on their pedagogical grounding in limits

This is absolutely how I was first taught integral calculus.

  ∫x^n = x^{n+1}/(n+1) + C
  ∫e^x = e^x + C 
  ∫(f+g) = ∫f + ∫g

etc. Only later did I really learn the details of limits and the rigorous epsilon-delta underpinnings in real analysis.

I used to think this was a bad thing, but now I’m not so sure. The word “calculus” does mean a system for calculation.

Leakiness is not referring to "correctness" or any sort of practical shortcoming.

It actually means that the abstraction is not actually abstracting the concept it is representing, and thus requires you to understand lower level concepts in order to grasp it.

An example would be that the C programming language is a leaky abstraction because sometimes you need to understand and write Assembly code.

That doesn't make C bad or incorrect, it just means that at some point, you'll need to break through the abstraction and learn what it was trying to simplify.

How does this definition make the Neural Networks not leaky? They sometimes work, they sometimes don't work, if you tweak a few parameters (the number of layers, the number of neurons in each layer, ...) perhaps they work or perhaps no.

Also, how does this definition make the Fourier transform not leaky? From the article:

> A leaky abstraction is the rule rather than the norm in mathematics. The only popular counterexample that comes to mind are integral transform (think Laplace and Fourier Transforms).

I don't particularly agree with the author, as I think they are mixing up the mathematical theory and the computational part of it.

Yes you can pull Tensorflow and start creating layers without much knowledge in Algebra, just like you can use cryptography software without a PhD in Number theory, or apply Photoshop filters without understanding convolution.

But this is only possible because _other_ people do have that knowledge and distilled it down to a form you can use from the get go.

These mathematical "abstractions" are not the same as "abstractions" in natural sciences: they are only a succinct notation that provides leverage to perform more reasoning with less writing, not a simplified and separate model of a system.

For example, positional notation decreases the size of natural number representations exponentially compared to straightforward application of Peano's axioms or the like, but the numbers themselves remain exactly the same.

Arguably the neural net is the leakiest of all abstractions. Inputting enough data, preventing overfitting. Not having a human readable implementation because of the computer figured it out with brute force. Those are all complex details leaking out of the abstraction. Machine learning has its uses in processing large amounts of data, but it is the opposite of a designed algorithm with self-contained abstractions.

Reply on news.ycombinator.com