Hacker News Reader

Comment by dmreedy | Hacker News Reader

I think some of the most exciting and interesting work comes out of proving, not just capabilities, but constraints for systems, be it Gödel, Shannon, Aaronson, or any of the others in the smaller-than-desirable tradition of those who say, "No". I think a better understanding what Deep Learning can't do (well) is fertile material for better understanding the kinds of problems it can do, and am very excited to see more work in this space, and movement towards an underlying structural theory.

View on HN

The paradox of the heap[1] shows that vagueness is an inherent, inescapable part of reality.

Machine learning would provide a way around that paradox: a heap is what a neural network says is a heap!

If this worked, it would be too good to be true. Thereore, it cannot work.

[1] https://en.wikipedia.org/wiki/Sorites_paradox

> The paradox of the heap[1] shows that vagueness is an inherent, inescapable part of reality.

Or it's an inescapable part of how we interface with reality, i.e. it's an artifact of the structure of the human brain.

Bak of the Bak-Tang-Wiesenfeld model called his model the sandpile, basically explicitly to remind one of the Sorites paradox. Interesting thing to talk about. You can definitely take it as a sort of different descendent of the Ising model: it has had much less influence on the world compared to backpropagation neural net.

The dynamical nature of the definition stays, which has bedraggled Sorites people for millenia, but McClelland has been talking about the dynamical nature of representation for decades too. I think some of the anti-neural net cognitive science people (Pinker, Fodor maybe? I forget) brought up some Sorites arguments when fighting the 80's connectionists.

I think the issue with Deep Learning is that it is like a hyperdimensional optimization heuristics sequence surpassing what human mind can comprehend and pushing limits of computing (depth of 100 layers max at the moment). Given how difficult are far more trivial optimization techniques and proving bounds for them, it seems the times where we could just define a new approach and prove some nice properties are behind us :-(

I don't think the problem here is what the human mind can comprehend because identifying where these methods break down is actually pretty easy from a math perspective, and the breakdown doesn't change as you go up and down with the number of dimensions. What has always surprised me about ML and DL is how far we can stretch simple regression techniques and how useful it is on such a variety of problems.

I agree that it has turned into an NP problem in a lot of cases now but the concepts behind the problem of N-dimensional optimization are pretty well understood.

A lot of useful analogies for N-dimensional space can be abstracted from the simple transition from 2 to 3 dimensions.

"To deal with a 14-dimensional space, visualize a 3-D space and say 'fourteen' to yourself very loudly." -- Geoff Hinton

A math student and an engineering student went together to a presentation called "14-dimensional space topology". As the presentation progressed, engineering student became more and more frustrated as she could only grasp a few simple initial examples but the deeper the presentation went, the worse was the understanding. Yet the math student was absolutely enthusiastic, often asking the presenter various complicated questions and looking he enjoyed himself a lot. With a massive headache in the end, the engineering student turned to math student and asked: "How could you understand the presentation? I was able to understand 2D, 3D but once we increased number of dimensions I got lost and never understood anything about 14D!" The math student looked at her with a confident careless look and said: "It was simple. I imagined everything in N dimensions, and then just reduced it to 14".

"and then let N=14" is how I've heard it.

It's not like we don't know anything about NP complete problems, neither. Critical phase transition in transition of alpha on random kSAT and other stuff, the realization that this is neither necessary or sufficient for NP completeness just a really common complementary phenomenon, etc etc

If you use 40M+ optimization variables and a really really bad optimization technique (gradients), which is however fast, as is the case in DL, the amount of nice practical things (e.g. limits) you might be able to say could be very low.

Yes, we all were stunned when somebody pulled some awesome trick and found a better upper/lower bound on something, yet sometimes it's better to be realistic - maybe once we transfer to quantum computers, we can test more limits practically when utilizing parallel power of QP.

I strongly dissagree. Broken down, Neural Networks are surprisingly simpel and elegant. The combination makes them so powerful. Ideally, an elegant theory would abstract over these to provide bounds for arbitrary combinations, or groups of combinations.

Why can't we pull out the statistical mechanics? You can't understand the 10^27 variables in a lump of coal, either, but you can still do thermodynamics. People already do it, Ganguli, some stuff from Bengio

I wish I were so optimistic - if you use statistics, then you might remove most of interesting outliers and study just those more stable prevailing characteristics and not the problematic ones, like when using Hooke's law for designing buildings while ignoring its limited scope. And physics hits hard math limits while studying dynamic systems (i.e. fractals) anyway.

We can always use Ramsey Theory stating there is no chaos possible and you can always find some regularity in anything, yet if you need to scan 10^132 items to find the order it becomes quickly uninteresting.

RNN is natively and obviously dynamical but FFNN can also be seen as transient dynamical system.

Little Ramsey theory ever used in neural network land, but theory of dynamical systems used from quite a long time ago, from Werbos-BPTT which was explicitly envisioned as a dynamical algorithm to the description of the dynamical instability of gradients in RNN.

The statistics of statistical mechanics don't necessarily need to be CLTish sorts of things, you know?

Possibly ;-) I'll study these things in detail soon (hopefully), so far just practical experience with all funny things from DL like self-driving cars, composing music with RNNs etc. but having extensive operations research exp in the past I tend to be careful.

The high dimensionality is merely for the convenience of optimization, so that we get smoother manifold on which a good approximation solution can be found. In that regard, it is interesting to think about of space of models that we humans can interpret, and the possibility of "distill" deep neural nets into models in that space.

Well, they already had a big instance of that, in the Minsky Papert book "Perceptrons", talking about linear separability. Talking about backpropagation (as opposed to delta rule) in opposition to that is a sort of mushing of history but it's interesting to think about

Reply on news.ycombinator.com