Hacker News Reader

Comment by workingon | Hacker News Reader

This conversation seems sort of unnecessary to me as a researcher who uses AI. Symbols and DL are not exclusive, just not yet thoroughly studied, I guess this is more of an argument about the semantic positioning of a few people in the field who think they are important. Some of the best current research in the DL space involves defining and searching for governing equations with symbolic matching.

View on HN

Also, there’s a deep relationship between DL and symbolic reasoning:

The tensor networks in DL end up looking really similar to tensor representations of the diagrams equivalent to a type theory — down to convolutions being a way to “type” data in an input.

We’re just now exploring that, but this may be another case of “algebra-geometry equivalence” with DL giving us a differential/geometric interpretation and symbolic reasoning giving us an algebraic interpretation.

Algebra geometry view makes sense to me. Considering ML as a learning to approximate scheme algorithm. Tensor representation is similar to the SDP trick achieving the optimal max-sat approximation. The difference is that DL approximates from inside of the high dimensional space (concave) while SDP approximates from outside (convex). The later one turns into polynomial algorithm, but the former one remains NP-hard. The success of DL just proved that there is a long way to go for P equals NP. Whenever we figure that out, symbolic approach and Tensor approach will merge.

I tend to think of it in terms of topology — since that’s my background.

The semantics of a system is mapping the topology of the input space to output space.

DL expresses that relationship geometrically; symbolic reasoning expresses that relationship algebraically. For every geometric expression of semantics, there is some corresponding algebraic one — which we can view as the “internal language” of the DNN.

Whoa! Would you happen to have a link to some material that fleshes this intuition out with some examples?

Unfortunately, I don’t know of much — the convolution thing is from a whitepaper I’m currently working on, the third in a series.

The first two look at implementing shapes as diagrams as digital images:

https://www.zmgsabstract.com/whitepapers/shapes-as-digital-i...

https://www.zmgsabstract.com/whitepapers/shapes-have-operati...

You can get a sense of the convolution idea from thinking about how you’d detect the encoded square is an interval of intervals, via detecting a pattern along the diagonal and the connective blocks.

I also have a few notes on connecting the concept to Curry-Howard:

https://zmichaelgehlke.com/journals/2021-06-14-curry-howard-...

And some (messy) notes about general research direction:

https://zmichaelgehlke.com/journals/2021-01-30-intro-to-effe...

The idea of connecting a geometric and algebraic representation is based on work by Michael Shulman — and the internal languages of toposes. And work on the ML side such as covering based models. (Having trouble finding references on my phone; sorry.)

Reply on news.ycombinator.com