What will the major ML research trends be in the 2020s? | Hacker News Reader

What will the major ML research trends be in the 2020s?

old.reddit.com

29 pointssebg12 comments

I think that 3d reconstruction and related computer vision is going to take over vision applications that have been so far handled by 2d methods.

There will be more emphasis on well-factored understanding.

That 3d understanding will be leveraged by question answering systems with a type of visualization of scenarios. Abstractions will be built on this grounded foundation.

Imagine full 3d understanding of a video and association with captions. Now say you train it on 1 million videos.

Combining well-factored and fully understood spatial/visual information with text for training along with recent techniques like transformers will finally demonstrate true natural language comprehension for a variety of domains.

Look at Hinton's recent paper on using transformers with capsules etc.

'Expert Systems' and 'Knowledge Based Systems' make a comeback, that's right, get your punch cards and rev up that lisp machine. We're going back to a time where some of these algorithms were more maintainable and somewhat transparent. We'll wrap em up in modern veneer a la 'random forests', and 'decision trees'. Then maybe we can actually make some real businesses where we take control of these algorithms rather than just running them and hoping for the best.

Meanwhile the big kids with their large datasets will force their way through to even greater levels of achievement that border on magic with things like image recognition, voice recognition, etc.

Commoditization of deep learning models and maturity of architectures that remove the need for expert tuning. NLP will see the most progress, with things like question answering becoming more automated. No true conversational ability, as the model can’t “reason”, but can trick you for a bit, till you realize the facts it spits out don’t work together.

Information retrieval and structuring will get really good.

I sincerely hope this decade will be when ML algorithms will finally hit human level intelligence level (a la Ex Machina)

In terms of foundational research:

There will be a bunch of work connecting type theory (algebra) to deep learning (geometry), by using the ideas from HoTT.

Quantum AI will become increasingly a thing.

I love this direction. I was studying homotopy a couple years back with this intention.

Embedding functional relationships is the future

Assuming this is neither sarcasm nor GPT3, WTF does it even mean?

For the first:

Michael Shulman et al have been working on a semantic model of type theory as category diagrams and discuss how sometimes it’s easier to solve your problem as a diagram picture and sometimes as a logical statement.

The mathematics they used for this “foundations” work in abstract math connects hypergraphs/CW complexes decorated with data to statements in a type theory, like a program.

The goal is to use this deep mathematical connection to find a type theory “close to” our AI, allowing it to make “intuitive” extensions of a statement via DNN, but validate that it makes sense within its “effective language” deductively.

My research in that area is automating proofs as a “basic” usage of that idea: an AI turns the desired statement into a (partial) picture it tries to use image completion on, to make it look like other proofs it’s seen, but the discriminator for that generative step can use the formal model, after turning the proposed image into a statement — is it logically consistent or not?

There’s a bunch of problems in that area, which has only started to be explored the past few years with the DL is manifolds and covering ML papers.

tl;dr: That math work gives us rules for how to turn geometric objects defined one way (tensors on a space) into objects defined another (formulae on varieties). Algebraic geometry meets AI.

For the second:

The Microsoft QuArC group under Dr Krysta Svore is working on developing ways to accelerate common industry algorithms via quantum computers — just calling out AI as a particularly impactful application since quantum computers will come online in 10-20 years.

They have some AI papers if you poke around — mostly that “best fit” for data is a good analog for “lowest energy”, and ways to use that for gains.

https://www.microsoft.com/en-us/research/people/ksvore/

What’s HoTT?

Homotopy type theory (https://en.wikipedia.org/wiki/Homotopy_type_theory). Some people think it's the new mathematical theory of everything. I'm more skeptical.

My perspective is that it’s a useful idea for abstraction — which combined with category theory’s ability for composition is cool beans, at least in computer science.

The square in a category diagram is a legit object — and we can have type theories with the same features.

That’s useful for computational equivalence (eg, spec matches implementation) and path bundling to prevent case explosion.

About what you’d expect from adding homotopies to type theory.

In the context of ML application to AI, the trends can be categorized into two groups:

1) Fake AI research:

- Continue falling into the deep learning rabbit hole, overfitting even more benchmarks and producing arbitrarily large and dumb models

- Pursue the infinite long tail in self-driving cars, question answering or anything that requires common sense reasoning.

- Get busy with endless permutations of layers and tweaking of hyperparameters, throw even more things on the wall to see what sticks.

In this scenario AI will always be "5 years away" as this approach is akin to alchemy and search of the philosopher's stone.

2) Actual AI research:

- Developing mathematical principles of natural cognition, biologically inspired memory & learning

In this case we might actually have a chance for toddler level AGI sometime in 2030s.

Reply on news.ycombinator.com