Hacker News Reader

Comment by bko | Hacker News Reader

Am I wrong to think that most modern machine learning models are simply about sophisticated pattern recognition and statistical inference?

If so, is human intelligence doing something similar? Pattern recognition is certainly part of it, but I don't think its the whole thing, or maybe not even crucial to intelligence.

The remarkable thing about gpt-3 is its size, 175 billion parameters. At that point theres a lot of room to store a lot of memorized patterns. Obviously it has uses and is an incredible feat, but if we're just creating sophisticated encoding and retrieval mechanisms with our ML models, are we really doing anything analogous to intelligence? Wouldn't this have a limit on functionality? Or is this in a crude sense what's going on in our brains as well?

View on HN

This is a well-reasoned question, and something I think many people grapple with when they first dig deeper into machine learning. There are a couple things here:

1. Yes, most machine learning approaches are explicitly about sophisticated pattern recognition. To be clear, people working on machine learning have never been unclear about this. On this note, I'd recommend checking out some of the work done by Arthur Samuel, the man who coined the term "machine learning." https://en.wikipedia.org/wiki/Arthur_Samuel

2. Related to the above, there is a common conflation of the goals of machine learning research and our more sci-fi, AGI ambitions. The goal of a given model, typically, is to make predictions that score well on a given task. Generating passably human text, classifying images, whatever. I don't know of any recently released models whose stated purpose is to recreate complete human intelligence.

3. If we are going to judge machine learning efforts by their ability to recreate the exact mechanisms of human intelligence, we first need a much clearer image of what those mechanisms are.

> Am I wrong to think that most modern machine learning models are simply about sophisticated pattern recognition and statistical inference?

You aren't wrong. Modern Machine learning is entirely statistical inference, mostly through minimizing an objective function.

> Wouldn't this have a limit on functionality?

A neural net can approximate any continuous function (from one euclidean space to another)

https://en.wikipedia.org/wiki/Universal_approximation_theore...

>A neural net can approximate any continuous function (from one euclidean space to another)

Which doesn't answer the question by the way because Neural Nets suffer the same limitations of Turing Machines, that is to say even a neural net can't solve the halting problem.

The fact that something is a universal function approximator should not be mistaken as equivalent to "isn't limited in functionality".

> that is to say even a neural net can't solve the halting problem.

Can anything?

well humans can certainly deal with a lot of halting problems, because humans can make decisions that aren't obviously computational or algorithmic in nature.

Which is actually the relevant question that people seem to have skipped over completely, at least from living organisms it appears fairly likely that intelligence isn't just computational, or at least not simply computational in the way turing machines are.

> well humans can certainly deal with a lot of halting problems

Could you give an example of a halting problem that humans can solve?

> it appears fairly likely that intelligence isn't just computational, or at least not simply computational in the way turing machines are.

Do you have supporting evidence?

> Modern Machine learning is entirely statistical inference

In virtually no cases that I can think of does modern machine learning approach statistical inference.

Statistical inference is always concerned with estimating confidence in beliefs and typically more concerned with the confidence in the model parameters rather than the model predictions.

Machine learning is almost exclusively concern with model predictions and the performance of these predictions.

You can easily have a model that is very useful for statistical inference but would be awful for machine learning, and most machine learning models (especially neural networks) as useless for statistical inference but great at prediction.

This is not a critique of machine learning or vice versa, its just the case that these two approaches to modeling are quite different and used to solve very different problems.

> neural net can approximate any continuous function

Technically a lot of Neural networks are universal function approximators to the limit of infinite data and infinite iterations.

It's like saying that with infinite time and motivation, I would be the smartest man on earth. Technically true, but an entirely useless statement.

> A neural net can approximate any continuous function.

Can't a RF or boosted regression tree theoretically do the same and then some (approximate discontinuous functions)?

Our brains have lots of highly specific bits of hardware that corresponds to different parts of the body, or different tasks such as recognizing faces. Most of what our brains do isn't necessary to replicate to produce something which is, at least in some way, generally intelligent. Artificial neural networks are only trying to replicate a small subset of our brain's hardware, and relatively basic (albeit large) architectures like GPT-3 are doing this by becoming skilled at recognizing patterns, and extending them. Pattern recognition is an enormous part of our higher level (generalizable) thinking. It also means that things like our brains and GPT-3 are extremely efficient but lossy compression algorithms.

It's a decent proxy for how our brains process language, but processing language is just one of many processes involved in having a conversation. The goal isn't intelligence. The best possible model would be able to identify the relationships between all words in all contexts. Which is cool, but isn't the whole picture.

The best possible model would also be able to identify the relationships between all the people involved in all contexts.

The philosophical question is whether there's more to semantics than a certain set of standard paths through a large set of textual relationships.

The answer is "Almost certainly", but not necessarily in all contexts - which means that in fact mechanical representations are already good enough for some contexts, although still with a relatively poor hit rate.

I don't think that you're wrong in thinking that. That is how I view these systems. Artificial neurons and networks of them pale when compared to the complexity of any biological neural network. At a basic level, ANNs are a simplification of biological neurons but I think that we lose something crucial when we perform this simplification. So I think that there is deffinetly more to intelligence than what ML/Deep Learning is doing.

Freeman Dyson had an interesting lecture hypothesizing that at least part of why we haven’t been able to make anything like the human mind with our current computers is that our brains are analog.

See here: https://m.youtube.com/watch?v=JLT6omWrvIw

The Chinese Room (https://en.wikipedia.org/wiki/Chinese_room) would agree it's the same

Reply on news.ycombinator.com