Comment by anotheryou | Hacker News Reader

Comment by anotheryou | Hacker News Reader

Parent Full thread

tl;dr: Children learn unsupervised and without back propagation.

So machines, for now, use two tricks to get a bit better in maybe more narrow fields. There is NNs without back propagation and unsupervised too though, they are just currently not the best, right?

Doesn't sound fundamentally different to me.

And about this paragraph:

"humans learn by making structured mental concepts, in which many different properties and associations are linked together. For instance, our concept of “banana” includes its shape, the colour yellow, knowledge of it being a fruit, how to hold it, and so forth.

As far as we know, AI systems do not form conceptual knowledge like this. They rely entirely on extracting complex statistical associations from their training data, and then applying these to similar contexts. "

That's plain wrong, isn't it? I'm no expert, but isn't latent space and layers basically these "structured mental concepts"? You have upper layers encoding for partly features, lower layers checking on many of them. That's abstract concepts. And you have the multidimensional statistics of the latent space that will see bananas as the intersect of yellow color and a long shape.

For an opposite "opinion": the layers of a NN map quite nicely on to what happens in the brain: https://twitter.com/JeanRemiKing/status/1533720262344073218 (they check which neurons get activated first, which later in NN and brain)

Calling children learning (or any biological system learning) unsupervised is kind of insane.

Kid touches hot stove. Gets feedback. Kid tries walking. Gets feedback. Kid talks to someone. Gets feedback.

I have hard time even coming up with an example of a proper unsupervised learning in the wild. Hermit phenomenologist, maybe? You have no way of doing anything social in an unsupervised way and most of the language happens socially.

EDIT: I did misremember RL as supervised, see chain below.

EDIT2: Hey, I never said supervised in the first place! Enough internet for me today, I need a nap.

Unsupervised doesn't mean "no feedback", it means classifying things based on feedback/experience without being explicitly told how to classify things.

When a kid touches a hot stove, they classify it as bad/dangerous through feedback - unsupervised learning. Not because a parent comes by later and says "Hey! Look, your hand is burnt! That's bad because I say it is!" - supervised learning.

You can even see supervised learning in kids to some extent: when they play barber shop and get a butchered hair cut, you often see kids be perfectly fine and happy even with the outcome, through lack of supervision they don't yet have any negative experience from doing this. Then a supervisor comes along (parent) who freaks out and the kid learns through supervision "oh no! I did something bad because mom said so!"

Pure supervised learning is straight forward to point out in our society I think, schools and all.

Let me start by saying, I misremembered and believed RL is considered supervised. Since it's actually not, that dramatically changes my position ^^

I'd argue then that most learning in humans (how strong analogy holds aside) is RL. The world is structured so that you learn to avoid pain - those who don't tend to not leave a lot of offspring behind.

Arguably that's a better metaphor for all learning than supervised/unsupervised: there is some further reward/punishment for even the school work. Misuse of muscles when learning to walk is also connected to similar feedback, so is misuse of language - be it grammatical failure or being a bad salesperson.

Generally I'd focus on pointing out that children do not learn in an unsupervised fashion in almost any setting, claiming it was supervised was wrong and based on my misclassification of RL.

As you say, there are straightforward examples of supervised learning as well. "This is a cup. This is another cup." What's different from the machine vision case is you don't need to stick the kid in a Skinner box and show them 100,000 examples of cups in different shapes, sizes, orientation, lighting, etc. They'll generalize (sometimes incorrectly) a lot faster than that.

The comparisons are nothing short of insane. The best models and datasets come well short of justifying even an infinitesimal representation of intelligence.

It’s quite concerning too that there seems to be this belief that one can be effective in this field with a weak math background. I recall a meeting with a consultant claiming that they could improve order flow by using ML with a data set that was small and inaccurate. Their background was an mba and a few years of working on mergers and acquisitions.

As humans we are adverse to complexity and naturally lazy, our goal is always to find easier ways to do things that benefit us more. It's why languages like Python have taken off.

A large portion of the population is being told that programming is where you go to make a ton of money, and then subsequently hear that machine learning is the future of programming where you don't need to understand your data, you don't need to understand anything. You just plug in some magic codes and get out great results because the computer learns what you want!

Of course, it's not like that. But we're going to see a lot more of this attitude, because Machine Learning is not pitched as "high level statistics" but rather "magic that does what you want!"

Social interaction could be called unsupervised, we'd just need to adopt a little chat bot :)

As for the physical parts: isn't that a loss function? Reward/dopamine if you reach your goal (walking), pain if you do what the rules/the body does not allow.

I'm sorry if I'm mixing terms a bit, as said, I'm not an expert and a bit rusty about what I once knew :)

Any kind of reading, listening or observation without interaction (action and feedback) is unsupervised learning.

Knowledge (very widely understood, including more nuanced understanding of words, internal simulation of future actions etc.) would have to never come in contact with the external world. I guess you could argue that at the point of observation it is unsupervised, since we're kind of assuming whole person to be a single object here.

Something that occurs to me is that Humans are a lot more "pretrained" than we think. And almost all of the heavy lifting comes from evolutionary learning. So it makes more sense to compare children learning to fine tuning, not pre training.

That's a super interesting thought, thanks!

There is quite few animals that can do a lot instinctively, right? Like a turtle hatching finds water, swims, eats.

Also reminds me of the experiment with rats that genetically transferred fear for a specific smell. ( https://www.newscientist.com/article/dn24677-fear-of-a-smell... )

> structured mental concepts

What you are referring to is an internal representation of the compositional nature of the input; the NN will holds a structure of patterns that represent the idea.

But the "structured mental concepts" in question are entities interesting for their logical manipulation upon reflection. Whether some NN definition (configuration) is apt for the latter should be seen in view to the latter.

> tl;dr: Children learn unsupervised and without back propagation.

Well, this has been talked about for at least 10 years. Just take a look at any talk/FAQ by Hinton and/or LeCun from 2012 and onwards.

ohh it goes back much further than that!

Reply on news.ycombinator.com