Hacker News Reader

Comment by akasakahakada | Hacker News Reader

Linguists e.g. Chomsky: No you can't. Proof is left as an exercise.

I think Chomsky's position is that _humans_ are primed for language, but it doesn't mean there exists no system that can learn language from 0.

Also, the ability to understand language is a spectrum, not a binary quality.

Yeah the argument is that there is a poverty of stimulus (not enough training data) in the case of children. The same doesn’t apply to ML models which need an abundance of stimulus.

There was some linguist (Deb Roy?) who videotaped his children growing up. One of his observations was that every learned syllable etc. was tied to observing it as a stimulus and trying to imitate it. Now it is true, children are pretty good learners, they can often learn something the first time they see it, but actually I have seen some LLM stuff about "instant learning" - e.g. the training is only done with one pass over the material. https://www.fast.ai/posts/2023-09-04-learning-jumps/

This is an interesting connection. If I had to guess, the Chomskyist response would be to say, yes but this only applies to LLMs that have been pre-trained already (i.e. have structures in place needed to understand language in general). I think a Chomskyist would say that language learning is precisely like fine-tuning and not the pre-training of foundation models.

And with that in mind there is an obvious counterargument to the original "No". If humans learn language through observation and innate knowledge encoded into the brain then we can construct an artificial system that discovers language purely through observation by encoding the same innate knowledge into the system.

Algorithms necessarily contain some knowledge about the data they are working with so that isn't cheating. Neural nets for example have to have some architecture and that predisposes them to learning certain patterns.

He does claim there is no system which can learn language from 0 by having just as much or less training data as humans. Which seems very likely false.

Word-object correspondances can obviously be learnt from observation in theory - it is widely accepted that there is enough information to learn such correspondances, even on a limited dataset. The argument by linguists is about grammar, not vocabulary.

Chomsky's argument is exactly the opposite, that vocabulary is learnt from scratch, but that the framework for grammar is innate.

> Proof is left as an exercise.

Rather conclusive one already established in the ML field.

Historically, Chomsky has been wrong about a great many things.

Particularly about things he never said.

Reply on news.ycombinator.com