Hacker News Reader

Comment by maest | Hacker News Reader

I think Chomsky's position is that _humans_ are primed for language, but it doesn't mean there exists no system that can learn language from 0.

Also, the ability to understand language is a spectrum, not a binary quality.

View on HN

And with that in mind there is an obvious counterargument to the original "No". If humans learn language through observation and innate knowledge encoded into the brain then we can construct an artificial system that discovers language purely through observation by encoding the same innate knowledge into the system.

Algorithms necessarily contain some knowledge about the data they are working with so that isn't cheating. Neural nets for example have to have some architecture and that predisposes them to learning certain patterns.

He does claim there is no system which can learn language from 0 by having just as much or less training data as humans. Which seems very likely false.

Yeah the argument is that there is a poverty of stimulus (not enough training data) in the case of children. The same doesn’t apply to ML models which need an abundance of stimulus.

There was some linguist (Deb Roy?) who videotaped his children growing up. One of his observations was that every learned syllable etc. was tied to observing it as a stimulus and trying to imitate it. Now it is true, children are pretty good learners, they can often learn something the first time they see it, but actually I have seen some LLM stuff about "instant learning" - e.g. the training is only done with one pass over the material. https://www.fast.ai/posts/2023-09-04-learning-jumps/

This is an interesting connection. If I had to guess, the Chomskyist response would be to say, yes but this only applies to LLMs that have been pre-trained already (i.e. have structures in place needed to understand language in general). I think a Chomskyist would say that language learning is precisely like fine-tuning and not the pre-training of foundation models.

Reply on news.ycombinator.com