Hacker News Reader

Comment by Barrin92 | Hacker News Reader

It's one of the fundamental critiques Chomsky already brought up decades ago with the ascend of statistical methods in linguistics.

They may be useful in the sense of generating commercial value, just like ML is useful to random-walking you to some solution that you couldn't have come up with, but it is not science.

There is little insight to be gained from this, it is more guesswork and art than anything else, and likely at some points the practical results will diminish as soon as one stumbles about more fundamental problems and has no model or structure to reason with.

ML is essentially behaviourism on steroids.

View on HN

My problem is that Chomsky seems ardently committed to the "botanical" approach to natural language as the One True Way (TM).

By this, I mean the assumption that there is some universal logical structure to human language, and that our job is simply to come up with the labels for each constituent part and arrange them all correctly like a jigsaw puzzle. Once the puzzle is complete - boom! We've "solved" language.

That would have been a reasonable starting point back in the 60s, but I don't think it's advanced the field nearly as far as statistical methods.

Chomsky would retort that "advancing the field" is meaningless unless it allows us to "understand the field". But if natural language is a largely arbitrary collection of rules in constant flux, there would be very little of this "understanding" to be had. You could be bashing your head against the wall for centuries trying to induce structure into a bunch of symbols where none was ever to be had.

Now I do believe that purely statistical methods will hit a wall - the same way you could never teach a baby to communicate by throwing it a copy of Wikipedia and nothing else.

But if some parts of language are essentially arbitrary, then statistical methods are the best available tool to uncover that.

Language is infinite but the generating process (i.e. our brains or computers) are finite. The generators come in two flavors: rigid rule based (like Chomsky suggests) or stochastic (more modern view). The stochastic generators are obviously more powerful but difficult to "understand". Unfortunately, we associate the word "understand" with ability to have complete list of static rules. In my opinion, probabilistic rules are also as much "understable". Then there is dynamical probabilistic generators which I think our brains aren't evolved enough to truly "understand".

> Now I do believe that purely statistical methods will hit a wall - the same way you could never teach a baby to communicate by throwing it a copy of Wikipedia and nothing else.

Nick Chater and Paul Vitányi wrote papers[0] demonstrating that "the learner has sufficient data to learn successfully from positive evidence, if it favors the simplest encoding of the linguistic input":

- ‘Ideal learning’ of natural language: Positive results about learning from positive evidence,

- The probabilistic analysis of language acquisition: Theoretical, computational, and experimental analysis,

- Language Learning From Positive Evidence, Reconsidered: A Simplicity-Based Approach.

[0] https://homepages.cwi.nl/~paulv/learning.html

>> Chomsky would retort that "advancing the field" is meaningless unless it allows us to "understand the field". But if natural language is a largely arbitrary collection of rules in constant flux, there would be very little of this "understanding" to be had.

Imagine a historian of the future reading reams of code written for all the software we have in the modern era, in all the programming languages of the modern era, from Java to Brainfuck and back. Would the historian of the future be justified to conclude that there is nothing that connects these languages with one another, that there is no common structure, nothing that can unify them, "very little understanding to be hand"? If so the historian of the future would be wrong, because all modern programming languages are designed to run on the same substrate, modern digital computers. The existence of the digital computer is the unifying characteristic that explains the existence of any modern programming language.

In the same way, the variability of human language, internal or external, is without any shadow of a doubt explained by some processes in the human mind. These processes may be too complex to understand, perhaps, but understanding them is a worthy scientific goal. And there is, indeed, much "understanding" to be hand in that direction.

I think Chomsky's opinions held back linguistics by decades, and we are only making progress because of the ascension of statistical methods.

Chomsky didn't just argue that the statistical method lacked insight, but that humans can't really learn language statistically. Given that our artificial neural networks are much worse at learning than humans are, and yet they can simulate language much better than expected, this refutes Chomsky's arguments.

Non-statistical approaches have basically zero successes, commercial or otherwise. ML at least gives us evidence of what aspects of language statistical models can explain and what they cannot, which is a hint for future research if linguists are just willing to take it.

How does demonstrating what artificial neural networks can do refute the argument that humans can't learn something in the same way?

All it does is refute the idea that artificial neural networks can't do that.

I'm not commenting on how we learn or Chomsky's correctness or otherwise; just pointing out the unjustified leap from what AI tools can do to what actual intelligence does.

Chomsky argued that it was literally impossible for a statistical model to learn language, so to the extent it's possible, it refutes Chomsky's argument.

We won't know how humans learn language until we understand how humans learn language, but the fact that a crude general-purpose model like ANN works as well as it does is certainly evidence that humans can learn language statistically. We don't know for sure, but we rarely do.

> but the fact that a crude general-purpose model like ANN works as well as it does is certainly evidence that humans can learn language statistically

Humans learn language intuitively (subconsciously), as well as formally (through academic training). Humans understand language (to a greater or lesser extent in any particular case).

Statistical NLP is not, in any human-translatable way, understanding. It is approximation and prediction, in a purely mathematical sense.

Perhaps one of us misunderstands, or is imprecisely using, the phrase "learning language statistically", but I am unaware of any evidence that humans do so.

Chomsky's linguistic theories are not about understanding. Hence his famous example of "colorless green ideas sleep furiously", which is a meaningless sentence that we recognize as grammatical.

Actually understanding language the way humans do would require AGI, I suspect.

Chomsky argued that it was literally impossible for a statistical model to learn language, so to the extent it's possible, it refutes Chomsky's argument.

That's a different claim to the one I countered.

I agree with your last point with the caution that there is no reason yet to think the hyped current AI tools have great relation to the way the human mind works, solely on the basis of the lack of progress towards AGI.

>They may be useful in the sense of generating commercial value, just like ML is useful to random-walking you to some solution that you couldn't have come up with, but it is not science.

This is just normal for the scientific process, and I'd argue that ML currently is more rigorous than any of the soft sciences. The truth is that we likely need a new field of math to properly model neural networks...until then we are gleaning vast amounts of empirical but still scientific data as to the behavior of this new tech.

The proof is that these nets work and are already doing amazing things.

Reply on news.ycombinator.com