Hacker News Reader

Comment by iandanforth | Hacker News Reader

Chomsky is right up there with Minsky in being part of the problem. His ideas about language being part of the genome are fanciful nonsense. Skinner produce reams of reproducible empirical observations of behavior which are, today, critical to the evaluation of the performance of AIs. Chomsky has produced interesting theories, but mostly derailed linguistics on the basis of the argument 'language is complicated, something magical must be going on.' His contributions to political debate are monumental. His contributions to science are negligible if not entirely detrimental.

View on HN

> His contributions to science are negligible if not entirely detrimental.

I'm not a linguistics expert, but from my vantage point as a computer scientist, it would be hard to conclude that. Chomsky essentially founded the field of formal language theory, and was the first to propose many widely-used constructs, such as the context-free grammar. They may or may not explain how English works, but pretty much everything on the Chomsky hierarchy has been put to productive use in computers.

In a much more general sense, his arguments against blank-slate learning are pretty mainstream in machine learning as well. Chomsky's specific approaches aren't too widely used (though there is some work on grammar induction), but the idea that inductive bias is key to machine learning is widespread, though perhaps in a weaker sense than the very specific and strong inductive bias Chomsky proposes for language learning.

Language theory predates Chomsky. The earliest known activities in descriptive linguistics have been attributed to Panini around 500 BCE, with his analysis of Sanskrit in Ashtadhyayi.[6] So, while Chomsky is famous his actual contributions are far less than you might suppose.

I don't think he was claiming that Chomsky IS language theory, or even that he pioneered the field. However, to say his contributions to science are "negligible" is absurd.

Possibly a linguistic (hah!) confusion. I don't mean that he pioneered the study of language, i.e. linguistics. Clearly people have been studying languages for millennia. When I say he pioneered formal language theory, I mean it in the computer-science sense, that he pioneered the study of formal languages. That's the theory of how to mathematically and computationally parse sequences of tokens into semantic meanings. Chomsky wasn't the first there, since work in symbolic logic (e.g. Frege's work) also considered the formal syntax/semantic mapping. But much modern work in parsing and related areas is derived from Chomsky's work. Any programming-language researcher has heard of and likely used the concept of a "context-free grammar" for example, which Chomsky originated. More information: http://en.wikipedia.org/wiki/Chomsky_hierarchy

Put more simply, I'm saying that he did valuable scientific work in developing his formal analysis of grammars, even if it fails to capture human language. He intended it to capture human grammar, but it's an interesting computation-amenable model of grammar even if it isn't how humans speak, because it is pretty much how we now write programming languages. When computer scientists today talk about "grammars", for example someone saying that they're writing "an ANTLR grammar for Clojure", they mean it in the Chomskian sense.

> > His contributions to science are negligible if not entirely detrimental. > I'm not a linguistics expert, but from my vantage point as a computer scientist, it would be hard to conclude that. Chomsky essentially founded the field of formal language theory

Chomsky did great work for CS (and NLP) through formal language theory, but his work in linguistics -- in particular, his anti-empiricism and hostility to statistical views of language -- was very unhelpful for getting computers to understand language. See e.g. http://www.cis.upenn.edu/~pereira/papers/rsoc.pdf

Chomsky revolutionized linguistics in the 1960s. Compiler theory incorporates a huge amount of his work. Calling his contributions negligible or detrimental is like saying Freud's contributions were the same.

Could you elaborate on Freud? I thought the general consensus was that none of his "theories" were accurate and evidence doesn't support or it directly contradicts his ideas. From the admittedly little I've read, it comes off as just "making stuff up" and generating frameworks that can explain anything (hence nothing). Psychoanalysis certainly has no place in modern psychology.

Or are you saying that Freud, despite being entirely wrong, got things moving and eventually helped folks to start taking psychology as a serious science? Or another point I am not understanding?

I am not the grandparent author, but I think your last paragraph describes it well.

Yes looking back Freud's theories are bogus. However, that is only looking back and taking in consideration what has happened after. However, whatever happened after in the theory of how mind and human psychology was very much influenced by Freud. A lot of it was a reaction but it was still a reaction to something. Those that came after studied Freud and it was their background.

To say it another way, it hard to predict what would have happened if Freud was not there. We could have been in a worse shape maybe today.

CogSci PhD student here, I'm no fan of Chomsky but I'd still say his contributions were a lot more useful than Freud's. Freudian Psychology is mostly abandoned by practitioners at this point; the Chomsky hierarchy is still canon in CS.

The Chomsky hierarchy while being but a small part of his work is undoubtedly a useful technical contribution, but I don't see it as particularly relevant for cognition and linguistics. As an example, for cognition memory constraints seem to be much more fundamental than the type of formal rules. It is also rather arbitrary: many kinds of formal languages don't fit into his hierarchy, there are many other ways to carve up the space of possible languages/grammars.

I think that his general linguistic theories might share the same fate as Freud's theories: large, complex theories for which there is simply not much empirical support, making it difficult for people to continue to work on them without their originator imbuing them with his authority.

Sadly this all sounds quite correct.

the Chomsky hierarchy is still canon in CS.

Though you have to wonder how much of that is a lack of interest in research. PCRE has been shown to not fit his hierarchy, they are more than regular but less than context-free.

What do you think of http://en.wikipedia.org/wiki/Nicaraguan_Sign_Language ?

It certainly makes me doubt language (the technology of it) as purely cultural and learned.

Language is a set of behaviors Behavior for humans and other animals requires a brain and body (almost always) The organisation, growth, and function of the brain and body are influenced by genetics.

So we can say first and formost that language is influenced by genetics. Having a tongue, vocal chords, hands, ears. These all help to acquire language.

So now the question is to what degree does genetics influence language and language acquisition?

Is it fully governed by genetics? We know that removing a child from the company of others during development eliminates complex language and severely stunts their ability to acquire language. So, in a single individual, language has never been observed to arise spontaneously and the facility to acquire language is something you can lose.

If you damage certain portions of the brain aspects of language can be removed. See Fluent Aphasia for a particularly odd example. However, these aphasias are known to remit and studies show that other portions of the brain have taken over from the damaged portion. So we know that the capacity for language is not entirely localized to a genetically determined location.

We also know there was a time when there wasn't language and now there is, so at least once in human history language did arise spontaneously. The article you mention is evidence that language can evolve spontaneously in groups of humans. In fact there is evidence that the rudiments of language arise in groups of many animal species including apes, whales, and dolphins. Many animals communicate, but with insufficient sophistication as to be described as a language.

So it is safe to say that there are genetic factors in producing communicative behavior. Sounds, postures, marking etc. I feel it is also safe to say that there are genetic factors that predispose groups of some species to develop more complex systems of communication (but these are not limited to humans) and that groups of humans are particularly good at complex communication.

Unfortunately for Chomsky there is little to no evidence that any one type of language is more likely than another. Grammars, phonemes, words, abstract concepts ... all of these have huge between language variation. The similarities between languages are well explained by either physical characteristics of speech production or regularities in the environment of the languages origin.

So yes, language is both learned, and cultural, even if it isn't purely so.

If I understand your point correctly, Chomsky does explain this, even in the article. Like with his discussion of "linear order".

I'm surprised he brought that up, it's a weird point. Because a different part of the brain processes the constructed language it's no longer language? It's also untrue. We construct language patterns, poetry for example, where the positional number of a word or a syllable changes its meaning.

That said his point about the brain being wired to take the less computationally intensive route is a very important insight which I think extends beyond genetics and throughout the evolution of all biological processes.

Actually, his point is that our language organ takes the more computationally expensive route. Not the easier one. That's the puzzle.

I don't have his books on me, so I may misconvey this point, but IIRC, he also mentioned regular languages (you know, like with regexes) as another example of a computationally "easier" language family we don't pick up with our language organ. We don't speak arbitrary languages. The space of languages is filtered by genetics.

He delves into this point more deeply in many places and addresses the points you raise with a precision beyond what's found in interviews.

I suppose it depends on your definition of 'expensive'. By that I mean, if your computational model is inherently serialised, sure, regexes are cheap. If fuzzy matching and rough temporal correlation between processing units turns out to be cheap, perhaps regexes are ridiculous extravagance on your hardware family.

I suppose you could turn it around - assuming our language processing is optimal (bit of a leap) you can infer things about our hardware architecture by the languages which we parse efficiently.

What's really fascinating is wild chimps use a rich form of communication that's far from an actual language. But, they can be trained use complex sign langue which they will then use to communicate with each other. So, language may have developed fairly rapidly relative to the underlying biology.

No, they can't be trained. Those studies were largely failures and the idea that they can be trained is little more than a myth.

You should read Steven Pinker's The Language Instinct. He summarises Chomsky and the field's main insights into language brilliantly. One of the points is that while we see languages as extremely diverse, most of the differences are actually quite superficial and can be efficiently modelled by a small set of rules with a few key configurable settings, and that this basic structure for language is universal among all humans. We are just tuned to focus on the differences and not notice the overwhelming similarities. Great book anyway you cut it.

For me, this perfectly sums up the futility of the Strong AI / AGI research project. So much of what we consider to be 'intelligence' or repercussions of intelligence are in fact language-based communications and culture - with an immense genetical legacy, bias, and quite frankly, burden. To create human-equivalent intelligence, most or indeed all of these evolutionary biases would have to be built in.

Consider a different example: constructing artificial vision. Human vision is the result of evolution, of course. It is incredibly inefficient, but evolution is blind (sorry). When we now construct computer vision, we capture an optical image and transmit it optically as long as possible, since this preserves information. The human eye does not: it sends information via neurons to the visual cortex, compromising immensely in bandwidth. That's why we need visual error-correction mechanisms in the brain, and redundancy of visual information (achieved by the eye moving rapidly many times per second, for example).

When we construct optical computer vision that achieves the a similar thing as human vision, the two have nothing in common. You can't plug in the artificial front end into the biological back end. The two systems produce literally different images, that are not comparable. The systems will not communicate with each other. We have skipped the legacy of biological evolution completely. To create an artificial system that accurately corresponds to the biological would be an immense waste.

The same goes for intelligence. Our 'wet' evolved intelligence is an entirely different picture from the project of Strong AI. They will produce very different manifestations of interacting with the world. Why would the latter ever result in the illusion of free will or the self-referentiality of personal identity, two things we assume are parts of human-level intelligence? They are like the neurological channels for conveying an optical image: hopelessly inefficient, but the ones that make sense in the light of our evolutionary legacy.

As a side note, yes we know of a time when there was no language, but it not a good representation to consider that a binary switch. We know of complex communication between other animals, and given that even current languages are in rapid flux, I think it is is fair to think of the transition from pre-language to language a continuum. Language is still arising.

I don't understand what bearing this should have on the probability of AGI succeeding. What we (should) care about is not whether the system is conscious or even human-like. What matters is that it performs well on the tasks it is assigned. I consider it highly likely that we can do better than evolution did, in less time than evolution took.

I hope you're kidding about his contributions to political debate. Chomsky has the maximalist attitude that represents everything that is wrong with political debate today. His hatred of U.S. foreign policy is so extreme that he will defend absolutely everyone the U.S. opposes which means occasionally defending tyrants and denying genocide.

He's no saint, but is commonly dismissed due to committing the cardinal sin of pointing out truths make people uncomfortable.

"My own concern is primarily the terror and violence carried out by my own state, for two reasons. For one thing, because it happens to be the larger component of international violence. But also for a much more important reason than that; namely, I can do something about it. So even if the U.S. was responsible for 2 percent of the violence in the world instead of the majority of it, it would be that 2 percent I would be primarily responsible for. And that is a simple ethical judgment. That is, the ethical value of one’s actions depends on their anticipated and predictable consequences. It is very easy to denounce the atrocities of someone else. That has about as much ethical value as denouncing atrocities that took place in the 18th century."

— Noam Chomsky

I'm amused by the idea that Chomsky is somehow some kind of dunce when it comes to science but a brilliant thinker when it comes to politics. At least with either claim individually, I can conceive of who might say that, even while thinking they're radically wrong (no pun intended).

But the idea that Chomsky's political "contributions" somehow dwarf his contributions to science? That seriously floors me.

> His hatred of U.S. foreign policy

Sadly enough for (U.S. foreign policy and its supporters) he actually supports his claims very well with sources and footnotes, and if you read through this works, you might find that perhaps there is a reason others (who don't just watch Fox News) don't agree with said policy, and also that somehow Americans in certain parts of the world are not "hated because of our freedoms". There are other reasons.

Cranks can load their screeds with sources and footnotes, too.

There's a reason no one outside the campus left takes Chomsky's political opinions seriously, and it's not a massive conspiracy.

I think his political thinking and language thinking share a similar problem: preference of ideal theories over practical solutions.

> His contributions to science are negligible if not entirely detrimental.

I think it would be nice to have some more references or more explanation before declaring Chomsky detrimental to science.

He dominated the field of linguistics for decades with anti-empiricist theories. You could argue that the popularity of his ideas held back alternative approaches; hence his contributions could be considered detrimental.

> He dominated the field of linguistics for decades with anti-empiricist theories.

But he dominated. Where was Norvig or IBM's Watson back then?

Unfortunately things like these don't happen in a vacuum. You can only say looking back that this is didn't or that was a bad idea. You should have said that at that time when the theory was advanced or came up with a better one.

We didn't exactly live in in a totalitarian regime where say some Politburo dictated what the official theory about Genetics should be and everyone else gets sacked, and now finally we have freedom from oppression and we can get back on track.

> We didn't exactly live in in a totalitarian regime

Ironically that is something that people allude to with regards to Chomsky (ironic because of his anarchist political beliefs); cf. the book the Linguistics Wars. The bottom line is that his theories were very dominant, not because of overwhelming empirical support, but because of his authority.

I'm not convinced that Chomsky is an 'anti-empiricist.' He's a theorist but his theories have been put to experiment, very successfully so I believe. Can you provide examples of Chomsky attacking the validity of experimental evidence?

How do you approach Heidegger's or Derrida's arguments that 'language is complicated, something magical must be going on'? Will you allow for no qualitatively different meaning to language than probabilistic associations of sounds with objects?

I can't speak to the OP, but extraordinary claims require extraordinary evidence; we are yet to observe anything in the natural world where "something magical [really turned out to be] going on", so this really is an extraordinary claim. It has been several years since I was current with the linguistics literature, but as far as I know no extraordinary evidence has been produced.

I don't think that requires that we view language solely as a probabilistic map between sounds and objects. All sorts of emergent behavior appear to be "magical" at first glance.

I did not reference linguistics literature, but philosophy literature (a distinction worth making because they are approaching the problem at different levels). I've never been good at the 'language problem' elevator speech, but how is it possible that we can capture the full power of language while using language to describe it? Language is a technology that allows for the production of concepts like 'probability' or 'apple'. Language as a tool is so fundamental to cognition that it becomes difficult to separate the two. Heidegger's Being and Time explores that line of thinking, and Derrida loves to play the game of "well that means you can't say anything!" Both have provided me with insight into what "magic" is referencing here (rather than your fairly pithy absence-of-evidence reference).

You're asking if I believe in qualia, and the answer is no. There are firing neurons and that's it. The great variety of ways in which neurons can fire, and the great variety of experiences that shape how neurons fire combine to form an exquisite set of possible firing patterns (this is literally what makes me me and you you) but ultimately, to mis-use Gertrude Stein's famous phrase, 'there is no there there.'

I don't believe I referenced qualia. The problem of language is an emergent phenomenon just as the utility of language is. I don't believe in subjective essentialism either.

It wasn't at all clear that that is what he was asking you. And you need to qualify the sense in which you "don't believe" in qualia. You don't believe that consciousness has phenomenal properties? Qualia certainly exist in some sense.

From what it sounds like, you are just dismissing compelling philosophical issues because it frustrates your beliefs.

> Qualia certainly exist in some sense.

As an epiphenomenon of neural firings? Or is saying that 'not allowed' somehow?

What could 'meaning' possibly be other than probabilistic associations of sounds and concepts?

That is EXACTLY the stance that Chomsky is challenging here. Don't get me wrong, I think the probabilistic view has truth value and is a powerful predictive tool. However, I don't think it captures the phenomenological properties of the act of language based cognition. See my cousin reply referencing Heidegger.

Your observation is so vague and general to the point of being rather meaningless. Almost every physical theory is described by an underlying mapping between inputs.

The interesting point is the expression power of your model. : to take an example I am somehow familiar with, current large vocabulary speech recognizers have millions of parameters. They work relatively well, but they are very difficult to interpret, and it is hard to see how they help us understanding how speech recognition actually works in our brain.

To make a somehow flawed analogy, every Turing complete language is equivalent, but getting the machine code of a very large project is not very interesting if you want to understand it, while it is mostly enough if you just want to use it.

Do you have any reason to suspect this isn't how the brain works? Maybe language isn't a small set of high level rules. Why should we suspect it to be? The probablistic models seem to be very similar to how real people actually learn informal language. Formal languages of course have high-level rules, and these are well modelled algorithmically.

I don't particularly have any reason to believe one way or the other. Certainly, the probabilistic models for language are created "out of the blue" without any attempt to model how human learn languages.

The idea that meaning consists of associations is extremely primitive. It works for concrete nouns and verbs but it quickly fails as things get more complex. Language is used to refer to refer to abstract things, imaginary things, counterfactual situations, etc. And even if you do arrive at a series of concepts using associations, you have to understand how they are supposed to combine, even for completely novel sentences. In all these cases, there's arguably nothing there to associate with. I can't answer your question (I think no one can), but we can conclude that meaning is more than associations.

Isn't that Chomsky's argument here (in this article, not in his approach to linguistics in general)? -- That it is a good idea to try to find a better understanding of how the internal mechanisms work so we can build or simulate it better. You do that with carefully constructed experiments not with just observing inputs and outputs and training a neural network or a Markov model with it.

Please argue that there is nothing to associate with.

Why is a real observation from your senses more privileged inside your brain that a random well-formed value by a (hypothetical) random number generator neuron?

I argued that if you only have associations with previous experiences, you won't be able to deal with novel input. Ergo, you need more than just associations (synthesis, imagination, counterfactual reasoning, etc.).

As to your second question, I don't see how it relates to my argument, but I'll answer anyway. If you're comparing a observation to a random number, you're looking at the observation qua value, in which case it has the same status. If, however, you look at the level of interpretation (what it means in your brain), the observation has a complex set of relations with the rest of your brain and gives rise to a perception, wheres the random number value is just noise that has to be tolerated by the brain.

Chomsky claims language is built-in. So probabilistic associations are the exact opposite of his claims. Interestingly, there have been some very good baby studies that show babies inherently know statistics needed to learn probabilistic associations. You can show them a couple different color balls going into a box, then take out a bunch of the rare color that went in, and they will register more surprise even long before they can even talk. Things like that. Chomsky's assertions are, well, what you would expect from someone that old. They didn't understand neurons and biology and genetics so well then, so yay, magic things are possible!

> Chomsky claims language is built-in.

In general he does. In this article he doesn't talk about he talks about approaches to AI.

> there have been some very good baby studies that show babies inherently know statistics needed to learn probabilistic associations.

Very good. Can we identify how that works and then build a robot that has the same mechanism in a more efficient way than simply simulating a brain at a molecular level. That is his argument here.

> They didn't understand neurons and biology and genetics so well then, so yay, magic things are possible!

So where were you 4-5 decades ago when he proposed his theory to propose a better one?

Saying everything is just probabilistic associations is like saying everything is made from quanta of energy and thus every higher level concept or model is useless -- just simulate the quarks and you are set. Not only that -- simulate it by recording and observing the energy patterns going in and coming out of a black box.

Yes you can get some things to work and some to work well but the idea is that perhaps there is a better model that describes the mechanism or the encoding of meaning. That's what Chomsky is trying to say in this particular article. Some stopping at a brute force approach is a fine engineering approach but that doesn't mean everyone should, it is still worth trying to find a better model for it, if at least, just to gain an understanding.

It's skyhooking. Given enough time and enough compute power, we should be able to completely simulate it. It may not be economically feasible, and it may take a long time.

I had the same notion from the first part of the interview - wondering whether Chomsky would actually bring something to the table to "unriddle" this mystical machine in our head. Leaving his aversion to computational systems aside, I believe he makes a good point about our language being computationally inefficient (at least externally). maybe his insight [that our language is computationally inefficient] can be explained by molecular mechanisms that are efficient in terms of energy consumption (rather than on a computational level). looking which neural systems are most likely to evolve given a simple genetic code and reengineering (computationally inefficient and obscure) algorithms based on that might provide some valuable insight into the human brain - whereas probability measures alone might fail to explain underlying mechanisms.

I made similar comments when the Chomsky vs. Norvig discussion came up on HN some time ago and got interesting replies: http://news.ycombinator.com/item?id=4291327.

I don't think you fairly represent Chomsky's theories about language, nor do I think you really understand them. There's no magic, and there's nothing even remotely ridiculous about language between a genetically-encoded instinct. Human beings have evolved complex brain modules which allow us to rapidly acquire complex combinatorial language systems. Language is to humans as hunting is to lions: a highly-evolved, highly-adaptive instinct.

"His ideas about language being part of the genome are fanciful nonsense."

Oh, so, linguistics is just magic then?

If linguistics isn't part of the genome, why can't you teach a dog to talk?

Reply on news.ycombinator.com