On Chomsky and the Two Cultures of Statistical Learning
norvig.com
norvig.com
In the case of language, observing and reporting statistical probabilities in written/spoken language output does very little to explain the cognitive systems used in acquiring and using language. Even one statistical anomaly serves to show that statistical learning is NOT the entire picture when it comes to language development.
There was another article on HN a while back that had another great quote from Chomsky that does well to illustrate what I feel is his main point here: "Fooling people into mistaking a submarine for a whale doesn't show that submarines really swim; nor does it fail to establish the fact". Creating a computer that can produce millions of grammatical utterances does little to show that we understand language systems. Now, if a computer could - like humans - learn to produce infinite, novel, contextual, and meaningful grammatical utterances, that's a different story. But that story will take a lot more than statistical learning to write.
in some sense this is the same argument as searle's chinese room. that's sufficiently well known and debated that it's fair to say that neither side can be dismissed as simply "missing the point".
This isn't one of them, though. And I didn't mean to imply that anyone was an idiot.
Instead of critiquing how I said something, or that I said something at all, can you tell me what I am missing? I'm obviously missing something - Norvig is no idiot.
but there's nothing to say that the same is true for intelligence or language. maybe the brain is nothing more than a particularly flexible "neural net", which statistical methods are modelling quite well. in that case, "intelligence" is not qualitatively different from "a good simulation of intelligence".
the same problem occurs in free will - does it "really" exist? if we're just (mechanical, predictable, although highly complex) machines then it is difficult to imagine how it can. yet the intuition is that there is clearly some meaning to the idea of a "free agent".
these are hard questions. people don't know the answers. instead we look to what daniel dennett calls "intuition pumps" (see his book "elbow room" on the free will problem) - simple parallels that "feel right". from those, we use intuition to argue in one direction or another. but the problem with that approach is that it depends on what you choose as a "hint".
some advance is being made through experiment. imaging of neural activity in the brain, for example, or the recent discovery that people who believe they have free will behave differently to those that don't.
searle's chinese room argument - http://plato.stanford.edu/entries/chinese-room/
unreasonable effectiveness of mathematics - http://www.jstor.org/pss/2321982 http://plato.stanford.edu/entries/mathematics-explanation/
i can't find the free will behaviour result that was in the news about a week ago. but that's perhaps less related to this anyway.
and i doubt that everyone who thought the earth was flat was idiotic, frankly. just because something is obvious now doesn't mean it was a stupid question, or easy to answer, when first raised.
now that I found quite funny :)
(yeah, most likely it is just that)
From an engineering standpoint, however, Chomsky's view of grammar has been remarkably barren when it comes to machine processing of natural language. It's made a major contribution to artificial languages but despite a lot of effort it hasn't added much performance to what can be done with statistical methods.
I'd agree that a hidden Markov model that does POS tagging with high accuracy doesn't provide an intellectually satisfying model for "how language works", but you don't need to have a model for "how language works" in order to use it.
That technology is impressive to the utmost degree, I just don't think it should be the endgoal of modern science.
We should look for it, but we should not expect to find it.
Moreover, rule-based NLP approaches also have their place, and they are often the direct result of theoretical advances. A case in point is the modelling of morphophonology (which is necessary for spell checking, dictionaries and text generation for morphologically complex languages): many successful approaches are those based on finite-state machines, which could not have happened without Johnson and later Koskenniemi using them to formalize the rule-based approach pioneered by Halle and (yes) Chomsky (well, not quite, but this is still the point of reference for rule-based phonology).
(I am a theoretical phonologist, but my colleagues who do actual NLP work of this type tell me that statistical methods aren't that great for the sort of work they do.)
I'd agree that an adhoc equation that fits observed core sample data doesn't provide an intellectually satisfying model for "how sedimentation works", but you don't need to have a model for "how sedimentation works" in order to use it.
That's actually from something I worked on a long time ago. People actually use adhoc models of sedimentation and the formation of sedimentary rock for practical purposes. I also think most people suspect that we'd learn something valuable by figuring out the underlying reason why the data fits the particular description.
The existing evidence overwhelming suggests that a computer that can "learn to produce infinite, novel, contextual, and meaningful grammatical utterances" will be based on probabilistic models. In fact it's hard to imagine how it could possibly be otherwise.
The computer is observing noisy sensory input and is trying to make inferences about how to communicate with some future reader. Mathematically, there is only one way to write this problem: probability. It's true that the learned model may have amazing structure to it, but this will almost certainly be learned via probabilistic models rather than being hand-coded by some future Chomsky.
That fact does not imply we will understand language systems or the human mind. The Chomsky route may be better suited for that task.
Statistical learning explains a lot. I'm a huge fan of it. Skinner's Behaviourism also explained a lot in psychology. But, just as in the case of behaviourism, I fear that statistical learning has/will hit a wall at which its explanations become futile and overly simplified.
My personal belief is that, at that wall, we'll see that human language instincts and evolved language-specific mechanisms will be what we are looking for.
"if a computer could - like humans - learn to produce infinite, novel, contextual, and meaningful grammatical utterances"
To perfectly achieve this goal, you might have to simulate 4 billion years of evolution under the same conditions as it happened on Earth, and a few thousand years of cultural evolution as it led to our languages and our cultural context. Language is incredibly complex and changing, many of its details might be incidental, i.e. results of random events, so it seems unreasonable to pretend that we can deduce it all from some elegant first principles. At least that is my reading of Norvig's argument.
In situations like this, I tend to speak in theoretical absolutes. A computer that "could - like humans - learn to produce infinite, novel, contextual, and meaningful grammatical utterances" isn't even on the timeline right now, but it's the theoretical goal in showing that we understand language acquisition (ontogenetic development), evolution (phylogenetic development), and production.
Just because that goals seems unattainable doesn't, to me, mean that we need to aim any lower. Now, this is premised on my belief that mimicking phenomena with statistical learning is not as intellectually satisfying as understanding the underlying cognitive systems, but that's not believed by everyone.
If that is the case, then the argument that Norvig is making is irrelevant to the argument Chomsky is making. Chomsky simply makes the point that statistical accounts lack explanatory adequacy. As someone who has worked closely with many of his students and who has received extensive training on his scientific program, I can say with confidence that Chomsky would have no objection whatsoever about the usefulness of statistical approaches to linguistic engineering problems. The results speak for themselves. He would go on to say, however, that how well a statistical approach solves a linguistic engineering problem is irrelevant to the question of how humans do what they do.
The answer to the question may well be statistically grounded. That is a valid hypothesis and a logical possibility which should be taken seriously. However, it is incumbent on the proponents of such an answer to provide evidence that it is what humans are doing. Here are some examples of the kinds of evidence necessary:
* evidence that humans are capable of performing the kinds of computations that the statistical approach requires,
* evidence that the statistical approach works with the relatively limited amount of data that a human receives,
* evidence that the statistical approach fails in ways that humans fail
How well a statistical approach succeeds at an engineering task is not an item on this list, simply, again, because engineering tasks are irrelevant to what humans actually do.
Let me specifically say that statistical approaches are not, from the start, ruled out as potential candidates for the algorithms underlying human language. It's just that a case has to be made for them using the right kind of evidence.
Finally, I'll reiterate what others have pointed out: from a scientific perspective, that something is hard to explain doesn't mean that we shouldn't try. And, those that have given up (as you suggest Norvig has) shouldn't fault those who haven't for calling them out on it.
Unless, of course, those cognitive systems are nothing more than some statistical probabilistic mechanism. I don't know anything about the field, but the article was interesting to me in that it seemed to at least partly argue that. I know, for me at least, I'll frequently produce a sentence and then repeat it to myself a few times to see if it "sounds right." Now, I don't know what is happening to determine that, but perhaps I'm comparing it to some statistical probabilistic model I have in my head?
> Even one statistical anomaly serves to show that statistical learning is NOT the entire picture when it comes to language development.
1) Does it? Maybe it shows the specific statistical probabilistic model in question is wrong. Consider, as Chomsky did, a model which predicts zero probability for a novel sentence. Clearly, as you say, one anomalous novel sentence is all it takes to disprove such a model. But what about other models which can handle them? The "anomaly" may not be an anomaly anymore.
2) Do you have some anomaly in mind which shows statistical probabilistic models don't work?
-----
The article was very interesting to me, but I don't know anything about the field. I guess my main question boils down to: Is it possible that language acquisition and production is nothing more inside our heads than a simple statistical probabilistic model?
Why not? Look at it from the bottom up:
Communication is a fundament of life, from intra-cellular to inter-cellular to inter-organism interactions (another fundament is the ability to keep oneself in a low entropy state, at the expense of the rest of the world).
Human speech is an evolution of mammal communication. It grew up in complexity, from grunts and other basic noises, along with our way of living, up to what we have now.
> And how do we make completely novel utterances that attempt to express our otherwise abstract thoughts?
Speech is a big collage. New is either the result of
* a recombination of the sub-parts of past speech
* the definition of a new word in terms of older words, or sometimes arbitrarily (for proper nouns).
Nothing fancy AFAICT.
This view is just to simplistic to hold its weight when you really look at the intricacies of language and its evolutionary history which, by the way, I would suggest comes from manual gesture and not grunting.
Mu![1]
But you're probably right about gestures.
Wild chimps have a vocabulary of about 66 signs. We can also observe tribes with languages more primitive than ours (no pronouns, for example). But there's a missing link of several millions of years of evolution between both.
What are the (known) intricacies of the evolution of our ability to communicate?
There's no definitive proof for the statistical argument, but a growing amount of (neuro)scientific evidence points to it. What's (are) your alternative hypothese(s)?
I think that most people who believe in some form of the motor theory of speech perception will also believe that speech evolved from manual gesture.
Others scoff at the motor theory. In fact, I'd say I'm in the minority by bringing it up with any regularity.
If the question, what is "known" about the evolution of our ability to communicate, I wouldn't have much to point you towards. Most is theory based on modern evidence, somewhat like armchair psychology. Other people point to our ability to integrate non-verbal gestures into our comprehension, activation of our motor cortex prior to semantic/phonetic network activation when disambiguating difficult speech sounds, our ability to synthesize visual/auditory sources of information when the visual information relates to speech gestures (mouth/tongue movements), etc.
Aren't these issues completely orthogonal?
That criticism can be lobbed at all abilities that we claim came about due to evolution - which, to be clear, is all of them. The statistical model would be the mechanism, but it wouldn't be the reason why it evolved. That answer is relatively boring, and is the same one as all evolutionary processes: it appeared randomly from mutation, and it provided benefit to those that had it.
Not just boring, but a totally banal and useless answer.
What kinds of mutations? In what sequence? How did it provide a survival benefit? What were the earlier forms of language like, and how did they become the languages spoken today?
Just saying "evolution did it" is about as informative as saying "God did it."
It's the domain specific tools that are up in the air, though, and what I'm interested in.
It's not my field so I can't give examples off the top of my head, but the argument involves rapid acquisition of syntax and near-complete absence of errors that you'd expect to see in a simple statistical model.
You don't need a raw MC to assess the likelihood of "The DOG ate my homework", "My WASHING MACHINE ate my homework", and "MY LEGALLY ate my homework". You need P(WASHING MACHINE = NOUN), and P("My NOUN ate my homework").
But that's not right. You could also have P(WASHING MACHINE = NOUN THAT CAN EAT STUFF). Or maybe P(EAT = HUMOROUS TERM FOR DESTROYED), P(HUMOROUS SENTENCE), P(WASHING MACHINE = OBJECT THAT CAN DESTROY HOMEWORK).
Anyway, it's really bloody hard to put it all together. But that's what humans do. I'd imagine that we store it in our short term memory, then make a few quick parses of it, under varying assumptions, and keep the ones that are most consistent.
In reality, Chomsky is fighting the same sort of battle that happened when Newton and Leibnitz were around (no, not the battle between Newton and ... the rest of the world really). OK, you have gravity. But what causes it? Why? It's an interesting question, but not necessarily one that will lead anywhere.
The failure of a machine to understand a sentence given a set of rules, may simple mean that it needs to be taught new rules.
I had a non-native Japanese teacher once who, when asked a question on proper Japanese usage, would often stop for a second, clearly playing the sentence or phrase over again in his head, and way "no, they don't really say that" or "yes, they do say it that way."
Clearly, he was using his extensive experience listening to Japanese over many years to determine grammaticality, so at least a statistical model, if not conclusively a probabilistic one.
However, the preponderance of evidence* so far suggests that we are just statistical processing machines. Hence why Chomsky seems way off the mark.
*We know that various layers in the visual and auditory systems basically just compute ICA, and we know that the brain is incredibly plastic. Large areas can be removed and the remainder will compensate. That makes it seem likely that all neurons compute something like ICA (or at least that degrades to ICA when confronted with visual or auditory input.)
Do we? Intellectual cynicism that makes people to publicly reduce humans and humanity to something mechanical and predictable is probably the most popular attitude I see online. This doesn't mean it's wrong in all cases, but surely it's not something exceptional.
Looking at the statistics and evidence is of great importance in trying to form models and answers to the "why" questions. Although mimicking a bee dance may not mean we understand it, it does provide a basis for founding and comparing theories.
No one, as far as I know, argues that humans actually solve differential equations in their head when they catch a ball. They just... catch it. Perhaps with some failed attempts along the way, but as a part of growing up, we learned basic eye-hand coordination.
The notion that syntax and grammar as we have formalized it exist in our brains is the same as saying that differential equations exist in our brains. I find it much more likely that we innately have rough models for syntax, grammar, mechanical movement and object trajectories, but that it takes significant trial-and-error for us to tune those models until the point of competence. I think these models have to be at least partly statistical - otherwise, we wouldn't need to learn anything - and that while our formalisms may be nice approximations of what we do in our brains, I see no reason why they have to be exactly it.
I would argue that we actually find approximate solutions to differential equations with what is likely best described as an analog computer.
So essentially, humans and computers as they are currently programmed do it the same way. The only difference is that computers do it better.
Are you saying that one statistical error in a probabilistic model makes the entire model wrong? Then you'd equally have to say that one logical error in a categorical model makes it equally wrong. And manifestly, there are many logical errors in all grammars. So I'm not sure what your point is here.
I'm interested to know: I quoted Chomsky: "That's a notion of [scientific] success that's very novel. I don't know of anything like it in the history of science." Do you agree with him? If so, do you judge all the Science and Cell articles as not being about accurately modeling the world and only about providing insight? Or do you think Chomsky meant something else by that?
I understand that there are two goals, accurately representing the world, and finding satisfactorily simple explanations. I think Chomsky has gone too far in ignoring the first, but I acknowledge that both are part of science. I further think that statistical/probabilistic models of language are better for both goals. This is obvious to me after working on the problem for 30 years, so maybe it is hard for me to explain why. I think Manning, Pereira, Abney, and Lappin/Shreiber do a good job of it. Also, I don't see how a system that successfully learns language could be anything other than statistical and probabilistic. I agree it is a long ways away ...
-Peter Norvig
On one end of the continuum, n-gram models for large n with infinite training data estimate the empirical distribution of language and thus are the best you can possibly do. On the other end, rule based grammars directly transcribe intelligible "rules" of language generation and comprehension. Both ends are clearly fraught with problems.
In the middle we have topic models, recursive grammars, decision trees, various ad-hoc smoothing methods, each of which both allowing for more tractable training and introducing more meaning to the parameters of the trained model.
I feel like effort here provides (somewhat unsatisfactory) answers to both criticisms. I think it's fair to say that probabilistic/statistical models deserve more attention in a lot of fields in order to overcome a history of neglect, however.
Could you give some concrete examples? As a linguist, I don't see that statistical models are currently giving us much insight in those areas where current syntactic theory does give some insight. So for example, we don't seem to have learned much about relative clauses, ergativity, passivization, etc. etc. through these models. On the whole, statistical methods seem very much complementary to traditional syntactic theory. This seems to be Chomsky's view also:
"A quite separate question is whether various characterizations of the entities and processes of language, and steps in acquisition, might involve statistical analysis and procedural algorithms. That they do was taken for granted in the earliest work in generative grammar, for example, in my Logical Structure of Linguistic Theory (LSLT, Chomsky 1955). I assumed that identification of chunked word-like elements in phonologically analyzed strings was based on analysis of transitional probabilities — which, surprisingly, turns out to be false, as Thomas Gambell and Charles Yang discovered, unless a simple UG prosodic principle is presupposed. LSLT also proposed methods to assign chunked elements to categories, some with an information-theoretic flavor; hand calculations in that pre-computer age had suggestive results in very simple cases, but to my knowledge, the topic has not been further pursued."
Anyway, if you want to pursue this critique of Chomsky further, I'd recommend a bit more background reading. This article gives a fuller explanation of the views he was outlining at the conference: http://www.tilburguniversity.edu/research/institutes-and-res...
>Or do you think Chomsky meant something else by that?
He presumably means what he said, namely that merely creating accurate models of phenomena has never been the end goal of science. You acknowledge this yourself when you say that you take both modeling and explanation to be part of science.
What it didn't seem like Norvig got was difference between understanding and a highly sophisticated pretender. Gut Level vs Self Aware intelligence. Both are valid forms of intelligence but only one is a valid form of understanding.
I think statistical methods are a form of intelligence that are highly mechanical and could never achieve human level cognition (ie fart jokes). But I could be wrong, usually am more than half the time.
Chomsky is a colossus, his achievements are well-known. However, at one point in many disciplines it comes to pass that the pioneers who pave the way in time become the very impediment to new ideas. His emphasis on Semantics have warped the minds of many generations of researchers (and some other ideas on universal grammar, too).
I experienced this first hand, my advisor, Prof. Raskin, a great researcher on semantics, nevertheless thought that statistical approaches were not the way to go. Sadly, in many Linguistics departments people are just not equipped with the statistical tools necessary to have a basic understand of what's being done in the NLP field. So NLP is generally taught under CS, EE, or CompE.
What are you talking about? Chomsky has always been highly critical of formal semantics.
This is the money shot especially since speakers are aware of the interpretive activity of listeners, and effective speakers play constantly on the ambiguities in their statements - structural (i.e. grammatical) ambiguities as well as semantic ambiguities. Listeners in turn are aware of speakers' awareness of this.. There is, effectively, an infinity of mutual awarenesses of structural ambiguities. In any instance of communication.
I think most technologists and (especially) businesspeople see this intuitively. I think many academics do not. Not sure how to articulate what I mean but I think I am saying something non-trivial about academics and their perspective on language.
http://www.nybooks.com/articles/archives/2011/mar/10/how-we-...
I became convinced that there is a strain of thought, one that is especially pervasive in the academia, which believes that knowledge/meaning is something irreducible and almost mystical. It probably has to do with the fact that people who fetishize knowledge as something incredibly worthwhile for its own sake end up being overrepresented in the academia. Those who are a bit more cynical/nihilistic tend to go into finance or start their own companies.
The old advice "do not make any gods to be alongside me" is still relevant except for the "alongside me" part, which probably only has any meaning if you consider yourself religious. I have a feeling that many academicians, especially the old-school ones, idolize knowledge to the extent of ascribing to it god-like powers even if said knowledge has little relevance for anything practical.
EDIT: stop giving me upvotes. I've got 11 points now for nothing more than a link. I don't deserve them. Stupid hidden points...
Part of the trouble w/ Chomsky is that real language doesn't draw a clear line between syntax and semantics. Even though an HMM doesn't correctly model the nested structures that are common in natural language, it makes up for it by encoding semantic information.
It's not the language that defines how we communicate, it's how we communicate defines the language.
But I also disagree with peter when he says the why is not important, it is this why or the understanding of the matter that separates us from the machines like watson, since our sole purpose in life is not to win at a game, but play/enjoy the game and most importantly "reuse the understanding" gained in some other facet of life, a feat that I beleive no machine is capable of.
No, it wouldn't disappoint him at all. In fact, one of his earliest works in linguistics discussed how transition probabilities could be used for chunking and categorization. (See http://www.tilburguniversity.edu/research/institutes-and-res... ) It's not as if Chomsky ever presented part of speech tagging as a poverty of the stimulus argument.
... My jaw is on the floor. It drives me nuts when people go from 'We can't explain that yet' to 'The only explanation is God.'
The tides are incredibly complex when you insist on 'why' all the way back to the beginning of the universe. Everything is!
I don't think anyone mentioned God, and, to be fair, the problem of what exactly constitutes reality and what would be the best ways to imitate it is quite complex. We, as a species, have been trying to find the rational answer to this problem for at least 2,500 years (since the pre-Socratics), but as far as I know we haven't come to any definitive answer, we don't even know if there is such an answer.
Science assumes that there is a "justification" (for language, the universe, you name it). A true philosopher always begins by asking himself if there is such a thing as "justification". Just think about it, science has helped us put a man on the Moon and create the Tsar bomb, but it isn't able to answer Epimenides's "All Cretans are liars" paradox, a 2,500 years-old problem.
You take as a given that paradoxes are or at least should be "un-answerable". This Wikipedia page (http://en.wikipedia.org/wiki/Liar_paradox#Possible_resolutio...) mentions a list of guys that tried to answer it, but anyway, this problem is way over my head, I should go back to writing Drupal modules before I also start rambling about Solipsism (http://en.wikipedia.org/wiki/Solipsism#Gorgias_.28of_Leontin...) :)
“Paradox” is just a word. There is nothing special about something being a paradox (except that paradoxes are cool to think about).
†That’s at least a common definition. That’s where I’m getting such crazy ideas like “paradoxes have no answer.” You are free to define “paradox” some other way. Be assured that it was never my intention to claim that the definition I used is in some sense the true definition. Definitions are all about communication (you need to agree on definitions in order to be able to talk to each other), not tools for finding the truth.
"Mistake" means there is a "right answer", a point of reference. What is that? Otherwise, your use of the concept "mistake" doesn't make sense.
> Paradoxes are defined as un-answerable
Yeah, of course we can define things ad-nauseam. That doesn't just make "reality" any more "real", it just helps us send men on the Moon, curing diseases or building atomic bombs.
That’s the ultra verbose version of that sentence. I can crank the verbosity up quite a bit still but I would rather not want to.
The point of reference you are asking about is the goal of creating a paradox. That was sort of implied but seem to be quite a fan of verbosity. It is, of course, possible that someone – for example – just stumbles upon something that looks like a paradox. The mistake would then be the identification as a paradox.
I’m not really sure what you are trying to tell me with your last point. You started trying to define paradoxes some other way as they are commonly defined. (Quote: “You take as a given that paradoxes are or at least should be ‘un-answerable’.” – thereby implying that according to you definition of “paradox”, the same can have answers.) I wouldn’t have brought definitions up otherwise.
I agree with you that merely defining doesn’t tell you much (maybe nothing) about the nature of reality and said as much. Definitions are for communication, no tool for finding truth. Those tools are the meat of science, not definitions.
There are several ways one could model language, from a top down purely statistical approach that Norvig likes, something in the middle which Chomsky proposes, to a bottom up neural model of chemical interactions. There are advantages and disadvantages to each method for many different reasons.
Yeah, I was just trying to take a step back (and maybe I was too OT, I agree), but at some point we should start asking ourselves more fundamental questions. Anyway, this discussion is way over my head, I'm just glad that HN users think there's an answer for everything, is like Godel or Kant have never written anything in their entire lives.
"God" (in its various versions and revisions around the globe) is just an anthropomorphized version of the previous proposition.
People tend to map a lot of other stuff onto it.
Boy do I have news for you!
Assuming solipsism (however absurd it may be), both me and the universe do exist.
Science doesn't really aim to answer the 'why'-questions, but rather the 'how'-questions. The scientific method boils down to falsifying hypothesis, and it's a lot easier with 'how does the tide work?' than 'why does the tide work (the way it does)?'.
Science can't say anything about 'Why does anything at all exist rather than not exist?', because there is no way to test any of the answers. So it's left to mythology, religion or philosophy to answer.
'How' asks about the current state of things and how they are possible.
'Why' asks about the past, and how the current state of things came to be.
Depending on interpretation of the question, 'why' can be a lot more philosophical than 'how'.
A possible answer to this stems from the anthropic principle. We evolved in a place with a moon because the moon helped us evolve. We don't see no moon because complex life such as us would not have developed without it. A stable rotation and gentle tide are conducive to the evolution of complex organisms; tides were instrumental in getting life out of the seas and onto land.
"Why is the sun the way it is?" can be answered similarly. A smaller star has too small a habitable zone where liquid water can exist. A larger star would have burned out sooner than the 4.5 billion years it took to develop sapient life. A double star has a much smaller set of stable planetary orbits. That the sun is an appropriate star for our life on earth is not divine providence or an enormously unlikely coincidence; it's the result of a universe-wide scenario of statistical multiple endpoints.
totally agreed with you up to that point which I have hard time understanding.
so you say universe is kind of fractal and we happen to be in the right place on that fractal, where all the ingredients come together?
While I disagree with almost everything Chomsky says about everything, and I think it was meant to be somewhat sympathetic, it's really unfair to propose an affinity between Chomsky and O'Reilly in this manner. What the hell.
Equally unfair is Norvig calling Chomsky a mystic for his invocation of Plato. Chomsky is a rationalist, not a mystic.
Some questions are metaphysical, not because they are complex, but because they are ill-posed and not subject to falsifiable experimentation of observation.
BTW, a lot of real-world phenomena, like some large events of history or sociology or macroeconomics, fall into the same category of scientific unapproachability, due to practical limitations of our civilization and any plausible future civilization.
Philosophy does not imply God.
And it drives me nuts when people don't see that many important questions cannot even be expressed within the framework of science.
Take physics, for example. It can tell you why some objects behave the way they do by telling you there are certain particles, interacting forces, etc.. In this way you can explain, say, the photoelectric effect.
But it isn't really an answer to the "why" question, is it? It just pushes the question one level lower. Why are there such and such particles and forces? Why the constants? The very nature of these answers is descriptive. It is a description of how the world works, not why it works that way.
Maybe asking "why" in this ultimate manner is an ill-posed question - but that's not the deal here. It just doesn't seem that science in its current form, unlike religion or philosophy, could ever even attempt to answer it.
Don't get me wrong, I'm strongly atheistic myself, but there are some inherent limitations of scientific exploration and clarification with respect to the answers it can provide.
Theory A: Closed form formula function.
Theory B: "Algorithm". Still a function.
Theory C: Memoized function (constant time!)
According to the article "nobody" likes C, especially the article's Chomsky straw man. If one had a procedure to convert C to A, then this whole issue would become hairsplitting. Such a procedure would aim to convert a memoized function back into a form that uses more symbols from a mathematical language. A good criteria of success would be the description length of the resulting procedure in the preferred language. One reason this could be useful to science is that once you identify a value that is useful in many theories it becomes part of the language. Making it available to the next problem may speed up the search for a "good" description of the next phenomenon. Identical procedures that appeared in various algorithms might acquire a special name. One such value might be called "pi", another "foldr" and so on.
Of course there may be many good descriptions, just as there are many languages. Also, the example could be extended to statistical modeling situations by adding room for error terms in the suitability criteria.
So if, you have a general procedure to convert a table into a definition you can make money and science at the same time!
Before you can figure out why, you have to make sure you can accurately characterize the what. So there's a lot of science that is focused on coming up with a descriptive tool like an adhoc curve, before the underlying principles are discovered.
I think Chomsky is afraid that statistical models will cause people to stop looking for the underlying principles.
Can there be such a thing as a conlang that demonstrates the ideal statistical grammar and semantics? (“All the words in this list are 60% likely to be used as nouns and 40% likely to be used as verbs....” But in the absence of a pre-existing linguistic community, how could you get students of the language to use them in the right proportions?)
http://www.youtube.com/watch?v=XbjVMq0k3uc
(I think it even had the same bee-dance example)
They take the coded sports results, and automatically generate narratives using statistical speech models. They have a startup that is doing it too, don't recall the name of it....
EDIT: I believe this is it: http://narrativescience.com/
http://app.bookmarkerpro.com/changes?fmt=html&id=2573
Quite a few revisions since first posted to HN!
http://mindfulpleasures.blogspot.com/2011/01/noam-chomsky-on...
Why are other metrics Norvig provides like articles published or prevalence in practical applications are considered more intellectual?
And besides, I don't think "accurately modeling the world" is the end of it. Classical Newtonian mechanics correctly describe 99% of our activities in the real world and were considered pinnacle of scientific achievement for several centuries. Yet we know today that they're just a subset of General relativity and Quantum mechanics.