Hacker News Reader

Comment by d_burfoot | Hacker News Reader

This paper is a good illustration of what I see as a critical shortcoming of modern NLP work: it is all about math and algorithms, there is nothing about actual language.

The paper has lots of information about: neural network architectures, parameter update equations, learning rules, and inference algorithms.

There is nothing about: part of speech categories, relative clauses, morphology, affixes or compound words, the theta criterion, the content/function distinction, verb tenses, agreement, or anything else related to actual linguistic phenomena.

To me this seems a bit like the Aristotelian thinkers who tried to reason about physics based on pure mathematical analysis, without any empirical work.

View on HN

Fred Jelinek famously quipped: "Every time I fire a linguist, the performance of the system goes up." Here are two hypotheses that could explain this observation:

1) Linguistic knowledge and theory is not useful for NLP work; instead, people should rely entirely on machine learning techniques.

2) Linguistic knowledge is useful in principle, but the field of linguistics has not yet obtained a sufficiently high-quality theory, and it is better to rely on ML than on low-quality theory.

Almost everyone in NLP adopted hypothesis #1; I believe hypothesis #2 is true.

The linguistic patterns that you mentioned (parts of speech, etc.) are too simplistic to describe language's idiosyncrasy and they ignore language acquisition.

Machine learning and data-driven methods require models with millions of parameters. Linguistic theories have traditionally not used such models.

> it is better to rely on ML than on low-quality theory.

Isn't ML (and especially deep learning i.e. neural networks) itself a low-quality theory? From what I understand, nobody really knows why it works so well but uses it anyway.

In this context, higher-quality == produces results == allows automatic translation with reasonable fidelity.

But I agree, it is a not a very explanitory theory since it is too black-box.

I'd extend hypothesis #2. We might not have a sufficiently high-quality theory, but we might also not know how to integrate the benefits of the theory we have with the benefits of data-driven approaches to NLP like those described in this paper.

It's clear to me that data-driven techniques are an essential part of the answer to the problem of language understanding. It's infeasible to design a complete set of rules for understanding natural language with all its exceptions and fuzzy relationships. I'd point to Wittgenstein for a stronger philosophical argument on this.

That's not to say that theory has no place in language understanding, but it's not immediately obvious how to combine rule-oriented theory with data-driven ML.

There has been work in this area though, and actually several important, state-of-the-art-moving contributions have been inspired by linguistics. My favorite are recursive neural nets, which benefit from the linguistic insight that language has recursive structure. This paper[1], for example, describes a system that combines a dependency parser with a recurrent neural net (gross oversimplification) to yield a state-of-the-art sentence understanding model.

[1] - http://arxiv.org/abs/1603.06021

Do you have any further detail or reference to Wittgenstein here? Curious to learn more

Not OP, but Wittgenstein covers this in his Philosophical Investigations. In the Tractatus, he tried to create an all-encompassing system for language, but in PI he more or less says it's impossible. It's an amazing read in any case, it has totally changed the way I approach philosophy.

Ok, thanks!

> which benefit from the linguistic insight that language has recursive structure.

Really? You needed an entire _field_, and not just, and by just I mean, in an, but not too, extended, sense, really, trivial observation, to get to that conclusion?

I love your work on #2, it's a really interesting avenue, but I wonder if both approaches are close to the point where reasoning about the sentence and its meaning becomes necessary anyway if we want to go beyond the current state of the art.

Almost everyone in NLP adopted hypothesis #1; I believe hypothesis #2 is true.

"Superior ML performance as a pain signal to be carefully listened to", as it were. I'm in camp #2, also.

> There is nothing about: part of speech categories, relative clauses, morphology, affixes or compound words, the theta criterion, the content/function distinction, verb tenses, agreement, or anything else related to actual linguistic phenomena.

When you learned you native tongue, you didn't need to know all of these. You just learned. So, maybe the problem IS about math and algorithms instead of linguistics.

You certainly didn't need to know the names for them. That doesn't necessarily disqualify them from a role in the learning of language.

That said, you still may be right. But at the moment, it's a question of dogma, not empiricism.

Who's to say that neural nets aren't implicitly learning these concepts as they train? In fact, I'd be very surprised if nothing about the networks' internal state corresponded to linguists' models of how language work.

Nothing about focusing on the math and not the language means that these concepts don't have a role in how the models learn language. Just as nobody is told about parts of speech as they're acquiring their native tongue, we don't necessarily need to explicitly tell machine learning models about parts of speech in order for them to learn how to use that aspect of language correctly.

"When you learned you native tongue, you didn't need to know all of these. You just learned."

Learning your native language is not at all the same task as translating between languages. People who do translation are usually quite knowledgable in questions of grammar. This is particularly true of people who pick up languages later in life.

Even if you know multiple languages "natively", it's often difficult to translate accurately without thinking about grammar. Or, for that matter, to speak/write your own language with any degree of competency -- we study grammar in grade school for a reason.

> we study grammar in grade school for a reason

Parsey McParseface is probably better at grammar than 99% of people.

Christopher Manning, a somewhat famous NLP researcher before and during the deep learning craze, has a great perspective on this that echoes your statements which I recommend reading:

http://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00239

We tried solving this by analysing language for decades; the reason we're doing this now is because it actually works a lot better.

The article directly suggests that isn't the case. There's a (debatable) improvement over Google's existing phrase-based system on these benchmarks, but the whole paper is about the huge effort they've had to put in to get past the shortcomings of the method.

A more realistic assessment is that state-of-the-art neural translation is maybe getting as good as phrase-based systems.

They claim 60% fewer translation errors than the phrase based system. That's not nothing.

Also, see this graph: https://1.bp.blogspot.com/-jOLa-LdidQU/V-qV2oJn1aI/AAAAAAAAB...

On French > English and English > Spanish it's almost indistinguishable from human translations. On the rest it still gets greater than half the distance between the previous best method and human quality translations. This is a massive step forward.

"This is a massive step forward."

Take it easy. The graph you're linking to is for a limited survey of 500 isolated sentences, and it's a comparison of mean scores without confidence intervals. The authors themselves admit that it has serious problems as an evaluative tool:

"Note that we have observed that human raters, even though fluent in both languages, do not necessarily fully understand each randomly sampled sentence sufficiently and hence cannot necessarily generate the best possible translation or rate a given translation accurately. Also note that, although the scale for the scores goes from 0 (complete nonsense) to 6 (perfect translation) the human translations get an imperfect score of only around 5 in Table 10, which shows possible ambiguities in the translations and also possibly non-calibrated raters and translators with a varying level of proficiency. Testing our GNMT system on particularly difficult translation cases and longer inputs than just single sentences is the subject of future work."

When you look at the other evaluations in the paper, it's clear that it's about as good as the current best phrase-based tools. In other words, this may be a massive step forward for neural nets, but it puts them at about par for the field.

I don't see any problem with this evaluation. Both human translators and the machine were subject to the same conditions and evaluators, making it possible to do an objective comparison. 500 samples is more than enough to get an accurate score.

If phrase based was really so good, it should have gotten approximately equal scores on this simple task. Instead in every case it did way worse than the neural translations.

Unless you think Google's existing phrase based system is significantly behind the state of the art (possible, but I doubt it's too far behind), 60% better is a pretty great improvement.

I only skimmed the paper briefly, but it seems like they don't depart significantly from the NMT literature, but they do need to tweak a few things to get their results.

Like I said, it's debatable. It was honestly the weakest part of the paper. The specific claim comes from a vaguely specified "human evaluation" (i.e. how many evaluators? based on what criterion? did you just ask folks at google to compare stuff?) on a tiny sample of 500 basic sentences, and the results were presented as differences between mean scores without so much as a confidence interval or a power calculation to back them.

Moreover, from the paper:

"In some cases human and GNMT translations are nearly indistinguishable on the relatively simplistic and isolated sentences sampled from Wikipedia and news articles for this experiment. Note that we have observed that human raters, even though fluent in both languages, do not necessarily fully understand each randomly sampled sentence sufficiently and hence cannot necessarily generate the best possible translation or rate a given translation accurately. Also note that, although the scale for the scores goes from 0 (complete nonsense) to 6 (perfect translation) the human translations get an imperfect score of only around 5 in Table 10, which shows possible ambiguities in the translations and also possibly non-calibrated raters and translators with a varying level of proficiency. Testing our GNMT system on particularly difficult translation cases and longer inputs than just single sentences is the subject of future work."

Basically, you should ignore the claims of improvement, and replace them with a big question mark.

I think you are overestimating human performance here.

The weakness in the evaluation system is mostly because humans are imperfect translators.

That is why there are metrics like inter rater reliability and significance tests and confidence intervals, which seem to be all missing in paper...

One of my favorite college courses was a translation course.

One of the key points repeatedly hammered home (I took the class twice) was that languages do more to constrain what must be communicated than what can't be communicated.

Consider gender in Romance languages vs. English as a simple example.

Additionally, each language gets it subtleties in different ways. With English, it's vocabulary by and large. Supine, lying down, flat on his back, prone, recumbent ...

Even with my own poor language skills in Russian, I can say that the subtleties are different. You must be more precise with your verb tenses, for instance, but in exchange you can communicate quite a lot more about the status of an action. Getting that into English can be unwieldy: prochital = something like "we totally and completely talked it over." Of course, that's horseshit as a translation, and here we have the problem of rating translations.

There will always be quibbles from translators because the languages have slippage between them. I don't think "perfect" translation exists for anything but the simplest of sentences in rare circumstances.

Another awesome example: "i-e" (pronounced "ee-ay") in Japanese = "No" in English. Except not really, because in Japanese it means something more like "contrary to your expectations". In English "no" can be the affirmative when we ask about a negative. "You didn't go did you?" "No."

In Japanese that would be "Hai" as far as I understand it.

As a nerdy sidenote, even worse, English does shit like turning double positives negative but only sometimes.

"That's a good essay isn't it?" "Yeah, Yeah."

> "That's a good essay isn't it?" "Yeah, Yeah."

That's sarcasm, which is not a grammatical construct but a cultural(?) one. "Yeah, yeah" is a positive statement (grammatically, a double positive never turns into a negative in English), but a statement said sarcastically implies that you meant the opposite of what you said.

Japanese also has something like "Yes, this is difficult" that is really a polite way to say "no".

Google translate is quite good, and uses KNN. Does this beat KNN?

Didn't work well for Chinese though, or Asian languages in general.

This reminds me of the DeepMind bit where the AI beat human players despite not knowing about the edifice of built up knowledge in that domain. I'd expect that an AI could be a great chess player with enough gameplay info and not knowing particular strength scoring, types of famous defenses, etc. In some ways the way that humans have constructed these relationships may actually make learning more difficult. We certainly don't teach 3-year olds about theta criterion, we just speak in 'correct' language and may correct them when they make mistakes.

That's the best part of this style of machine translation: the code knows nothing about parts of speech or other categories that linguists try to force-fit language into.

Instead, it learns the structure of both languages from training data (sentence pairs in source and target languages).

> There is nothing about: part of speech categories, relative clauses, ...

Because it's not the best level of abstraction. Human generated features don't capture all the nuances. In vision there used to be heavy engineering and math involved in feature creation. Now it's all neural nets end to end. Same thing happened to language.

Because this isn't linguistics work. It's AI work. We are bypassing the impossibly hard problem of natural language processing. Instead we build AIs that can do the hard work for us, and much faster and better than any human linguist ever could.

Faster yes, but better remains to be seen.

I have my doubts on whether machine translation that isn't based on language understanding can ever be trusted not to make egregious mistakes.

It's better than previous machine translation systems, I mean. Sure humans can translate better, but they can't describe how in a useful way. Linguists have spent decades trying to come up with hand designed language models, input knowledge about the world in computer readable formats, etc. But this simple AI model can learn to do a better job in a week from raw data.

My hunch is that the more apt analogy is to Skinnerian Behaviorists, or apostles of the Hilbert Program. Deep learning is a very a powerful tool, and the current zeal for it has yielded rapid progress in previously stagnating fields. That said, we have yet to see what its limits are in this space; it's still too young to know. It may be that, like in so many cases beforehand, synthesis is the path forward, and there are no silver bullets.

Maybe it's your expectation that anything interesting in linguistics needs the language of part of speech categories, relative clauses, morphology ... Maybe that assumption needs to be updated? Maybe the language of network architectures, parameter update equations, learning rules ... is a better way of talking about language?

Given the successes that Google's machine translation has had, and the lack of success of traditional linguisitics, it might appear to be the case that this paper is directly about language.

It's not a better way of talking about language: it's a better way of talking about how to build a computationally useful model of language with features primarily extracted from data than building language explicitly in is. Pretty much by definition.

Google's paper isn't talking about language -- it's talking about how to tune a generalized computational device to build a computationally useful model from a large corpus of data. Of course it sounds different than talking about language, per se.

There is a problem that arises in this case that isn't present in the analogy you gave. The physics they were using pure mathematical analysis was logical. Language, and how humans always change the way they use language is largely illogical.

You're right in saying that we can't really get anywhere trying to analyse language piece by piece. But pure mathematical analysis is not getting us great results either.

Linguistics fails to come up with a competitive alternative that leads us to better real application, that is why. In a data driven world, that is nothing to complain about except themselves.

> There is nothing about: part of speech categories, relative clauses, morphology

This is an interesting and deep question. I believe that there are multiple ways to tell the story of a phenomenon. Using POS categories or verb tenses helps us to organize and learn about the medium. It is important for humans to know what a verb tense - in theory and practice - is, to be able to form not just grammatical sentences, but also ones, that express things clearer.

However, there is no rule that says that these concepts must be learned first before you can do anything else. In a sense, I see the (neural) machines taking a step back.

Whereas earlier attempts at language modeling took into account a lot of linguistic concepts, maybe even tailored to a specific human language, the more end-to-end approaches ignore these things and get better results. What can this tell you about the concepts? They are certainly useful and have been established by a scientific community. The machine, as it reads not just tens or hundreds of sentences but billions, creates a mathematical model, that happens to have aspects, that have very useful real world sides to them. A machine that learns a language model - or that learns translation, can teach us new things about words, and ultimately also about the world. I believe this is a beautiful aspect of the current neural renaissance.

----

Kanerva in a preface to Geometry and Meaning writes:

> We think of some sciences, by their very nature, as being more mathematical than others. We even refer to physics and other heavily mathematical sciences as "hard sciences," as if to imply that social sciences and humanities are somehow soft or easy. Another way to see this picture, however, is that mathematics itself is hard and that it would be applied first to sciences that themselves are relatively easy. [...] It matters greatly that the mathematics be appropriate. [...] Mathematics is a tool for coping with complexity. [...] This brings us to the substance of [this] book: the exploration of mathematics that would be appropriate for describing concepts and meaning. The branch of mathematics that has traditionally been associated with meaning is logic. Logic is also discussed here, yet the main emphasis is on identifying mathematical spaces that would lend to modeling of meaning. (http://www.puttypeg.net/book/chapters/foreword.html)

I guess there are two issues here:

- Most of the available texts have some form of "errors" or are not grammatically complete

- Machine translation does not care about grammar, it cares about probabilities

NLP with grammar knowledge is feature engineering basically. It is useful when you're "shallow learning" but not so much when you're deep learning

Google's recent language modelling paper achieves state of the art performance on exactly these tasks, and it's based on a somewhat vaguely similar architecture.

It's becoming pretty clear that many of these concepts may not be fundamental 'things' but just things that make it easy to talk about.

Actually it seems a little like the opposite. Their software is actually translating language quite well in the real empirical world, while the the programs built with language rules were never able to do nearly as well.

Why do think any of those labels are critically important? They are entirely derived from human desire to understand, process, and define a taxonomy for language. There is absolutely no reason to think that they are inherently necessary to achieve the goal or high fidelity translation (or anything else in this domain). Similarly they might be meaningful but inferred naturally through the learning process.

If your goal is to map machine learning processes to established dogma, this makes sense; if it is just to build an effective learning machine that solves an extant problem, then it's not.

No seriously, this is exactly empirical work. It works somewhat at translating texts.

On the other hand, attempts that tried to use linguistic categories failed.

A nice new development is multilevel deep networks with explanatory states built in. It will tell you why it detected the thing it did. It may of course be lying or wrong. :)

https://en.wikipedia.org/wiki/Laputa

Care to elaborate?

Peter Norvig explains why.

http://norvig.com/chomsky.html

Reply on news.ycombinator.com