Hacker News Reader

Comment by Der_Einzige | Hacker News Reader

BLEU and ROUGE scores are almost useless for what they are trying to do (measure how good translation or summarization systems are). The author shows a great example of an excellent translation getting a 0 score.

I tried submitting a NLP paper where I explicitly laid out my reasons for avoiding evaluating my system with ROGUE scores and I learned really quickly that despite having a terrible scoring metric, the NLP community would rather reject all papers without ROGUE scores reported rather than admit that there is an incredible lack of methods for automatically evaluating summaries or translations.

How many good translation or summarization ideas are not published or utilized just because they don't get high BLEU scores? I bet it's a lot of them...

View on HN

> BLEU and ROUGE scores are almost useless for what they are trying to do (measure how good translation or summarization systems are). The author shows a great example of an excellent translation getting a 0 score.

The author also (perhaps unintentionally) shows a great example of being unable to tell whether a translation is excellent or awful.

猫坐在垫子上 is not a great correspondence to either reference sentence. Going purely by grammar, the equivalent of "the cat is on the mat" would be 猫在垫子上 (note the missing 坐, which is the verb "to sit", not present in the english references), and "there is a cat on the mat" would be 垫子上有猫 [literally "the top of the mat has a cat"].

猫坐在垫子上 could be a good translation of "the cat is on the mat". But it could also be a good translation of "cats sit on mats". Those two sentences are radically different in English; to judge the translation, you need to be aware of whether e.g. someone just asked "where's the cat?" or "what do cats do?"

The point of BLEU (and ROUGE and METEOR) is to correlate with human grading. It's unusual to find cases where a MT model change increases BLEU but hurts human ratings.

It's not so unusual in my experience. I used to do research in this area and we ran side-by-side human evals with our experiments. Yes, huge jumps in BLEU typically imply better human ratings, but the variance is big. Many times, significant jumps in BLEU had no significant effect on human ratings. ROUGE is even more useless.

You just don't see these things reported in the literature because 1. human evals are a pain to run and can be expensive 2. researchers need to publish or perish.

Maybe that was the case when we were using DUC datasets with several gold standard summaries or translations. It certainly is not the case on classic datasets like CNN/Dailymail where there is only one gold standard summary.

A recent trend has been to use a language model to do the scoring: https://arxiv.org/abs/1904.09675

Computing a perfect metrics is as hard as the task itself, so lf course scores like BLEU and ROUGE use and easy approach instead that can be gamed. Every researcher reading this kind of paper normally understands that.

In this post, the 0 example for a good translation arise because the question is not formulated in the standard English way which use sujet/verb inversion. This problem is easily solved by extending the corpus with more diverse way to formulate questions. It’s also notable than the way a question would be translated in Chinese would also similarly be affected by the construct used by the MT system and the constructs present in the corpus. Instead of ditching the scores completely, improving the corpus seems to be a more productive approach, which would also benefit other researchers and the field in the long term.

Which metric did you use instead? Human evaluation?

Frechet Inception Distance using language model is a solution (commonly used in text GAN). I think the core idea is that you should never rely on one metrics for evaluation but rather a mix of them ( even statistic ones: unique word count, word distribution similarity etc )

"You have a model evaluation problem. You decide to use Frechet Inception Distance. Now you have two model evaluation problems."

Ultimately there are no good automated metrics with regards to summarization in my opinion, human evaluation alone is also no silver bullet since it is very labor intensive and what constitutes a good summary is highly dependent on the domain.

Reply on news.ycombinator.com