Long-Form Question Answering | Hacker News Reader

Long-Form Question Answering

ai.facebook.com

33 pointsstablemap9 comments

I'm not sure "we scraped ELI5"[1] is really such a substantive advancement of the state-of-the-art that it deserves such a large write up. The Stanford Question & Answer Dataset is much more carefully curated.[2]

ROGUE[3] and BLEU[4] are pretty meaningful metrics for translations and for fairly short answers that that really only be phrased one way. For example, "What is the biggest mammal?" should be answered "The Blue Whale." There is little room for ambiguity: the words "Blue" and "Whale" must appear, as must the bigram "Blue Whale" for the answer to be correct. For a large or complex answer, the situation is different. Metrics based on word overlap like ROGUE and BLEU must either incentive memorizing the answer given in the training set (overfitting) or the inappropriately penalize semantically equivalent answers. For example, for the question "why is the sky blue?" if the algorithm produces "The sky is blue because Raleigh scattering off of water droplets preferentially scatters blue light at right angles. This is also why sunsets are red." and the answer on file is "light with long wavelengths passes straight through moist air, while light with short wavelengths tends to be deflected." Both answers are correct - indeed they are basically the same answer - yet they share so few words, bigrams, and trigrams that they would have to marked "wrong."

[1]: https://www.reddit.com/r/explainlikeimfive/

[2]: https://rajpurkar.github.io/SQuAD-explorer/

[3]: https://en.wikipedia.org/wiki/ROUGE_(metric)

[4]: https://en.wikipedia.org/wiki/BLEU

Am I weird for immediately jumping to conclusion that the first thing this will be used for is generating tons of rather low-quality content for "SEO" purposes?

It's interesting to see the path towards increasingly sophisticated question in --> answer out. The way people do this is with dialogue, but that doesn't fit as easily into standard supervised learning with an easy-to-collect dataset.

If you asked me "What's a good restaurant nearby?" I wouldn't answer with a list of restaurants, I'd say "What kind of food do you feel like?" Seems like we aren't even working our way in that direction. Maybe the sample complexity of RL and language modeling needs to come down a ton first.

I'm having the dickens of a time understanding who Facebook built this for. Marketers, right?

Content farms. Maybe they have a vested interest in making all search results completely suck?

did you bother to read the article? It's in the very first line "To help advance question answering (QA) and create smarter assistants" -- aka messenger bots / AI assistants / etc

Yes. In fact I did, yet I'm a mere passive user of FB and have never come across QA messenger bots on the platform before, who uses them? I presumed marketers and advertisers.

Is this presumption incorrect? I'm trying to understand who a potential/target/ideal user of this technology would be from Facebook's perspective-a few sentences about mobile users searching for songs but that's it. That doesn't seem answered in the article, I'd appreciate that in the place of snarky assumptions about my reading skills, please and thank you.

Messenger bots are of interest to marketers, but they don't really seem to use them too frequently. Customer support is a much more common use case from my observations.

All that said, even if it were marketers, it seems pretty clear who this is being built for.

Facebook has already launched some features that are clearly related to this, such as the Recommendations product. Imagine posting "anyone know why this thing happened to me?" and being able to answer it in real time?

Writing it out, that example actually seems counter productive to Facebook's vested interest increasing sharing, although I guess if the answer resolves as a comment it aligns incentives?

Reply on news.ycombinator.com