A Gentle Introduction to Text Summarization in Machine Learning | Hacker News Reader

A Gentle Introduction to Text Summarization in Machine Learning

blog.floydhub.com

325 pointsReDeiPirati28 comments

A large part of the effort in text summarization is in the quality of the stemmer. If you're working with English, you're golden, there are several high-quality stemmers available. However, if you're working in a language that doesn't have a stemmer yet, it's a colossal task to write one.

Yes, definitely. I have also found sentence segmentation to be important. I recommend using Stanford CoreNLP for sentence segmentation if they have a pre-trained model for your language. In my experience, their pre-trained models are the best and NLTK is a good fallback. I think OpenNLP is the easiest to train a new model if CoreNLP and NLTK don't work.

For training, I wonder if NER can be abused to produce a good sentence segmentation model. My idea is label the first word of every sentence as NEW_SENT and every other word as nothing. Then the model would learn which words start a sentence in a document. I haven't tried it nor know if anyone else has, but I keep meaning to try.

What about pictorial languages like Mandarin and Japanese?

Not using spaces to separate words of course makes dealing with those languages a bit harder. The same problem applies to a lesser degree to languages like German, where compound words are fused together.

In those cases, the quality will depend heavily on the performance of the tokenizer as well. For Chinese you don't need a stemmer, since it's an analytic language, but Japanese is agglutinative and German is synthetic, so stemming is required for those.

Japanese isn't pictorial ; it uses two alphabets, one more for native words, one for foreign words, and then substitutes in Chinese symbols as shortcuts when widely known.

I think Chinese is really not a pictorial language either; using your apt description, it's like if every character was a "shortcut". Harder to learn, but still fairly standard in conveying meaning as far as modern languages go. Some characters are still visually similar to what they represent, but at this point they're mostly a bit of a stretch.

A true pictorial language would convey most meaning through the symbols themselves, and I don't think any modern languages fulfill that definition. Maybe sign languages are the closest thing we have to pictorial language, in terms of the way some things are expressed symbolically?

Right so under the hood do you normalize to romajii? How do you deal with kunyomi vs onyomi - do they even matter? Even if you had a stream of kana, how do you tokenize when they're not delimited?

(don't know much about NLP beyond surface level software experience but have an amateur interest in Japanese)

Interesting topic and well written .. I kicked the tires a few times with newspaper3k https://github.com/codelucas/newspaper/ it has nlp and summary methods that work pretty well, I think I'll peek under the hood to see how it's being done there. Curious to see if your method is an improvement, if so, hey they're both in python!

Why waste time say lot word when few word do trick?

"Brevity takes effort": "If I had more time, I would have written a shorter letter."

This is great stuff. One question -- what about applying the role of the word of the sentence into the weighting, along with the frequency. As in general Subject, Verb and DOs are going to be MUCH more important than, say, adverbs and articles. Is that not done because it's harder to automate vs. just frequency?

I wish there were also tools like this that did paraphrasing. The article mentions the possibility, using deep learning, but doesn't go into any details; and I haven't seen anything that can summarize below the sentence level anywhere else either.

Can I ask what you mean by summarize below the sentence level? Thanks!

See the subsection titled "Abstraction-based summarization" in the article. Basically, instead of copying important sentences verbatim (and spending all your analysis on choosing those sentences), rewrite entire paragraphs into new and unique sentences. No more plagiarism, because nothing is a direct copy.

That's interesting. One thing about plagiarism is that derivative works are considered plagiarism, even if the direct language is no longer used. https://www.copyright.gov/circs/circ14.pdf A court would have to rule, of course, but I wonder if this would actually get around that. That said, as US caselaw currently stands summaries are completely legal to a certain length -- you can't stop someone from summarizing what exists, it's like sharing a fact -- but you can't create a derivative work. So as long as the resulting output was short enough you shouldn't have to worry anyhow about plagiarism, since summary is covered, but if it's longer, derivative could be an issue. Anyway, thank you -- very interesting!

> So as long as the resulting output was short enough you shouldn't have to worry anyhow about plagiarism

Ya, that's what I was trying to get at when I mentioned summarizing paragraphs into sentences. I think we're on the same page there. :)

Ah, gotcha! :-)

Abstractive summarization not yet working (like text generation in chatbots), extractive is ok and better done with RNN/BERT/something else neural network based, that's all you need to know.

This is very similar to how TLDR-bot on reddit summarizes reddit posts or linked articles: https://smmry.com/about

The results can be surprisingly good, even for such a basic algorithm.

This is a very basic introduction to the topic. I don't understand why it is getting so much traction here. Am I missing something?

Might be basic for you, but I know very little about the field. I found it very informative. If you don’t have something constructive, why post at all?

This article is brilliant, thank you!

A gentle introduction to text summarization using a python NLP library.

Am I the only one who hates reading grammatically incorrect pieces of text?

For me this technology is still in the "unusable" phase, and urgently needs more work.

Reply on news.ycombinator.com