TextTeaser – An automatic summarization algorithm

Thanks!

But I did not get what you mean by "copying the link is borked in Firefox". What link are you talking about? :)

I browsed the article and wanted to look at what else you have written but didn't find any references in the URL: http://www.wired.co.uk/news/archive/2013-08/22/filtering-the.... Not even your name. Is that normal for Wired?

Yes, tried it on a couple of my own pages. Seems to work ok.

Who is going to be first to couple this library with an RSS feed reader and mailer so that I can get auto-generated summaries of recently written articles sent to my blackberry?

A comparison: of Summaries of http://www.wired.co.uk/news/archive/2013-08/22/filtering-the...

Stremor's TLDR: Pointless reeling off the numbers. The challenge each of us now faces is a brand new one. Our filters were once the media, our friends, and our families. We were aware of, and understood how our filters operated. EdgeRank isn't something that Facebook users understand. Just 20 tweets out of thousands. We need better filters.

Text Teaser: How do we create a balanced diet of content with so much junk being thrown at us? Now, large media organisations create mountains of content, then track our reading habits and online behaviour in order to build a profile of us. A set of favourite news groups; a list of RSS feeds; a well-curated bookmarks folder; these are all filters we once built ourselves. The less we understand our filters, the more we will come to accept that the world they present us with is true. The more control we have over our filters, the more we can understand what we're not seeing.

Text Teaser goes over 350 Characters which is the Established People can't sue you for stealing it limit... So also weigh that when deciding which you like better.

Are you using LSA?

I require the title because I need it for the algorithm.

You are spot on on the paper that I referenced. As you can see in 4.3 which is in page 3, the paper mentioned two algorithms for sentence selection. These are Summation-Based Selection and Density-Based Selection. Which is SBS and DBS respectively.

This was also the foundation for Summly. The problem is that it is flawed. It doesn't take in to account emotion, or emphasis.

What is the most important sentence in this:

Drakaal is a poopy head. He often posts to hackernews and calls people an idiot. When this happens I get mad.

The Premise is "Drakaal is a poopy head" so that is the most important, but it doesn't inform the user. What he does is the most telling of the sentences, but with the "He" as the first word you can't actually make sense of the content with out the prior sentence. The last sentence is the least important for the understanding of the content.

It is important to know that sentence number 2 is the most informative, but that to figure out what it means requires Sentence 1.

When computing the results of a summary you have to weigh sentence dependencies, density of information, amount of emotion expressed and number of characters available to you.

And Keywords aren't enough, you need noun entities and the ability to tell the relationships of words so that you know "Cars, Trucks, and Automobiles" are all the same concept in many contexts.

You are right, I'm not taking account of HTML tags. It is because I extract the text beforehand using Pythoon Goose. In that sense, only the text will be feed in the algorithm without any HTML tags.

Hahaha. It sucks. The algo is not meant for this kind of websites. Try out news articles! :)

You are right. I think Scala is a good language and handle functional programming well. But the code is too abstracted that even me might not get what it is doing. I just placed as a reminder for me. And also for everyone else to easily get what that piece of code is doing.

that patch of code could be restructured a bit to make it more readable, here are a couple of small suggestions that jump out at me:

1. perhaps `.reduceLeft(_ + _)` could be replaced with the use of a `sum` function or method (assuming one exists in scala?)

2. if the `topKeywords` collection returned a default value with a `.score` of 0 when queried with a key it doesnt contain, the headOption getOrElse null match null would not be necessary.

e.g. in python it might look something like this:

    Keyword = namedtuple('Keyword ', ['score', ...etc...])

    top_keywords = defaultdict(lambda : Keyword(score=0, ...etc...))

    def sbs(words):
        if words:
            return (1.0/len(words)) * sum(top_keywords[w].score for w in words)
        else:
            return 0.0

(apologies for making superficial comments about the code. the algorithm itself certainly seems interesting)

Is this a well known text summarization tool? I hadn't heard of it before this post.

Hi! I will still retain the API in Mashape. That is for the developers that do not want the hassle to deploy it in their own servers. On the other hand, the open source code is for devs to check out the algo, hopefully improve and contribute to TextTeaser. If they want to use it and deploy it on their own, they are free to do so. :)

Think MongoHQ for MongoDB.

Hey, you can contact mojojolo in Mashape through the Contact Now button at the bottom of this page https://www.mashape.com/mojojolo/textteaser#!pricing

He can set up a limited free private API for you to test. Let me know if you have questions about this process - chris@mashape.com

The corpusEN.bin file is the training data provided by OpenNLP which I used to split sentences (http://opennlp.sourceforge.net/models-1.5/). It's not the training data used for summarization.

It's mostly statistical (simple scoring). But as you can see in this lines of code: https://github.com/MojoJolo/textteaser/blob/master/src/main/... I keep track of the keywords used by the blog and category before. Through it, TextTeaser employs a little bit of machine learning to improve the quality of the results.

Just checked it in Github. It's 100% Scala. :)

Why use Python instead of Scala?

I'm seeing good things about Python and NLTK. But back when I develop the core algo of TextTeaser (few months ago), I still don't know Python.

Right now, the TextTeaser website is coded using Python and Flask.

It shouldn't matter what language the OP decided to use, as long as it allows him/her to do whatever he/she set out to do in the first place.

Why use Python when Scala is a good alternative?

you can either rewrite it in python, or use unirest/requests to summarize text using the API.

Didn't manage to remove it.

I created a news reader for Philippine news (http://www.readborg.com/) using TextTeaser. The word Philippine appears most of the time and I decided to make it as a stop word. Forgot to remove it in the stop words.

This made me laugh. And $15m is big enough for me.