Comment by MojoJolo | Hacker News Reader

Comment by MojoJolo | Hacker News Reader

Hello. I recently finished my thesis for my MS CS degree. My thesis is about automatic summarization. It undergoes research, defense, and I think its result is good enough for me. It uses statistical approach and machine learning. My main issue about it is not the summarization part, but the text extraction part. I can't seem to extract article in a web page well enough. I'm using boilerpipe (https://code.google.com/p/boilerpipe/) for it. It can do most tricks, but it's not that good for me. May I ask how you extract the main article in the page?

Here's a preview of mine (http://www.textteaser.com/ui/article?link=http%3A%2F%2Fwww.p...). Go to its home page to read more news. It caters Philippine news and will soon enters alpha stage. I'm planning to open up the API or open source it. HN, which is better? The API is ready, registration is the only thing that it lacks.

You can try the API here: http://api.textteaser.com/api/?url=http://www.theverge.com/2...

Just replace the url parameter with the URL of what you want to summarize. Some URLs are not tested yet, and may produce errors. :)

Hi. I study CS with an inclination towards ML but I don't know anything about the topic of automated summarizing. I'm curious, since you've taken a ML approach, did you still need to rely on NLP and if so, was this very problematic? Also, do you perhaps know an article or a paper that could serve as a good starting point/overview of what approaches there are to summarizing and what are the current difficulties. Thanks

Hi, I recommend this following research: http://www.cs.cmu.edu/~nasmith/LS2/das-martins.07.pdf http://www.aclweb.org/anthology-new/W/W03/W03-1204.pdf

I still have other papers, but those can be a good starting point.

In my thesis, NLP is done via statistical approach. It learns from it's previous summaries does have somewhat learning. I don't see any problems combining NLP with Machine Learning. Can you elaborate on this?

Thanks, these look like exactly what I was looking for.

Re NLP, I meant to say NLP from sentiment analysis perspective (not sure if that's the right way to put it). So my question was whether you had to extract meaning of one or few related sentences or only process the text in a statistical manner (which you answered).

Look up text rank. Essentially it amounts to representing text as a graph with sentences as nodes and some sort of similarity measure as edge weights. You then run page rank on that graph.

Interesting. Thanks! :)

You should try diffbot. They use a vision based method to extract text from webpages. The tool looks pretty polished and seems to work rather well.

Yeah. I tried it. It's good! The problem is it's not free. Although it's cheap, I still don't have that money/revenue to pay for it. The free tier is not that much. If I have the money, I'll definitely change it to Diffbot. :)

Shoot them an email, they seem nice enough about academic/nonprofit work.

Reply on news.ycombinator.com