Hacker News Reader

Comment by grayrest | Hacker News Reader

What's he using to pull out the articles? I had a hacky version set up using the Readability algorithm but never bothered to make it public.

View on HN

Boilerpipe is by far the best tool for this that I've ever found (http://code.google.com/p/boilerpipe/). I'd be interested to hear if he is using something better, but I'd be surprised if he is.

I think this is a great idea and very similar to a lot of stuff I have worked on recently. It's cool to see so much interest in these text-related services.

I don't think it's quite as good as what he's doing though. He has the title and date specifically pulled out and he doesn't have any extra text included. I think he manually handles CNN. If I try a HuffPost feed it doesn't work at all.

Yea, I'd be curious to see exactly what he's doing. I can only guess there is a heuristic which results in a lot of failed feed processing noticed on here (I know it's just a weekend project :)) that doesn't generalize well. Boilerpipe, in my experience, works very well on almost all news/blog type content. Finding the date in the first few sentences and the title are extra heuristics that can be added later.

EDIT: The date and title are in the RSS feed already! No further analysis needed.

Thanks for that link - exactly what I was looking for

btw I know that at Techmeme, Gabe spent years perfecting his story parsing for the 50k+ sites he tracks. Even something that would seem simple such as parsing the date of a story from a webpage has a ridiculous number of permutations that you have to grep for.

That is freaking awesome! http://boilerpipe-web.appspot.com/

That library is very robust. On the company I was working last year, We built a news crawler using that tool, and adapted it as a plugin for nutch.

Goose article extractor has a full suite of unit tests and also does pure text and image extractions: https://github.com/jiminoc/goose

Possible to do it with Readability - my PHP port is here: http://www.keyvan.net/2010/08/php-readability/ - and similar tool (free software, can be self-hosted) using the PHP Readability here: http://fivefilters.org/content-only/

Reply on news.ycombinator.com