Hacker News Reader

Comment by martinkallstrom | Hacker News Reader

Readability has been ported to Python and Php: http://blog.arc90.com/2009/06/20/readability-now-available-i... There is also a C# port here: http://code.google.com/p/nreadability/

View on HN

There is also a ruby port here: https://github.com/iterationlabs/ruby-readability

The beauty of our approach is that we didn't have to port it and we can adapt much quicker when new versions come out.

Ruby-readability is one of my projects. Let me know if you have any questions.

Great, i searched for a readability's ruby port on github but the closest i can find is pismo: https://github.com/peterc/pismo

Pismo used to use ruby-readability (linked above) but I ended up writing my own system. It works similarly to Readability but is better on certain types of poorly formatted content (but worse on others, so YMMV). Pismo is more a general purpose content extraction library than Readability and better suited for machine processing and summarization (which is what I use it for).

Pismo also comes with a command line client built-in, so you can do stuff like this:

  $ pismo http://preona.net/2010/11/ever-wanted-arc90s-readability-as-an-api/ title sentences

  :title: "Ever wanted arc90's Readability as an API?"
  :sentences: Over at Preona we have been wanting something just like that for a while now. So we built it! Some time ago, while developing LazyReadr, we were faced with the fact that RSS feeds simply aren't all that lovely anymore.

Note: "sentences" picks the first few sentences by default, but this is ideal for a summary by an automated system or for a news page :-)

You should check out topicmarks. It does summaries in a very smart way from what I've seen. We'll likely start using it to do automated summaries for RSS content in our iPad app.

Sadly, though, it "takes minutes" (and they even seem to make a big point of that..) It might be useful for slightly better summaries though I've had great luck with going with the first paragraph of an article so far (or certain other metadata if it scores better, like <meta> description).

I recently converted the Python port to use lxml instead of BeautifulSoup.

If you need speed (BeautifulSoup is slow), and not portability (lxml won't work on google app engine), check out http://blog.interstellr.com/readability-in-python-using-lxml

I ported it to Java a while back and was amazed at how a simple heuristic (looking for blocks of text with lots of commas) went a long way. Unfortunately, it has several drawbacks which have prevented my full use of it and came up with an algorithm which turned out to be similar to boilerpipe's (using decision trees) that worked better in some cases. I haven't gone back to this problem, but I have some ideas on ways that it can be improved.

I actually suggested that there be a <usercontent> tag to one of the people on the HTML5 committee because among other things comments sections throw off many of these web scraping algorithms. After 3 emails, he didn't buy my argument, so I gave up. So all web designers, please oh, pretty please, use the up and coming <article> tag correctly. Many data/text miners will thank you dearly!

Reply on news.ycombinator.com