Hacker News Reader

Comment by sho_hn | Hacker News Reader

This is only tangentally related to the article, but on scraping HTML in general: If you're a Python user, use lxml for it. I know most content on the web will tell you to use BeautifulSoup, and lxml is something you've only heard of in connection with reading and writing XML, but lxml actually has a lovely HTML sub-package, it's faster than BeautifulSoup, and it's compatible with Python 3. I've gotten lots of good mileage out of it (and no, I'm not a developer on it :)).

View on HN

FYI, BeautifulSoup is actually dropping its own parser entirely for the next version, in favour of being a wrapper around lxml/html5lib/html.parser http://www.crummy.com/2012/02/02/0

Ah, interesting, thanks. I will confess that I haven't followed the latest round of developments in BeautifulSoup - I abandoned it last year when I made the switch to Python 3, and the previous effort to port BS to Python 3 had stalled/failed at that point (looks like they're back on track on that one now, too). Then I found I didn't really miss BS while using lxml.html - not sure I see a point in putting it on top of it.

I think the idea behind BS is that it saves some typing and thinking for common scraping tasks. In that light, it fits perfectly on top of some other (faster) parsing library.

Yup, but all of the stuff in the Quick Start section of http://www.crummy.com/software/BeautifulSoup/bs4/doc/ has more or less close equivalents in lxml.html.

That being said, I think it's great that he's still maintaining BS, and porting it to Python 3 in particular - it keeps existing code working and will allow more people to make the switch. And a responsible and committed maintainer is good advertising for a package in itself, of course.

I admit to using BeautifulSoup, lxml, scrapy, nokogiri and mechanize. But these days I just stick with PhantomJS. Just load the DOM with WebKit. http://phantomjs.org/

I'm even lazier. :) I used the pjscrape wrapper around phantomjs.

https://github.com/nrabinowitz/pjscrape

Another tangentally related recommendation: Use Scrapy if you need something more than one-shot script using BeautifulSoup or lxml.

http://scrapy.org/

I use HTMLUnit for scraping javascript based sites. Since Python is a lovely language and HTMLUnit is written in Java I use Jython. You can take a look at one of my articles on the subject: http://blog.databigbang.com/web-scraping-ajax-and-javascript...

I need to build / find a tool for parsing EDGAR filings for their financial statements tables (not so bad) and parsing those financial tables into usable information (pretty bad, the tables have very bad and often inconsistent HTML layouts).

Any suggestions to where I should be looking? Python? XSLT?

Contact me, email is in my HN profile.

Use HTML Tidy + XSLT.

Reply on news.ycombinator.com