Hacker News Reader

Comment by laktek | Hacker News Reader

I found a lot of use cases for web scraping is kinda ad-hoc and usually, occurs as part of another task (eg. a research project or enhancing a record). I ended up releasing a simple hosted API service called Page.REST (https://page.rest) for people who would like to save that extra dev effort and infrastructure cost.

View on HN

I agree, and I find scripting a web browser via the developer console a really productive approach.

First, it's completely interactive.

Second, it's the browser, so absolutely everything works. It doesn't matter if the data you want is only loaded by an obscure JS function when a hidden form is submitted on a button click. Just find the button, .click() it, and wait for a mutation event.

I have a write up on this[1], but I need to extend it with some more advanced examples.

[1] https://github.com/jawj/web-scraping-for-researchers

That may be fine for javascript heavy websites for a site with a few pages, but for anything with more than say 1,000 pages it is much more efficient to scrape using requests with lxml. The requests can be made concurrently, are scalable and there is no browser overhead with page rendering.

I've done a lot of scraping in my day, and I've found that lxml/requests is 2-3 OOM more resource efficient than a Selenium based browser. That JS/rendering engine is HEAVY!

You can even control Chrome remotely via Python with a pretty simple web sockets api:

https://github.com/marty90/PyChromeDevTools

That's really the best of both worlds.

Reply on news.ycombinator.com