Hacker News Reader

Comment by MatthewPhillips | Hacker News Reader

I'm confused, is the window object from Jsdom a live object? Can I scrape interactive sites?

Isn't Phantomjs already perfect for scrapping, what is the advantage of this exactly?

Phantomjs has a lot larger overhead (it runs a full browser). Plus it doesn't give you access to the full Node.js ecosystem (e.g. access to databases, etc). You can use Node's Phantomjs driver, but it spawns a child process to do the work, and seems a little complicated in how all the interactions work.

And of course you can scrape interactive sites - interactive sites still basically just use HTTP to request data. Just watch the network window in Chrome's developer tools, and figure out what HTTP requests the site is making that you are interested in. Then code them into request() calls.

Phantom doesn't need access to node apis. Separation of concerns. Run your phantom script to scrape and then pipe the results to your node script for processing. I've worked on 2 scraping projects using this method and it works great.

Sure, it can work great if your needs are simple. But if you're posting data to forms which you need from a database, and you don't know what you might need from the DB until runtime, it can get a bit more complex.

Also worth noting that jsdom is quite slow, and has a strict HTML parser. If you want something faster that will cope with more web pages, look up cheerio.

Remember that Phantom just runs a web browser. So writing a Phantom script is like writing an app on top of another app. This means you can use XMLHttpRequest or WebSockets to communicate with a back-end if that's necessary.

Yes that's what we ended up doing with Kapow. Just felt... dirty :)

Kapow? You mean this company: (http://kapowsoftware.com/solutions/content-migration/index.p...)?, mind to share about your experience?

By default jsdom doesn't run JavaScript files unless they're specified within the program. However, if you call 'jsdom.jsdom' it will then run the scripts specified in the webpage, and those you specify in your program.

But will the pages scripts respond to events? For example on Twitter after the page loads it looks for a tweet in the hash bang and if there is one, does an XMLHttpRequest, then updates the DOM. Will Jsdom be able to scrape sites like that?

"curl" can scrape sites like that. Just watch the Chrome "Network" window and see what requests get made. Then use that. Scraping doesn't need a browser.

Of course curl can. And curl was the best option 15, even 10 years ago. But to scrape a complicated site with curl (or any other non-browser scrapping utility) you wind up basically rewriting the application you're attempting to scrape.

Browsers sole purpose is to run web apps. It doesn't make sense to use anything else.

At the low end of the scale I agree. But once you start scaling it's very difficult to run more than around 40 concurrent sessions with a browser-based scraper.

With a pure Nodejs scraper we can run over 1000 parallel sessions per CPU.

Reply on news.ycombinator.com