Hacker News Reader

Comment by malandrew | Hacker News Reader

You can use PhantomJS for automated scraping. You have two options:

(1) Design your scraper is usually via a 1-to-1 correspondence with the routing logic and the client side templates. Create a new scraping module for each of the templates and use the scraping modules according to the data visible at each route.

(2) Another simpler approach is just to design your scraper to hi-jack the app's own XHR or Sockets module and collect the data directly via the API exposed to the web-app.

The latter approach is the really smart way to scrape client-side web-apps since you can get a lot of additional valuable metadata that may not be written to the screen.

View on HN

For people who don't have much experience in web scraping, it's important to note that there is no a better/worse approach. Everything depends on the specific site.

I don't see (2) as a simpler approach that scraping from the browser rendered page: the information retrieved between the browser and the server can be obfuscated and will require too much effort to analyze the protocol between them. You mention the Sockets module but if you are using HTTPS request the information at that level will be encrypted.

I really encourage people to explore the Data Big Bang site since offers a lot of different approaches for web scraping. Just look at the resources at the end of the http://blog.databigbang.com/web-scraping-ajax-and-javascript... article.

Why use Google Chrome (or Firefox)? One of the reasons is that you are sure that it is updated very often and their performance is great.

I wasn't suggested sniffing traffic. With WebSockets, it's very hard to do in an automated way. I was suggesting looking at the Models and Collections in a web app via de-obfuscation and a little reverse engineering. Injecting JavaScript into a page via PhantomJS is a great way to control a web app.

Interesting. Do you have an example of the latter approach? I have been trying to build a web scraper, but many times after the HTML is returned, the javascript then loads the data dynamically. I'd love to learn of a few techniques to fetch the AJAX data.

Get firebug, or use developer tools in chrome and look at the requests the browser is making when you load the page, and when you scroll down and it loads more data. Then you just implement that in whatever language you are scraping with. I usually use pythons requests module.. That's about as easy as it gets for something like this.

Not an example I can share, but you can approach this starting out looking at the global variable which basic bootstrap data is assigned to on initial payload. From this variable you should be able to work backwards to the function that reads this data on page ready and instantiates some models and collections. From there, I'd study the model/collection code to see how they are configured for fetching data. If you tap in at this point, you can just request data directly via new models or collections.

If the site use jQuery, maybe you can hijack Ajax requests by using [jQuery.ajaxPrefilter](http://api.jquery.com/jQuery.ajaxPrefilter/).

(2) is also much less resource intensive as well. Who knows how much JS on may end up executing on Twitter's homepage.

Benchmark your personal time developing a Twitter web scraper similar to this one to the time to follow the (2) approach. The code of the article was done in a few hours while the (2) approach would require further analysis and risks (analyzing protocols can be very tricky). Google Chrome and Firefox are very fast and one developer day may cost more than one PC.

Well at the end of the day, the application really matters. In most cases I find just sniffing the XHR requests with Chrome's Developer Tools to be the most productive.

Reply on news.ycombinator.com