Hacker News Reader

Comment by michaelrm | Hacker News Reader

:) We explored using the API unfortunately at the volume of data being pulled down, the API is not feasible due to the number of authorized API keys needed.

View on HN

And you don't see this as unethical?

This is a huge topic of debate, but the consensus from "experts" seems to be that if it's public data that's not behind some auth, then it's fair. By the way, this is what web crawlers, like Google's crawler, does.

See: https://www.quora.com/Is-website-scraping-legal-and-ethical

But I don't necessarily want to go down this rabbit hole here, as it detracts from the interesting technical issues outlined in the article. If you haven't read the entire article, I recommend that you do, because they had to overcome many interesting technical challenges while working on this.

A little web scraping is fine, if and only if the website lacks a canonical way to do so (i.e. an API). Ignoring the API and scraping hard enough that you have to write an article assessing scalability of the scraping is bad.

It's weird that you're dictating rules around scraping not outlined in Github's TOS.

The article actually mostly talks about how they scaled working with the accumulated data, and only a little about scaling the scraper.

Just like the interesting part about Google is how they process and index the data and not their crawler, the most interesting part of this article actually not the scraping, but how they handled all the data and processed it.

Edit: I found Github's policy on scraping. I hope the link brings some closure to this concern: https://help.github.com/articles/github-terms-of-service-dra...

A bit more context may be helpful. While scraping scalability was a concern for us (for latency reasons), it was only significant to retrieve historical data as a one-time job, after which we throttled back. A latter segment talks about a priority queue object (SRRPQQ) which we use in lieu of scaling further.

Reply on news.ycombinator.com