Hacker News Reader

Comment by blantonl | Hacker News Reader

Unpopular opinion around here, but scraping Web pages is often unethical, and the scourge of many Web developers and infrastructure managers.

For Christ sakes people, develop your own products!

If you have to develop and use a cloud based tool to "get around" businesses blocking your scraping app, maybe reevaluate your business and life choices.

View on HN

Sure, let me just develop my own monopoly on all real estate information.

no business is under any obligation to provide you access to their data and services for you to build a business on top of.

If you feel that the owner of data or services maintains an illegal monopoly over that data, work to lobby government to resolve that issue.

It's not your individual place to unilaterally decide that a business owes you anything other than what is described in their terms and conditions of use. Period.

Or if you don't want info public, don't make it public then complain people are using it. If you want to stop scrapers, offer an api. Nobody scrapes by choice if there's a supported alternative to getting the same data.

> no business is under any obligation to provide you access to their data and services for you to build a business on top of.

I mean, they're the ones who gave it to me, I'm just using it. Not my problem if they don't like it. I never saw any terms and conditions when my browser loaded the page, no reason I'm going to go looking for them before I scrape it. I have just as much right to access it as my web browser does or my phone does.

I'm not saying they owe me anything, they're offering it to me, I'm just taking it.

I'm not saying they owe me anything, they're offering it to me, I'm just taking it.

No you are not. If you are deploying proxy containered services on cloud computing hosts orchestrated by yourself to circumvent mechanisms to prevent such behavior you are delibertly and unethically acting in a manner directly not intended by the owner of such data.

How so? They're still offering it to me, just on a different IP. I fail to see how that becomes unethical. I'm accessing information they publish to the public at large.

I used to have a similar attitude - most typically applied to rules & laws, etc. “If you don’t like the rule, go run for office and get the law changed.”

But then I realized that’s just a cop out. A lot of structures and systems are in place specifically to make it difficult for people to change things and specifically to maintain monopolies.

Nobody’s going to lobby government to resolve this issue. That’s the point. Real estate companies are super happy to keep the status quo.

Sometimes you have to break some rules to innovate. There is a line of course (where you draw it depends on your own ethical code). But I certainly wouldn’t put “scraping real estate data” behind the line.

In the analog age, imagine someone made a card index of some interesting data you can come in a shop to consult, for a fee.

Would breaking into their premises to steal or copy the cards be on the same side of the line as the digital variant?

Scraping a website in your analog analogy would be a store where those index cards would be plastered to the shop’s windows, visible from the outside. The “scraper” would come by every day and manually copy (as in, write in their own notebook) what was on those index cards that are visible from the street.

So the question is: would you consider that stealing?

It's more like you going to the public library to borrow every single book they have. Thereby forcing the library staff to handle all your requests. You would probably not expect to be allowed to do this in real life.

Good point! There is an increased pressure on a public service. Hadn’t thought about that parameter.

Except its aprivate service.

Stop trying to lawyer this through analigies people, think about the actual situation at hand, instead of drawing broken parallels to some hypothetical

Users are also victims: captcha walls are basically an anti-scrapper countermeasure, which everybody pays for in reduced web usability.

"Proof of work" style countermeasures (where data has to be decoded browser-side in expensive ways that are bearable for a regular user but onerous for a mass scraper) is another externality everybody pays for (the total amount of CPU wasted by regular users is at the system level a total waste).

Most use cases for web scraping is usually related to data analysis, creating machine learning dataset, etc. Google and other search engine services are scraping the web too.

Reply on news.ycombinator.com