The Ethics of Web Scraping

scrapeops.io

3 points·Ian_Kerins··3 comments

What does one get for actually being nice and being an "ethical web scraper"?

I don't see many people bothering to be honest, developers who are out there scraping are mostly just doing a job, they want to be done as quick as possible and most don't care about things like this unless there are repercussions

It is this type of attitude that is why websites are becoming so aggressive in blocking web scrapers.

Being an "ethical web scraper" is about your own ethics, not abusing other peoples servers/data, and preserving the open internet for everyone.

Yes, if you slam someones website you will probably get the data you want but it will damage it for the collective. If everyone just slammed websites then:

1) the website owners will just get pissed off and either shut data behind logins or make it inaccessible. This is basically what happened with LinkedIn.

2) make it much harder and costlier to scrape by using more advanced anti-bots, requiring everyone to use more expensive residential proxies and headless browsers to get past them.

Web scraping can be a burden for websites, so everyone should approach it in as responsible and ethical way as possible.

I doubt many website owners feel they have an ethical responsibility to provide public APIs to their data so people wouldn't scrape their content. It is up to them what they want to do with it.

I just feel like the whole thing is a chicken and egg problem - both sides could be nicer to the other but no one wants to go first!

I do like the idea of identifying web scrapers in the user agent but I'm not sure how many websites would use it...

Reply on news.ycombinator.com