Hacker News Reader

Comment by TrevorFSmith | Hacker News Reader

Is it ethical to scrape when a site has explicitly blocked bots? I know a fair number of people who run small sites who are already considering closing them down because the bots are relentlessly hammering their sites and driving up hosting costs.

View on HN

Use cases range from sending 100s of requests per second just to bring a website down to doing a montly request to a municipality's endpoint to get a local dashboard of when trash is picked up. I don't think you can pass a single judgement of automating web requests in general.

> Is it ethical to scrape when a site has explicitly blocked bots

Ethical? ppl get jailed in China for this. "Breaching computer systems" is a felony.

I know for a fact companies in China do this. One paid me directly for work in this space, and in another, I found my own OSS project in their artifact cache while working there.

I've only ever gotten in trouble in the UK

I suppose it's like trying to enforce no picture rule in public.

I think it really depends on how the bot is run. If the bot is replacing me navigating there manually, absolutely. If it’s sucking up content to rip off and make someone else billions of dollars, no.

Why do you feel entitled to navigate to a website you neither own nor pay for via a method the owner expressly forbids?

Why do you feel entitled to dictate what user agent I use, if it’s well-behaved?

Why do your opinions about what a "well-behaved" agent is override the wishes of a site's owner?

Site owners should not have full control over what users do. Their wishes do not apply past certain boundaries.

For a non-AI example I run into occasionally, any wish to stop time-shifting should not be relevant to what I choose to do as a media consumer.

There actually is a well defined demarcation point where the site owners wishes do apply. It's called... A demarc point! Everything that happens past this point is by definition entirely in the control of the admin. Which includes closing the door on whoever they wish. This is a technical reality, not something you can effect with opinions or wanting things more than other people want other things. It might seem contradictory but this is actually the main thing that makes the internet as free as it is.

When we're talking about ethics, I think we should give the owner more than that. If they don't want to be DDoSed we shouldn't say "too bad, the attackers are outside your network, your wishes don't apply, get good".

A request is a request. None of your business what I do with it after that.

Accessibility, saving time, personal preference. Any number of reasonable things that don’t put undue stress on the host

This page is only viewable in Internet Explorer 5.5. Please switch to a supported browser.

If it's just impersonating me to help me better consume the content as if I were the one driving, then it's perfectly ethical, and not even related to a bot ban. Shades of gray and all...

Because I should be able to choose whatever user agent I want. The owner gets to "forbid" things on his computer, not on mine.

It's most definitely unethical.

Reply on news.ycombinator.com