Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by spiderfarmer | Hacker News Reader

Parent Full thread

spiderfarmer·

Just stop scraping. I'll do everything to block you.

View on HN

> in my case, users add their own domains

Seems like they're only scraping websites their clients specifically ask them to

Now you've gamified it :)

It's a pretty easy game to win as the blocker. If you receive too many 404s against pages that don't exist, just ban the IP for a month. Actually got the idea from a hackernews comment too. Also thinking that if you crawl too many pages you should get banned as well.

There's no point in playing tug of war against unethical actors, just ban them and be done with it.

I don't think it's an uncommon opinion to behave this way either, nor are the crawlers users I want to help in any capacity either.

What is the crawler is using a shared IP and you end up blocking legitimate users with the bad actor?

He said "it's pretty easy", probably not realizing there are whole industries on both sides of that cat and mouse game, making it not easy.

If you think the game is played on a single IP address, you are not adept enough to be weighing in on this discussion.

So you're blocking the absolute bottom of the barrel scrapers and feel like you 'won' because you don't even notice any scraper that isn't complete trash.

Then again why block them if they don't cause any issue in the first place? Instead of going ballistic on IPs that you don't vibe with you could also just do proper rate limiting.

Reply on news.ycombinator.com