Hacker News Reader

Comment by andai | Hacker News Reader

Would this be solved by just providing APIs? Scraping is a pain and people only do it when there's no API. (Though these days a lot of businesses offer unofficial 3rd party APIs by scraping.)

It's also stupid and massively wastes computation and traffic on both ends.

Then people would think "but then I'd have to pay money to offer a free API", but they're already doing that in a more expensive way via the web interface anyway.

View on HN

> people only do it when there's no API

Legitimate scrapers, maybe. Everyone else does it to circumvent the API limitations, by posing as real traffic. APIs imply API keys which can be traced and banned.

And what makes a scraper legitimate?

In the case of GP, I believe they're referring to legitimate intent of the bot owner.

Legitimate scrapers adhere to the websites terms of service, for starters.

Well, that's what happens when you make the API more limited than what people can do with scraping (or with using the public "our own web client" API key).

No, there’s a huge number of bots out there just doing stupid hacky stuff like trying for wp/admin/. Vast numbers of what I’d characterise as script-kiddy type attempts. Looking at the raw logs of a webserver not behind something like CloudFlare really opened my eyes to this. There’s an extraordinary number of garbage requests.

Yea, there was an HN thread complaining about WAF's a few days ago, but damn, if you're on the public internet having some kind of WAF cuts down the bullshit responses by a massive amount.

Same thing goes for email. I had one particular domain I managed spam filtering on and for every 10,000 messages we received, 1 was delivered to users. The amount of crap requests on the internet is insane.

And that's just requests on 80/443. Hook up an IDS to the WAN interface and, you'll see thousands of attempts per day to exploit vulnerabilities across other services too.

My low-traffic personal projects get 90%+ malicious traffic.

> Would this be solved by just providing APIs? Scraping is a pain and people only do it when there's no API.

Many sites intentionally make scraping as painful as possible because they don't want anyone scraping them.

I think a more efficient approach would be to prosecute those who do it - bandwidth costs money - and to implement mechanisms to deter them. A not insignificant number of ai workers think are entitled to using people’s bandwidth, servers and intellectual property for free. Until a couple of such people are taken to task this wont stop.

Reply on news.ycombinator.com