Hacker News Reader

Comment by pluto_modadic | Hacker News Reader

codeberg's one job gets interfered with by freeloaders hammering expensive views (like git blame) inconsiderately.

You’re so right. I have a public-facing Forgejo server. Before configuring Anubis, scrapers were sending it about 600K requests per day. Copying and pasting from my blog post about it:

* For every Git commit, fetch the version of every file in the repository at that commit.

* See git blame for every file at every commit.

* Attempt to download the archive of each repo at every commit.

* Run every possible pull request search filter combination.

* Run every possible issue search filter combination.

* Fetch each of those URLs at random from some residential IP in Brazil that had not ever accessed my server before.

Afterward, it dropped to several hundred. Expect anti-attack features to keep getting stranger and more visible as scraper get still more aggressive.

That's crazy. So 600,000 someones are absolutely starved for data.

Or one someone with too much money and too little sense misplaced a decimal point in their ScraPy setup?

I think it was the latter.

And the cruel irony is that these are FOSS Git repos I'm publicly sharing. I'd've been fine with them cloning the repo and analyzing away to their heart's content. That's not the way their scraper's wired, though.

What do you use to monitor this? I don't really keep a close enough eye on my services to know what the traffic is doing (and haven't had any issues) but maybe I should start

I just tailed the web logs for a bit and saw that it was wild. For fun, I fed 10 minutes of logs into an AI and it picked up a lot of signal I didn't catch at first glance, like clients claiming to be MSIE 7 on Android 3 and such. I added some reject rules to the webserver in front of Forgejo but that only made a dent in the traffic, alas.

This matches my experience of running a public mediawiki server. The bots (mostly Facebook/Meta) will for every single change fetch every single page again, over and over.

I can’t tell if it’s incompetence or malice.

Couldn't they just rate limit them? Are they literally using a new IP for every request?

There are indeed scrapers which use tens of thousands of distinct IPs, and so rate limiting them isn't a solution.

We used to call this snow shoeing.

In my experience, yes. At the peak of a scraper flood I was dealing with, I'd say about 90% of the traffic was from a unique IP. I'd never seen anything quite like that before.

Reply on news.ycombinator.com