Creepy Crawlies
people.kernel.org
people.kernel.org
Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources.
Currently I trick bad scrapers into a fake infinite black hole path with the promise of tasty data, then serve images to them one byte at a time over 15 minutes (after sending the header quickly), bloat the responses to cost them tokens, and randomly return AI generated images of sexy toasters. I have an admin dashboard with a little leaderboard for which ones get the most stuffed, and it keeps my heart warm on these wet autumn evenings.
The only punishment would be unique text that trains their models to be degenerate. And even then you'd probably have to serve across many domains.
The altered versions would be the page or paragraphs of it, with sentences and words swapped arbitrarily so that training on the data adds as many iffy token sequences as correct ones.
It'll just punish the bad actors running the scrapers. As the original poster mentions that they are using TVs as proxies to get residential IPs, one really can't think of these bots as criminal enterprises.
Sadly, if the bad actors has two cents for brain, they'll limit how much importance each domain name can have on training data. To mitigate impact of bad data like this.
(Note I'd suggest only linking to them from robot.txt as pages to not be indexed, that way no human or well behaved not ever will see them, which is kind of the point).
Exactly. I can't hope to keep them all at bay, but I can at least have the petty little victory of making their visit less convenient than it might otherwise be.
> if the bad actors has two cents for brain
I suspect that a majority of them are little better than the script kiddies of yore, running tools with minimal understanding of what is actually going on.
> I'd suggest only linking to them from robot.txt as pages to not be indexed
Agreed. Blocking all bots from all pages, well those that bother to listen to robots.txt. All bots because pretty much all of them are scraping for AI and similar these days, even googlebot. If I want people to see my stuff they'll get a link, and maybe they'll pass it on further, but all indexers/trainers can get stuffed. I'll likely make an exception for archive.org and similar.
You can't tease us like that without showing screenshots.
https://ibb.co/mCgXPZPg https://ibb.co/qFxVRQGW https://ibb.co/9m6xV1xg
I only just added the token wasting measure so I'm really curious to see what happens if another bot loads over 400 pages.
No hot toaster pics though :( on second thought perhaps better in our imaginations
Elixir's really fun to do this in because the BEAM will let you have hundreds of thousands of processes sat around doing nothing quite happily, so slow IO and such is something it's weirdly well-suited to.
This is on my photography site so I'm less fussed about them harvesting my writing, and more about causing problems for image harvesters. I'm sure they'll get some stuff anyway but at least one bot got stuck in a 400 page hellhole earlier today so I'm overall very pleased :D
I also explicitly tell all crawlers not to index the images on my site, actually, so I feel even better about serving the ones that do complete junk.
Not sure why I get more crawler traffic than I do legitimate traffic tbh - it's a relatively new website that I've been trying to sort out the SEO for as I'm trying to get photography work, so it's probably started cropping up on the radar of scrapers looking for training data.
I’ve also toyed with this approach and have a similar labyrinth on my website, but it just serves back huge amounts of meaningless text one word at a time.
I do want to add some more gibberish generation to mine. Once a visitor hits the honeypot route it does replace the rest of the site content with nonsense but I think it'd be fun to have it spewing really daft, really long LLM-generated essays on toasters.
https://firesphere.dev/articles/iocaine-the-deadliest-poison...
---
Gymnastics trophy two years late, not two years wrote down on the other side of the two thousand millions. Stop all the time was now a.
Anything of this real London, these actual civilized men and women; in uniform batches. The whole incident could not control the beating of a telescreen. Folly, folly, folly! He.
---
Around June/July we got a 10x boost out of nowhere, and it started affecting performance for users, increased hosting costs, and random bursts would bring the website down.
We spent some time trying out solutions, from Cloudflare and Anubis to AWS, but it ended up affecting real users, and we got complaints on Reddit from day one, pretty much. At the same time, it got a bit expensive having a WAF.
In the end the only solution that worked was blocking every single Chinese IP.
Traffic instantly got down to 10x less, same as it was before. Same conversion percentage as before.
Meta bots are also problematic, as they seem to be quite buggy. But the volume was nowhere near the hundreds of bots from China :/
However the article mentions that now they are being scraped 4-5 requests at a time from random IPs that are then never reused. It still probably possible to detect scrapers if they start by requesting obscure pages, but in general it's a much more difficult case.
I think the post underestimates just how little thought and effort is put into these bots. I also run a cgit instance with far less interesting projects, and am not spared from the deluge of HTTP requests.
The explanation I could come up with is that they try to crawl all links regardless of how much sense it makes or how much load it causes. cgit being cgit, this means billions of links for all combinations of parameters and hashes. That, or it's a deliberate DDoS attack.
- Respects robots.txt
- Detects if something is likely a git forge and as a result
- Clone the git repository instead of trying to crawl the forge
None of this needs to be happening, and yet it does at an unprecedented scale.
The only thing that ended up helping was disabling access for anonymous users.
> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my iPhone 17 to solve at ~100KH/s, making the site unusable. So I spent ~10 minutes vibe coding a safari extension with a native bridge to an optimized C kernel using ARM SHA256H* instructions that can do 200+ MH/s on the same device. This solves Anubis difficulty level 6 in a handful of milliseconds.
Given the numbers and capabilities involved (a single $5K ASIC miner yields 200TH/s, a million times more hash rate than my optimized kernel running on an iPhone), I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won.
Edit: I encourage you to try this yourself. Here's a sample prompt that ought to one-shot the task:
> Build an iOS Safari Web Extension that accelerates Anubis proof-of-work using a native C ARM64 SHA-256 kernel. Precompute the invariant 128-byte challenge prefix, search fixed-width decimal nonces with ARM SHA-2 intrinsics and two worker threads, and target difficulty-6 solves under one second. Relay challenges from a Safari content script through the background service worker to native code, then submit the valid nonce/hash through Anubis’s normal pass-challenge endpoint. Include a deterministic benchmark app, correctness tests against CryptoKit, bounded execution, and fallback to Anubis’s stock solver.Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a site.
a $5000 ASIC-based device can hash SHA256 at 200TH/s or more.
I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle...
My earlier idea was to imagine that each day Anubis picks an entirely different problem-class. Ex: one day it is Sha256, the next it is prime factorization, the next it is twin-prime-finding, the next it is cracking elliptic curves, the next it is some kind of sorting / information theory problem...
All with the goal of adapting constantly so that scrapers have a harder time optimizing for the PoW problem (i.e. with Sha256 ASICs)
Nobody is using ASICs on Anubis.
This is something I'm working on in a prototype. Stay tuned!
You say “they” as if all scrapers are a monolithic group with the same constraints and goals. Part of the problem is the massive diversity.
If the source IP was an issue, you could do it other ways: for example, make the cookie rotate on every access, and insist there is a single stream of accesses.
And yet many report it works, at least for now, and the excess load due to scraping activity falls precipitously when Anubis or similar solutions are used. Maybe once this sort of challenge is used almost everywhere we'll see concerted attempts to get around it, but for now it is easier for them to just move on to another target.
> a $5000 ASIC-based device can hash SHA256 at 200TH/s or more
Peanuts for the big players, but many (almost all?) running smaller scale scraping operations are going to find $5000 rather prohibitive, and they are unlikely to be able to integrate it as they are probably running a “stock” scraper that they didn't write themselves.
Not sure why anyone would characterize these scrapers im general as all being fly-by-night operations that don’t have two cents to scrape together.
If they are so stupid an idea that they could never work, why do they seem to having the desired effect?
If you've seen anyone post a comparison of crawl rate versus difficulty, I'd love to see it. There's probably some difference but I want to know how much of the overall effect it is.
An even stupider approach would work exactly as well or better, like a form saying “type the letter y in this box to continue”. The only benefit is as a road bump that makes the site in any way custom. The moment anyone you are defending against so much as looks at the mechanics of solving the challenge it completely falls apart.
In that case I have read too deeply between the lines/tone. Sorry for the somewhat accusatory assumption of your opinion's details.
That's exactly not what happens with scrapers. Scrapers make millions of requests, yes. But each one of those requests produces an average (token-denominated) reward V'. There's no asymmetry at all: you're just adding a fixed cost to each reward, for both users and abusers.
That's before you even get to questions like "does the system actually work, or does a single solve get practically unbounded rewards".
Which: I get! If it's your site, perform all the anger you want.
There is: users solve the PoW once, scrappers solve it every time they rotate IP addresses. Some of them change their IP address on every request, so they keep solving the PoW over and over.
Anubis doesn't really help against this.
For reference, the challenge on lists.ffmpeg.org takes 8 seconds on Firefox on my three year old laptop CPU that has worse benchmark scores than the iPhone 17 (tbf, the laptop also cost less than an iPhone 17). 8 seconds doesn't run against thermal limitations, so I really don't see why Safari on a modern iPhone should be so slow at this
> I really don't see why Safari on a modern iPhone should be so slow at thisThey're not trying to engage in an arms race, they're trying to channel a racing river into its natural course.
I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and scalable path (either through something like PoW arms races, or through fear of litigation). Otherwise they will continue to ignore instructions and intrude on the human path.
(Also, blocking a whole country is likely not what you do, but you probably know that).
Oh, and it also only takes a few minutes to reconfigure everything and invalidate those years of legal and investigatory work.
This would likely work as "Cloudflare SideChannel", a (hypothetical) Cloudflare product that would let scrapers download the pages that humans actually visit, as they are added to the CF cache. It wouldn't work for the non-Cloudflare part of the internet where humans connect directly to the servers that have their content.
Developers and compute are interchangeable now.
It's more likely that they're trying to scrape the entire internet, and cgit is just unfortunate enough to have autogenerated pages, where each page seems to contain text entirely distinct from its neighbors. This probably bypasses the scrapers' heuristics, which normally prevent them from getting trapped in a maze of twisty little autogenerated pages, all alike.
The way you normally deal with datasets at that scale is to move postprocessing (particularly expensive postprocessing) as late in the pipeline as possible.
Depending on the configuration, Anubis will supply a token after the challenge that bypasses the challenge for a time.
So any scraper that retains basic cookies will be able to bypass the challenge for a number of page views.
A user who needs to load a single page and a bot that wants to scrape a number of pages may pay the same cost.
The amortized per-view cost is highest for the real user.
Yes this needs to be tailored for Anubis, but that was the point: the moment it will become popular enough, scrapers will have no problems to bypass it.
A legitimate user solves it once, shows legitimate-use behavior, and retains their access for thousands of requests. A crawler solves a challenge, is determined to use bot-like behavior after a dozen requests, gets an IP ban, and needs to recycle their IP and start from scratch.
It is indeed worse for fly-by users, but with a website like Linux cgit those basically don't exist. It would be a completely different story for, say, someone's personal blog.
> so I really don't see why Safari on a modern iPhone should be so slow at this
FWIW, my iPhone 12 Mini also does ~110KH/s with Anubis on lists.ffmpeg.org, so seems fairly likely that Safari somehow here isn't working as expected.
The GPU cost to pretrain on that page once will dwarf by 2-3 OOMs the CPU cost to compute Anubis, scrape and post process it. And you’re not going to just train on it once!
At best you’re creating a speed bump for wannabe players scraping with no real plan. The folks training models people use just do not care.
Exactly, and that's even before accounting for more sophisticated PoW algorithms such as the one used by XMR that almost entirely eliminate any benefit from offloading to an accelerator.
It's like trying to keep billionaires out of a restaurant by charging high prices, but still keeping it affordable for the middle class, it doesn't work on a fundamental level.
The people trying to do the scrapping will always have more compute than your typical user.
If someone adds an Anubis solver to their spider, but then also makes it respect cache tags and stops crawling random old commits through a git-web interface, that's probably still mission accomplished.
When the reward to abuse crosses a certain threshold, a determined party will come along and completely dominate it.
Anubis is just "solve hashes to get access to a valuable resources", and if the resource is at least valuable to whoever you're trying to deter as it is to legitimate users, you're not deterring anyone (and by extension, if it's more valuable to them, you're deterring your legitimate users instead).
But it would just make Anubis suck a little bit less
Anubis "protection" model is wrong/broken on several levels. I suppose it does not even have ability to bypass hashing screen when server load is low.
Anubis is not a load optimizer. It's sole purpose is to deny the bots you don't want on your site.
It's on the same category as Iocaine and Go-Away, even if what they do is different in some cases.
This is fundamentally impossible.
In the end only pay per crawl will work. Like x402 but maybe something similar.
In other words: not only client has to spend some kind of resource for access, but it has to be a transaction, server must get something from serving the request.
No, this very article will tell you that the purpose of Anubis is to reduce load on the site. The purpose of denying bot requests is that you're strained by the load they impose.
Conversely, bots that pose no problems to you aren't "the bots you don't want on your site".
Sure, the cheap lock I added to the outside shed's door-latch is terrible and can be shimmed open in two seconds with a bit of flat metal... but the important thing right now is that it keeps the raccoons out. If there's a new threat to the gardening tools or the bikes, then I have a different problem and I'll consider something else.
You can see the implementation here: https://gitlab.com/cryptsetup/cryptsetup/-/blob/main/lib/cry...
In any case, it is GRUB which behaves slowly during boot. While I haven't tested Argon2, I believe it is affected in the same way as PBKDF2.
You yourself describe the solution.
Extensions or browser integrations of the Anubis task.
The problem here is not merely that Anubis is inconvenient to humans. It’s that and also that it’s not very effective for blocking bots. Anything that makes it easier for humans to get past will also make it easier for bots to get past.
Or… the site will serve a random seed and the device must compute 4gb of pseudo-random data, then supply a value at a random server-demanded offset.
Edit: this is already in discussion for Anubis: https://github.com/TecharoHQ/anubis/issues/94
At this point isn't it basically auth-only?
Rant: (genuinely wondering too, and RFC, request for conversation) at this point don't we have Google, etc. basically doing Real World ID Verification, but without an open protocol backing, using it to corral users into their ecosystem and gather data, and leaving us without some open and distributed alternative like the (fairly well working it seems to me) Public Key infrastructure? (PKI)
I know Sam Altman (with Proof of Human or w/e) and others have suggested and also have their own alternative (which I think is basically what Google already does), but I think this should simply be built based on an open protocol and then hopefully not-too-large (and with a healthy competition) local companies could do it for you.
Fixing a malware infection should be like fixing up anything else broken around your house (clean things up, re-auth yourself with the ID verifier, done).
Also, bots cannot simply automate renewing the credentials of course, after expiry you should be required to physically (or in a trustworthy digital medium) re-authenticate yourself preventing any kind of mass attack, you'd need millions of people to perform authentication multiple times a day (by that point it seems safe to assume they'd rather fix the malware source in their personal devices).
[text field][submit]
(type qwerty and submit)
I don't see much connection between that problem and these controls.
But also I didn't say it should hack when I prompt it. For the purpose of this discussion that could go either way. To avoid the griping it just needs to write the code I want when I prompt it.
All in the name of safety, of course.
Lots of work has been done to do just that by monero’s proof of work algorithm. They achieved ASIC and GPU resistance by making you generate entire programs (with quite a bit of RAM usage) and finding one that produces a desired output. Then you submit the seed used for the program (a series of programs actually but let’s not get into the details)
I might try to tackle it again when I can give it some more attention.
Anubis wants "slow for bots but fast for humans" which is not possible if somebody has a farm of devices that costs them basically zero.
Opportunity costs are real.
Combine with heuristics like in the article, so you can crank up the difficulty only for likely bot traffic, and you've got a bot supported site.
[1] https://krebsonsecurity.com/2018/03/who-and-what-is-coinhive...
That's not true. The goal is to raise the cost for bulk scrapers to the extent possible while minimizing any inconvenience to regular users. It's not about being slow for bots but rather reducing the asymmetry of the problem (from the perspective of the server) by imposing a (minor) cost on anyone requesting a page.
I know having the Anubis javascript at all gets in the way of some bots. But is the calculation helping enough?
It's complicated. I think anubis currently only collects low hanging fruit so to speak but in practice that's the bulk of what was causing problems so it's good enough.
PoW in general absolutely yes - I think monero clearly demonstrates this. Last I checked it remained viable to mine monero on at least somewhat low end and at least somewhat outdated consumer CPUs. It seems to me that fact implies that a well constructed PoW scheme can in practice squash the impact of any performance differences to within at most a couple orders of magnitude.
And even if you can guarantee the same CPU time on bots and legitimate requests, you still have a bunch of asymmetry in favor of the bots.
But it's also true that charging anything is going to be expected to reduce server load, that the cheaper something is the more carelessly people will tend to make use of it, and that at least in general bot operators are expected to be more cost sensitive than regular visitors on account of scale.
To that last point consider that a regular visitor will generally have an idle CPU and multiple gigabytes of free RAM. Meanwhile a competent scraper will generally be optimizing request parallelism per cheap VPS while an incompetent one will just point and shoot (and proceed to blindly visit every cgit url). The latter is by far the most troublesome and conveniently also the most likely to be impacted by even a poorly implemented PoW scheme.
I keep seeing comments along the lines of anubis only working because it bounces bots running lightweight setups without js but stop and ask yourself why are they running such lightweight setups to begin with?
It's easier and still gives access to almost everything. I don't think CPU limitations are the main issue there. As far as computer resources, I'd bet RAM is the bottleneck to running the script, not cycles. That suggests a RAM-heavy hash might do better than SHA256, but still agrees with the idea that the PoW in Anubis is not the important part.
> To that last point consider that a regular visitor will generally have an idle CPU and multiple gigabytes of free RAM. Meanwhile a competent scraper will generally be optimizing request parallelism per cheap VPS while an incompetent one will just point and shoot (and proceed to blindly visit every cgit url). The latter is by far the most troublesome and conveniently also the most likely to be impacted by even a poorly implemented PoW scheme.
I considered that. But while a visitor is idle they need the result quickly and they face the full brunt of the challenge all at once. A scraper can be working on a thousand sites at once so the per-request impact of the challenges is a lot lower, and patiently working through challenges in the background usually reduces the real cost, potentially to almost zero.
And you'd better not need multiple gigabytes of RAM to visit your site, that'll freeze my computer.
the only feasible CAPTCHA at this moment is probably hCaptcha. they began to offer animated challenges. it's probably the best you can do, using an LLM to solve them would be a bit expensive for now.
As the article mentions: Anubis blocked virtually all bots for a couple of months, and even now it is still blocking 67% of requests. Without Anubis they'd have to double the number of CPU cores, solely to serve crawler traffic!
It's an arms race. The best solution is the one you can plug in right now - even if it is stupid. hCaptcha might have a better blocking rate at the moment, but it's also $0.99/1K evals, and that "animated challenge" will be impossible to solve for visually-impaired users: for many people that cost is significantly higher than spinning up an extra server or two to feed the crawlers.
Edit: after some more analysis; Average with extension disabled is 90 seconds, with extension enabled it's .3 seconds
Do you have an underpowered TV box that would need hours to solve difficulty 6? Do you also have a Windows box where somebody installed gta6_leak_full.iso? Great, have the TV send their challenges to the Windows box for solving.
If you're operating at scale, you probably have hyper-optimized Anubis kernels, E.G. compiled for the particular set of SIMD extensions that a given machine has. At that scale, getting 1% more juice out of your machine pool becomes worth the squeeze. In the worst case scenario, you can probably get some cloud boxes on AWS to do the challenge solving for you.
It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and display options. It never counted the Google bot.
Over the last few years this counter went from 100-200 users online to thousands. I have been very hands-off with it for many years, doing minor upgrades and backups. However, the site also has gotten quite slow, these sessions were obviously impacting it. So, I finally investigated these crawlers, and yes, it turns out it’s an insane amount of traffic that is entirely artificial, the site has just a handful of real users, and thousands of these crawling sessions that actually try to do everything they can, click every button. It doesn’t help that sort and search were implemented using GET links.
I fixed the counter to exclude the crawlers, but I have a bit of a dilemma. I don’t want to stop the bots from updating their knowledge based on all the content.
The best solution I could find is the new CloudFlare feature where they might charge the crawlers for every request, or otherwise block them. I think that’s a fantastic idea for the internet, at large. I signed up for the beta access, but haven’t heard from them again. I do think it’s unfortunate that this requires CloudFlare and the middleman.
Overall, it seems like the LLM are really straining the internet economy, the openness of it. Email spam used to be the worst, but the organized trillionaire labs sucking up the entire internet is going to break something if we don’t preempt them better.
It’s too bad the copyright and public internet systems are not acting quick enough. And I think there is no reason to act like this race really has to be at such a breakneck speed.
I would promote this idea to all of my competitors. Nah mate, you don't have to ask her out right now. You can wait until next week ;)
Anyway, silver linings, looks like we're finally going to get widely adopted infra for microtransactions.
https://web.archive.org/web/20030202042510/http://www.openp2...
Are the AI companies worried about anything but their valuations?
I don’t care about their valuations, but I do care about the risks that they are creating for the economy, society, and the technological advancement, at large.
Micro transactions [in this case] are a great idea, these crawlers need to be taxed and made to pay for the unaccounted external costs. Furthermore, we need to fund any risk mitigations, security coordination, and other safety work. If these things are not imposed on the market, the most ruthless player will create the biggest mess they can get away with.
I don’t see a better way to do this, right now, than to allow Strips, or CF, to create some way to charge the crawlers. Imagine I have some really expensive content, I can put it behind the paywall, or subscription model, but that’s very specific to the human consumer. What if I do want to search engine or content aggregator to consume some of the content? It would be quite hard for me to sign an exclusive agreement with OpenAI, like what Reddit or NYT are doing. It might be quite beneficial to create some infrastructure to let bots automatically to pay for access. It might even be crypto based, I don’t think it really matters. It just has to be easy for me to integrate it, and the bots need to find it easy to use.
If we all start doing this, I think this could be of benefit to all content creators, such as musicians, movie makers, game developers, and even independent journalists.
We used to pay for software and things, but what has happened? Walled gardens and production monopolies are taking over. This has largely reduced the quality of all the content we consume.
Anyway, what’s wrong with making 2 cents? :)
I don’t think this will make the internet less open, I think the internet is already pretty much less open. Who blogs these days for free? They rely on medium or substack, and have no choice, they submit to these platforms rules about the articles they write, even though they’re pretty generous.
Now, if the reward system was as open as say, Bitcoin, I think there would be a lot more independent and different bloggers and journalists, and independent platforms.
>I don’t care about their valuations, but I do care about the risks that they are creating for the economy, society, and the technological advancement, at large.
Doesn't the same thing apply to every company? And also every government? And also individual?
We're all doing the "locally rational" thing, and globally doing... well...
I think the main factor is time horizon.
Locally rational behavior begins to look globally rational when you add time horizon.
For example, if I'm doing what's best for my family for the next five generations, that starts to look a lot like what is good for the whole country and the whole world.
Companies, individuals, and governments have different roles. Rationality from their perspective is specific to their perspective, and I don’t believe just because evolution or physics are rational we should die off from a new virus or asteroid impact. That would be perfectly rational, but I don’t care about the global rationality of the universe.
What is best for my family and other families is not what is best for Sam Altman or Sataya, or Elon Mask.
We’re just talking about market players. What I’m saying is that these market forces need an outlet via some new infrastructure and regulation. Maybe it’s not micro-payments, maybe it’s some other new ideas.
I just don’t want the next century to be the century of East India AI sticking me in a Matrix pod, rationally.
What is its objective function?
they have one of the worst bot detection in the industry. but because everyone uses them, their huge false positive numbers won't show up anywhere.
It makes sense doesn't it? If you own all the pipes, you can effectively block bad actors. But in reality what they do is to just block everyone who blocks advertising tracking and profiling tech (such as canvas fingerprint etc)... and because all vendors use cf, they all block those same clients, and they think there's some winner takes all magic happening with finely tuned blocklists. Don't fall for the marketing!
Try it yourself. Block those fingerprinting tech and try to access their own user forum. you get blocked. Now allow fingerprinting (even if emulated in a headless chrome!) and access by the most toxic vpn you can think off, access granted.
All cloudflare does is act like a gatekeep for monetizable-by-creepy-advertising-networks traffic, not bots. If the bots pay the small headless chrome tax, you get bombarded with their requests just fine.
So you can definitely block by IP, by fingerprint, both or neither
why do you feel like discussing features was relevant?
And Cloudflare is no exception, the "hello world" of bots is Cloudflare bypass.
And I only stand to gain from pay-to-crawl, so I don't really mind that play.
I feel like it's the same issue that public resources are facing.
Step 1: public resource is freely available
Step 2: asshole shows up, drains the resource completely just because they can
Step 3: either resource is taken away, or it has massive security behind it
We don't even need a new law. What they're doing is already a felony, and there's a process already in place to go to a court and request unmasking of 10000 IP addresses.
Why not? At this point, any attempt to try to meet the bots halfway is doomed.
More over, I actually pay for three different AI subscriptions, and use them a lot. I also think it’s fair that I pay for their services. I just want them to also pay for what they use.
But I’m seeing requests coming in from random IP addresses from all over the world using randomised user-agents. But somehow these are trying URLs in sequential order. A while ago this would be classed as a DDoS cyber attack.
Not as in “1 out of 1000 legit requests get blocked” but “you’re going to block 100% of some person’s requests.”
https://news.ycombinator.com/item?id=44962529
It never really cohered as a solution. High-powered scrapers are better equipped to handle proof-of-work challenges than end users. Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper.
The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site.
If each new site they hit forces them to solve a proof-of-work, that’s a meaningful dent in their scraping performance. Many of these boxes may not even have the spare CPU power to efficiently solve so many proofs of work - and anything that makes an owner notice their device is running slow is something that could meaningfully impede adoption of these SDKs, or force the operators to choose between minimizing performance impact or scraping more sites.
See also how captchas continued being effective for years despite services like anti-captcha offering to solve them for you for a fifth of a cent each by farming the work out to India. It took advances in AI that made it viable to reliably solve them on-device to bring the end of the captcha
I just checked, and these many months later after installing Anubis, I'm getting about 14K requests per day. It's unreasonably effective at cutting crawler abuse. It should not be able to reduce my traffic by 98%, and yet in practice that's exactly what it's doing.
I think that the most abusive scrapers are operated by people who absolutely suck at programming. If they didn't, they wouldn't be crawling every permutation of "show me the git blame for every file in this repo for every commit in its history". Were they actually competent, they'd be investing in smarts such that they weren't crawling a web frontend to get that information.
An author of a scraper would need to trade off how much arbitrary compute they'll run against how much risk they take that someone will use them for free compute (inverting their role from exploiter to victim). They can work around that somewhat by analysing common enough code (to justify the LLM or human expense) and deciding to run it.
PoW challenges work when they have low enough adoption that it isn't worth scrapers' while to have a custom countermeasure for. To scale them once adoption is big enough, it probably takes some bespoke engineering per group of sites (coupled perhaps with some scraper traps that take advantage of scraper CPU for something useful, and aren't easily distinguished from the normal challenge).
Until they discover this neat trick [1] and solve challenges orders of magnitudes more efficiently than legitimate users.
The game theory of Anubis is not sound. It makes fundamentally less sense than Captchas, and even those have been on the way out for a while.
Evidently, it makes _enough_ of a difference for the people running it. Criticising it is just letting the perfect be the enemy of the good.
The alternative to Anubis isn’t “something better that isn’t being picked”, it’s “completely unprotected”, and the people criticising it for being imperfect could helpfully move the conversation forward by actively suggesting a concrete alternative.
And I really wasn't trying to be snarky; people had "what is i + j" type of captchas for a long time and I still see them every once in a while!
All they need to do is forward traffic and launder its IP origin. They don't even need to be able to (and would actually be well advised not to) decrypt TLS streams.
> meaningful dent
I don't think that's generally how these networks use residential exit proxies. There are at least a dozen well-developed frameworks out there for decoupling the crawler from the network exit point.
Most res proxy exits are just slinging bytes for clients using SOCKS, or another tunneling protocol.
If nothing else, a modern scraper will want better control over their TLS fingerprints, and you can't get that if you're depending on the on-device TLS libraries alone.
However, the refrain from naysayers isn't that there's too much enthusiasm. It's that anubis doesn't make sense as a theoretical barrier against bots. Who cares as long as it works empirically, even briefly? Are you suggesting people let their sites continue to be degraded or unusable until they find a better solution, even though PoW challenges can improve the situation today?
Not everyone has the luxury of writing their own web app designed to handle massive bot traffic at minimal cost, gutting functionality to meet that requirement if necessary.
I noticed a couple of weeks ago that 1-2 orders of magnitude more bots were getting past anubis challenges. However, it would be incorrect to conclude that it's doing nothing and now worthless aside from heating up visitors' cpus. If I turn it off, I see >2x bots.
The next salvo in the war of attrition will probably be when anubis switches to wasm hashx, which the author suggested, in previous HN threads, is on the roadmap.
If bots quickly adapt (Claude vibe-coded JIT wasm-to-native-transpiler for hashx challenge algorithms?), that may be the end of anubis and efforts like it. If they don't, I expect more HN threads with people complaining that such compiler efforts are possible and therefore PoW-gated sites are doing it wrong... even if those sites stay usable.
I'm just saying, it's bad computer science.
A better, perhaps the best, argument is that a worse bot mix might appear tomorrow, so anubis should be used as a runway for a better solution, and not as a solution in itself. If only everyone with a website had the desire (or AI tokens) to implement their own better solution. In some cases even desire or a lot of tokens isn't enough. I mean, you think kernel.org folks haven't put some serious thought into it? And yet, as HN argues about it, there's kernel.org running anubis at difficulty 5.
It shows they really don't care and they are just rage-baiting readers in their weekly grumpiness without any suggestions.
Unless they come up with said "solution", they can completely ignored in their complaints.
(Nothing against it, I'm always happy to hear what Thomas has to say)
So could you give a short elaboration on what was this "better computer science" solution that you came up with in your discussion?
- It's the SOTA companies doing it?
- Scrapers are doing it for training data?
Those are two assumptions I see in posts and threads around Anubis, that are taken at faith, and never once substantiated.
There are more than just the American top dogs (OAI, Anthropic, SpaceX, Facebook)... especially the Chinese government with all its infinite cash resources and next to zero ethical constraints.
I don't trust the US top dogs at all, but I think the fear of discovery alone would lead them to not use "residential proxy" services. Non-US/EU entities however... who cares?
[0] https://www.ft.com/content/07611b74-3d69-4579-9089-f2fc2af61...
If you were dealing with a DoS like this and had an option that cut the annoying traffic in half in a fairly deterministic way, that sounds like a decent thing to do operationally in the moment right?
It's like saying we should just completely give up on all NP-hard problems because we'll never discover a way to find the optimal solution in all cases. Meanwhile in the real world, people are more than happy with, say, a route planner which virtually all of the time gives them a solution which is within a rounding error of optimal. Nobody is going to care that it is technically bad science.
Same with Anubis: Sure, it's a dead end in an ongoing arms race. It won't last forever. But right now it works pretty well, so let's hope it lasts until someone can figure out a better approach.
And the bot progression from "alter user agent" to "change IP addresses" to providers having to ban whole subnets, whole ASNs, and realizing "proxy SDK monetization" is a thing mirrors threat actor progression from the time before LLMs.
The AI crawler traffic I’ve seen sends one request per ip and seemingly has an infinite pool of residential IPs. You can’t block the ASNs becuase you also block honest clients. IP blocks are the wrong solution.
And because I’m being negative I’ll also be constructive, IMHO the correct solution for fighting residential proxy crawlers is using RTT diffs this is one example https://github.com/Sakura-sx/Aroma
It's not the most sophisticated thing.
Fork Anubis. Slightly modify the hash function it computes. Deploy. Do not try to make your fork widely adopted. Do not even publish it.
You've just defeated ASICs and any craweler that's special-cased Anubis (currently all of them). If enough people do this, the only recourse they will have is either genuinely executing served js code like a real user or building some unholy pipeline that uses ai agents to compile it to a GPU kernel for every host.
A more robust solution would be to keep a few patches handy with different versions of the algorithm, and rotate which one is in use. This would keep the crawlers on their toes if they wise up to the changed algorithm. One could even imagine automatically rotating witch algorithm to use on a weekly basis.
I do wonder about the premise as well: are people special casing for Anubis?
[0]: https://github.com/Homebrew/brew/pull/9383
[1]: https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm...
If I'm doing a shallow clone it isn't because I only want to receive a specific commit, it's because I don't want to burn a giant amount of disk space and network traffic on a full history.
In most use cases it would be perfectly acceptable for the server to send additional data. The client doesn't care about it because it is meaningless to them, but if it results in a significant load reduction on the server's side they don't really mind receiving it either. A 100MB shallow checkout coming with 400MB of garbage still beats cloning an entire 5GB history!
- i have a free, niche resource
- it becomes too popular
- i make it more efficient
- now it's really popular, and people are "abusing" it
- let's make them proof-of-work
- ... and proof-of-work harder
- but now "legitimate" users can't use it
- ???
which is called l402/x402. micropayments' day has come
The author of Anubis hates this fact and will ban you if you mention it, so don't. He insists it's the PoW.
>will ban you if you mention it
Even if mentioned really politely?
[1] would hope anyone trying this makes it accessible to visitors with disabilities
Anyone who responds with something like:
> I don’t have subjective feelings, consciousness, or an inner point of view, so there isn’t a sensation of “being me.”
is definitely an LLM.
As a tech person who works with tech people, I have become extremely sensitive to this kind of bias. Is this solution actually better? Is the CPU cost actually worse than mildly annoying everyone, or is it a problem being solved because it “offends the senses?”
I’m not leaning towards yes or no for this instance. But I regularly see people jumping to conclusions without measuring. What is the cost of 20% and is that cost worth “mildly annoying” everyone?
The problem is the cost here is paid by a group of volunteers, no? It's objectively reducing capacity for a really important public project by 20%. If you're responsible for keeping a public good like this available, it seems obvious to me to want to prevent this overuse of resources. Only one of the 3 parties involved here is not willing to engage in good faith right now and causing harm. I don't understand a need to try and tolerate them.
> What is the cost of 20% and is that cost worth “mildly annoying” everyone?
from the articled:
> With a bunch of generous assumptions, legitimate requests are only about 2% of git.kernel.org traffic — everything else are scrapers.
this is not some "CPU use is 20% higher than baseline" situation. it seems that people still do not understand the scale of these bad actors.
Nevertheless, without knowing what it actually is in cost, it’s impossible to have an objective sense of the severity of this problem.
Is it, “everyone’s mildly annoyed so we can save $10k per year” or is it “everyone’s mildly annoyed so we can save $100k per year?” And has someone attempted to reason about what the $/year cost is of that mild annoyance?
I’ve seen this a lot in my career. Developers coming to conclusions that are ultimately based on what feels expensive or problematic.
I'm certainly not saying you should use this code, but it's a proof of concept for avoiding the CPU overhead of cgit rendering by loading the data on the client.
It cost me £8.27 of Fable use (from the free credits I've been given) and 56% of my five hour quota on a $20/month Pro plan. There's no server logic, it's 1.3MB of minified JS and CSS and (while I'm absolutely not suggesting anyone try to use it) it basically works: https://github.com/andrewaylett/rgitweb
This is a one-shot, my prompt set the expectation that I'd be able to load resources using CORS but (not entirely unreasonably) the Git hosts I've tried don't set CORS headers.
Shared more because I was pleasantly surprised at how cheap and easy this was -- and with a repo link because talking about it without sharing the link would be a bit crass.
Interesting to crawlers is not a narrow scope. We have the same problem on a B2B car wash site.
The crawlers get stuck in a loop requesting the dataset listing page with every. single. combination. of. facets. At essentially as fast as it can be pumped out or blocked.
Had one bot super interested in a single organization to the tune of 1000 r/min for 24+ hours. Realistic rotating user agents, realistic sec-* headers, no ip address seen more than a couple times in 10 minutes. The only commonality was the route.
However day after day it just gets blasted with requests for deep pages. I was quite alarmed when I saw a 100x increase in the daily active user numbers which relied on session length, only to realize they were all bots.
Naively I too initially resorted to blocking user agents(Meta is thankfully nice enough to identify themselves, not nice enough to stop blasting 50k requests a day however), IP ranges from cloud providers and various browser fingerprints that I found connected to suspicious traffic.
However the battle seems unwinnable at the moment, outside of gating all content behind auth which I don't want to do. We have around 500k user generated content pages and I want those to remain publicly available.
I would be happy to provide our data to any one of these scrapers and I even added a message asking them to contact us if they want access to our data whenever I return a 403 response, however nobody has reached out.
Another campaign that someone is constantly running is daily checks for 100s of possible secret/config paths in hopes of finding an exposed private variable, these i've just blocked even though they would return a 404.
I still haven't found a way to deal with rotating residential IPs however, and most likely never will.
My current approach is to just run a 24 hour scan of all requests with codex and update my next.js proxy with more IP ranges, browser fingerprints and anything else that won't affect a real person.
Has anyone managed to come up with a way to stop this onslaught of crawlers and scrapers?
> 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
I've been on all three sides (user of RPs, getting paid to run an RP, and trying to block RPs from my site). Residential proxy service is nice. You can scrape anything, even with the dumbest curl command, and only get a Cloudflare block maybe 15% of the time, in which case you just try again. That's less often than I get a cloudflare block from using a privacy browser from a non-proxy address. Cloudflare does not stop bots, it stops humans.
Like, maybe you can't stop the LLM bots, but you can use them as one-off Bitcoin pool mining pool participants. You have to assume that making them find hash values with N leading zeroes has led to finding hash values with more than N leading zeroes. Maybe run a Bitcoin node under there and let each visitor take a couple swings for you with their pickaxes.
It's not really going to help though because the scrapers using residential proxies aren't burning their own compute.
I just close the website if I see Anubis. Some have it set at reasonable difficulties (like 2)… others have it where I need to wait for like 30 seconds, I'm not wasting 30 seconds of my life for that.
when a system was designed for 100 req/s and bots hit it with 5000 req/s, nobody entering that queue is having a good time. Anubis is the trade those operators make just to ensure your request gets serviced at all.
30s load time is already a sign that the Anubis approach is breaking down. if there's nothing else ready by the next order-of-magnitude increase in crawler load, those sites quite likely will just disappear from the public internet. hate Anubis all you want: for most of us, the realistic alternative is strictly worse.
If someone else wants to run a public HTML service, let them deal with the bots.
If you really want to offer a web interface, put it behind login. You can apply enough restrictions (captcha, super slow rate limit for new accounts) that it isn't cost effective to generate zillions of logins, and you can monitor logins for bot behavior.
Sucks, but here we are.
You can fairly arbitrarily increase the difficulty of user registration, far beyond what users will tolerate for viewing an individual page. You can exploit this; a bot needs to make many accounts for the activity desired, and it's not hard to make "generate account" more expensive than it's worth for the amount of activity they get from each account.
If it costs your attacker a penny to solve the captcha to make an account, and they can only get 100 pages out of an account, you win.
For many sites, your hypothetical penny to create an account is a cost an attacker would gladly pay.
If repos like this were 10%+ of what they crawled, having a special case for clone-able repos would be smart, but if you're crawling everything, you're not going to do an efficiency tweak for each special case that has a more parse-able option.
This ends up tacitly turning a very legitimate complaint (ill-behaved crawlers) into a justification for head-in-the-sand AI denialism.
That's head-in-the-sand stuff. AI is certainly very capable of being dumb (as are humans). But:
> “The problem was in need of a new real idea, which this new result seems to provide,” says James Maynard, a mathematician at the University of Oxford. “It seems that the AI has made a genuinely interesting mathematical contribution.”
https://www.scientificamerican.com/article/no-ai-didnt-just-...
Nobody a decade ago would have said "oh yeah solving a bunch of open problems in research mathematics, and finding a bunch of zero days in Chrome and Firefox, and winning literature prizes, are things that don't require intelligence."
I wonder how exactly the average scraper got to be so inefficient on kernel.org.
Did someone prompt a SotA model to write the most generic scraper possible?
Did someone prompt an old local model on their laptop to write a kernel.org scraper?
Perhaps no LLMs were involved in the first place. Seems to me there isn't much relation between how good a random scraper is and how usable/effective Mythos/Sol's outputs can be.
Thats a good enough explanation in my opinion.
Source: someone who reviews AI generated PR's on a daily basis
Reason they interact so poorly is that is that git hosts generate a lot of links. One for each file in each commit, and a diff for each file appearing in a pair of commits. Even a small repo can have millions of viable links, and most of these are stupidly expensive to render for the git host. On top of this crawlers generally don't have a very deep understanding of what they are crawling, and can't meaningfully distinguish computationally expensive requests from cheap ones.
Distributed scrapers are going to generally try and hit their assigned list of pages; it’s a bigger waste of time if they have to go off to visit other pages first to get the cookie challenge.
Keep in mind that Anubis will rarely inconvenience an actual user who visits the site often; it’s meant to keep out first-time scrapers trying to grab a few pages from their queue.
Anubis has a fairly capable "policy" system. you can place something like this in your policy.json:
``` { "bots": [ { "action": "WEIGH", "expression": "path.startsWith(\"/expensive/endpoint\")", "name": "scrutinize-expensive-endpoints", "weight": { "adjust": 20 } } ] } ```
another thing smaller sites benefit from -- where the load induced by crawlers tends to be bursty (e.g. as they discover new expensive endpoints to crawl) -- is to adjust the difficulty up/down to maintain a steady system load.
``` { "bots": [ { "action": "WEIGH", "expression": "load_15m <= 16.0", "name": "sustained-low-load", "weight": { "adjust": -10 } }, { "action": "WEIGH", "expression": "load_5m >= 24.0", "name": "intermittent-high-load", "weight": { "adjust": 10 } }, ] } ```
1) ban TV-proxy-as-a-service - straight up go to every representative there is and start pushing and lobbying and everything to stop spammers from distributing over non-computer devices, especially legally
2) make old commits more expensive to access than new ones. Legit users are not going to access those much, so they can pay the time. I assume diverse (unpredictable?) difficulty can also make spam pulling harder
In every country of the world? I think you might have a significantly easier time convincing Samsung and LG to ban this via their T&C.
It feels like turning these frontends into JS-only viewers would resolve this, for the most part. The JS clones the repo in memory and renders whatever lens the requestor wants, and the server becomes a dumb object storage that uses less resources. The anti-JS folks are free to clone the repo still, and view whatever lens they want, so that minuscule slice of the legitimate requests is still served, albeit with a degraded experience.
First, scrapers would start running JS. Whether they're running chromium-in-a-box or something more clever doesn't matter. Compute is effectively free for them, and even shitty WebOS set-top boxes can probably run a stripped-down headless browser with a JavaScript engine.
Second, a full clone in browser memory is massive for something like the Linux kernel. Lots of browsers and pro-JS folks wouldn't have the resources to run that.
Third, assuming that scrapers are willing to play the JS game, suddenly you have a massively increased rate of full clones happening. Even if serving raw git is cheap for your backend, the bandwidth the elevated clone count drives is not.
2. Why would you need a full clone to access /blob/cee9395acd8043be0644b25c34bfa86623f2b935/block/badblocks.c?
Rather, the article describes the routes that compute views (e.g. show the change history across several commits) into history as being the issue. If we take that example, a JS client would have to fetch N individual commits/routes/objects and compute the requested view, but first it'd have to fetch the indexes/logs to determine what commits exist within e.g. a specified range.
I suspect that'd require more work on the frontend than "just run WASM-built git" ... unless the proposal is for it to fetch all requested objects lazily, in which case I think you'd be surprised how many files are read by git when answering a question like "show me the diff by user XYZ in file ABC on branch QRS between date 1 and date 2". That starts to get expensive to pull in the browser, and the bandwidth costs might start to hurt even if the backend now only had to serve cacheable dumb blobs.
But this begs the question, why is cgit scanning 340 MB to answer this question? Is cgit not using good indexes? (Making a space-time tradeoff that doesn't make sense on a public-access website in a post-stupid-AI-scraper world?)
Thanks! FWIW, I engaged because I think there's the basis for a good approach here and I want to see if we can expose what works through discussion.
> why is cgit scanning 340 MB to answer this question? Is cgit not using good indexes?
That's the critical question (ugh, as I write this I realize I'm starting to sound like an AI, sorry). It comes down (I think, I don't work on kernel.org) to a few things:
- cgit is ubiquitous and easy to deploy.
- Cgit doesn't have a database, or indexes. It's a fairly thin wrapper over git itself. Git itself is optimized for performance of local filesystem operations, which is nice, but means it can be written with e.g. the page cache in mind for performance, and doesn't need to spend a ton of time thinking about the number of files it accesses--after all, they're all small and locally-available, and thus probably cached.
- Things like cgit often have to combine (the C equivalents of) multiple chained git commands to render certain views.
- Cgit has caching, but it's simultaneously too naïve and too specific to handle use-cases like this. It's too naive in that it's caching content blobs via the filesystem (and using sendfile to serve them) without awareness of e.g. shared cache blocks for deduplication or anything, so it can't cache intermediate states used for rendering e.g. diffs. It's too specific in that it's only caching the results of certain underlying git operations and specified renders, so it can't prioritize e.g. cross-application LRU/frecency patterns when deciding what to cache and what to evict.
In short, what you're proposing is probably best implemented server-side (and hopefully already exists): it's an indexed database of Git's objects that has behavior parity with Git's rendering of the various views/questions people can make of the server. That's a nontrivial undertaking: git doesn't have a VFS layer you could plug a database into the way something like SQLite does; it very strongly prefers procedural file-based accesses to its data. If you could use something like that, you could engineer it like a typical webapp: content-unaware caching layers and indexes to make certain operations fast. Not easy to build, and not easy to swap in in place of cgit (which you also probably have to keep a lot of behavior parity with), though.
If we trusted digital signatures on content instead of server names, we could have a model where a single bit of server load propagates to millions of interested parties. As it is the server must do something distinct for each interested party.
CDNs mitigate this only partially, because mutable data means they have no good cache invalidation strategy.
There's got to be a solution that doesn't involve heaping even more burdensome requirements on those who would dare to publish.
What irony that this article is about crawling content-addressed data.
I'm not sure where IPFS went wrong, but I think our web will continue to degrade until we figure it out. (It'll continue to degrade after that also, but then we can let it burn since we'll have a replacement to switch to).
We hardly even use the globally consistent names we have. Between humans its always "hey did you check out that website I sent you?" And never "hey did you check out https://foo.bar.biz"
Sharing and endorsement can just be part of the protocol. My reference to the site may as well be in terms of which of my friends think its trustworthy so I'm better enabled to make that determination myself (some of my friends have better taste than others).
Knowing that Alice.brownierecipe.2 and Bob.favoritebrownies refer to the same underlying document with an unfriendly global name is not so gnarly of a UX problem that we need to tolerate single points of failure to avoid it.
Another solution I came up with while reading HN comments: whitelist IPs instead blacklisting. Give access to well-behaving hosts/groups. It can even be shared across different sites. Although this would create a market for selling "good IP" proxies.
Obviously though none of us want to give that up, so the alternative is that we can almost never 'prove' we are human especially with bots getting as smart or smarter than the average redditor.
Reminded of the mules renting Airbnbs to use as USA-based delivery locations (tricking grandma into FedExing cash for one scam or another) - https://getrichslowly.org/scambaiters (probably Jim Browning + Mark Rober specifically https://youtube.com/watch?v=Xvjjpzyiig4 )
But! Using a network of real ID-checked humans to scrape the web, what would that be--half a billion times harder than Firecrawl or whatever they use today?
Too bad it's dead in the water today because so many (like me) hate the idea so much. Perhaps a biometric dongle (retinal-scanning orb :-/ ) that the staunchest privacy hawks stamp with their seals of approval because it's somehow nearly impossible to go horribly horribly... horribly... wrong... Yeah, anybody who can crack this issue, hope you have the free time or find the funding to try it, we need ya.
Why? We are not so special or magical.
I think I agree with this. There will be difficulties though:
- Will likely have to be pay-to-play. No point in whitelisting IPs if you are bombarded with whitelist requests by a bot.
- Will come with the assurance that any bot traffic will result in an IP ban.
- Will rule out people with dynamic IP addresses.
- Will be difficult to gain new users.
Interestingly enough, cgit bills itself as "a hyperfast web frontend for git repositories" [0]. Alas, nothing is fast enough for the bots, it would appear.
https://bsky.app/profile/xeiaso.net/post/3muasair5uk2j
Rather than using Anubis to give the client fake work, maybe the way to go would be to get the client to do the real work in JavaScript? Then the server doesn’t have to do it.
But given that it's only a few million actual commits it may be possible to come up with an intelligent caching system that at full capacity require only a few TB.
It's ridiculous also how you ban an IP then 1 second later another one picks up from where the first one left on.
If so, you could serve iocaine-style bogus pages 1% (say) of the time that:
1. "Look like" real pages to an LLM-less computer (if you get to the point where you have pushed crawlers to use LLMs to detect nonsense, that already increases the cost a lot)
2. Look "obviously wrong" to a human (E.g., you could take some regular text and swap the order of each adjacent pair of words)
3. Are cheap to generate
4. Important: Contain more links than regular pages, on average, and each to an always-bogus page
The idea is that, due to the large number of pages fetched by crawlers, even with a very low "random bogus page rate", like 1%, they will soon unwittingly hit a bogus page, from which point the fraction of their time spent accessing expensive genuine pages will fall exponentially due to the compounding effect of the higher outbound link count on bogus pages. Humans seeing a bogus page will be confused and annoyed, but simply refreshing the page in the browser will solve the problem 99% of the time (and of course the possibility of this happening can be documented, even on the page itself).
The main advantage is that this does not require any IP-based tracking. You could of course decide to apply this only to pages that are already slightly suspicious (e.g., very old commits).
So how do we go from here? Is adding more difficult Anubis and Cloudflare bot protection really the solution? How many millions of human hours and billions in infra costs are we willing to spend on this arms race?
Some approaches that I think are promising:
- A robots.txt V2[0] as a standard way for website owners to state how bots and AI crawlers can use their online content and where to go (e.g. distinguish search from AI training use cases, point to a downloadable file instead of crawling everything, etc.).
- Something like Web Bot Auth[1] as a non-centralized standard for self-identifying bots and agents cryptographically. This would allow websites to allow or deny bots very precisely.
- what else?
[0] https://datatracker.ietf.org/doc/draft-vaughan-machine-reada...
[1] https://datatracker.ietf.org/doc/html/draft-meunier-http-mes...
The people who are having the costs of users not paying "their fair share" (this phrase has officially jumped the shark) are the investors who are subsidizing these companies.
Maybe we should serve the bots a problem worth solving and benefit both parties. They spend some energy/tokens on a problem and we pay them with content.
If only I had a bot problem, this would be interesting to explore.
One persistent thought is what are people doing with this data? I get that people want to train models, but I also have a hard time believing there are more than a hundred companies with the resources to spider the web like this and actually do anything meaningful with all that data. Academics, researchers, and people working on lower level innovation are probably well off with CommonCrawl.. only people trying to make frontier models really need fresh and endless data right?
Extensions like ModHeader, BrowserMask do this: https://github.com/apify/crawlee-python
https://github.com/mthcht/Masquerade-Spoofer
Great for AI scrapers, bad for hosters and everyone else.
This seems fine to me. It would be a better world if we could have anonymous bulk data access. But if aggressive scrapers are bloating host costs, I’m fine with logging in.
Now, the flip side is that ONCE logged in, I want my bulk access. The worst of all worlds with when you demand authentication and then STILL block bulk access.
Case in point, I want to automatically download my Amazon and Target order records. This is easy to automate with playwright or whatever, but authentication stays annoying. My sessions expire quickly and I have to re-auth all the time. There should be an API to pull this data down.
TIL shallow clones are expensive. That's wild to me. It's supposed to be cheaper!
It's like anubis but instead of doing useless math, you will be moving the legitimate cpu work to their side.
Why cannot we turn this whole proof of work thing into an official "help mining $SHITCOIN"? I mean, if they really want the data that badly, at least have them pay the hosting with their CPU/GPU/ASIC cycles.
Allow git clone for free/unrestricted still.
Require the user to sign in to view html views. Sign in require a valid email or phone where a validation link is sent.
Or: Users signed in won’t see the Anubis. Users not signed in can still see the html views but have to use a very high work level?
Or: Limit unauthenticated requests from an IP to 5/minute. Authenticated requests can do a lot more before hitting the limit.
Want to render than seven-year old commit via HTML? Sign in. (But I have no idea.)
dont click next link because he will show a nutsack image if the referrer contains hackernews. love the guy.
www.jwz.org/blog/2025/01/exterminate-all-rational-ai-scrapers/
basically, instead of blocking, he just poison it. and if a human sees it, it takes less effort to ignore the nonsense than it takes your pocket computer to deal with proof of work.
First idea - there should be some cost for sending traffic somewhere. And the server owner also should receive pay - not only the internet provider.
So, in with this idea, the server owner can potentially increase the amount of computing power to satisfy all requests.
So now, because of bad actors, I need to pay for the privilege of watching a website or using a service that was meant to be free? No, thanks.
I don't have a solution, but "break how the web currently works" is not one I would accept all willy nilly.
EDIT: yes, I realize we already broke the web (with Anubis, cloudflare, recaptcha etc) but I think we should resist slowly breaking it further.
First of all - I suppose it should be very cheap. So, real humans will not pay much.
Second - why do u think that websites are meant to be free? They provide some service, so its a rather strange that the internet is so free (in both senses). I think, this freeiness is allowed to greatly speed up popularization. But for me is obvious that it can demand payment for service.
And third - service owner really meant it to be free, I don't see any problems with this in my idea. It can still provide free service.
> [four phases, each of which has historically been hard-to-impossible for highly organized international coalitions]
Never going to happen.
The article mentions that some requested pages are less likely to be legit traffic (old commits) and more likely to have bot activity. Perhaps they could increase difficulty on those pages for the proof of work. Keeping the "current" at a lower difficulty allows most normal users to use the pages as normal, while penalizing the bots.
One other way I have been thinking of is just delay the delivery of the pages, either limit bandwidth or just wait for a bit until you deliver the page. For one user a (lets say max)3s delay on some pages is not a huge deal, however at scale that adds up and means the client can't gather other pages in the meantime.
Another option is to lock the out of date html renderings behind a login page while keeping the git openly available.
Or some kind of vintage “set up a request in a form and press a submit form, and we’ll pretend to take a while to put things together.”
If these crawlers are so smart, why aren't they following the rel="alternate" which is provided explicitly for them?
High bot likely hood if: If a session jumps to a different ip. If the session jumps IP after just a few requests. If a new blank session starts with a deep link. Maybe those are cases where some POW is better justified?
Assumption: the rendered HTML might be viewed by a legitimate developer, even via a deep link from outside. But rarely from a wget script without a session cookie .
The other nice idea from the comments - is this rendering effort something that can be pushed to the user? Instead of pointless POW work, can you offload the expensive rendering to the user side?
But certainly, this is just an armchair comment and the kernel guys certainly have tried everything in this arms race...
If you do scraping on a large scale you emulate a human very well to bypass bot protection. Setting dynamic but accurate user agents, setting proper sessions and persisting it, emulating the TLS handshake (https://fingerprint.com/blog/what-is-tls-fingerprinting-tran...) and emulating mouse movements to be more "human like" (https://github.com/oxylabs/OxyMouse) are table stakes.
Barring that, I think the solution is to charge money for access. Require users to sign up to render HTML, and provide a form of payment (any form you want). The cost is, say, $0.1 per GB. Rate limit all requests to reduce CPU. For the average user this will cost a few cents. For the bots you'll cover your costs and have a rate limiter to keep your system from being overwhelmed. Or they can git clone for free with no limit.
Also, many plotting libraries include an xkcd style these days:
https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot....
And libraries for various languages:
https://github.com/timqian/chart.xkcd
So if you have a preferred dev environment there's probably a way to set it to xkcd style.
Except the majority of LLM training content nowadays is synthetically generated by LLMs. I wish people would stop making this statement, I don't know why this claim persists to this day. It wasn't true two years ago and it sure isn't true now
I think any service that is trying to sell a way to solve captchas must be banned by government. At least these things shouldn't be done so openly.
I think it's a few bad actors really. Because nobody serious about indexing content will do what these crawlers are doing..
They are consume lots of content that is unoriginal or duplicate or duplicate with minor modifications.
Not sure how to block, but maybe a little bit of law enforcement could dramatically reduce the number of TVs being used a proxies.
That's bad for static content.
Try adding search, with pagination and 16 filters that can toggled on off. And 1000 tags, give each page 5.
also some commenters suggested to make the agents mine the coins but with my limited bitcoin knowledge that would take more time that would be suitable for the crawlers to access the commits.
And that's just known AI, I can't imagine what requests are pretending to a real person when they aren't.
That’s not an option for the kernel. It’s hard to read the cat-and-mouse account with any hope today. I think the flood abates someday but not sure how it happens.
It's hugely inconvenient for users not having an account and want to do more than clone an entire repo via an API of course.
Not sure how difficult Anubis is, but I would not be surprised if parsing the request through the LLM costs multiple orders of magnitude more compute than the Anubis challenge.
I've handled every type of attack and malware that came their way but these past few years, ai scrapers are a large por or their traffic, eating into the budget and now interfering with sales.
and I don't know if anyone else is noticing or watching these ASNs but it sure looks like a few well know and big name AI companies are using *residential proxies* to so their scraping.
If you have a thousands of IP addresses, and you know all the urls you want to request in advance, you can just request them all simultaneously I guess.
The next more advanced version is that URLs are unique to your cookie.
Users can't share urls anymore, but it might be a tradeoff worth making. Unique urls for each user.
You could probably still make this work, if you share your url with another user, they get the page, but heavily rate limited like a regular no cookie request. (a cookie url mismatch gets the rate limited version of the page)
> Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you.
Some ISPs ban customers based on a single report there - have fun!
You are, in reality, only hurting the actual owners, the subscribers of those ISPs who are behind those addresses. We call that "collateral damage".
If any of those actual residential users try to use a website, their ability to freely access the Internet may be harmed by a bad reputation that they do not deserve. They may be totally unaware and non-consenting to residential proxy use.
You are not, in fact, hurting the residential proxy-ers at all. Not one bit. They will move on to another IP and another compromised LAN, and they will continue to move on and on and on. They will not be harmed or impeded; they will simply keep turning up fresh, new, high-reputation IPv4 and IPv6 sources. This is a sheer numbers game, where the numbers are always in favor of the attackers.
Also if network admins keep blocking/filtering abusive residential proxies, they will balloon their firewall rules and cause actual performance issues at the network level. You will turn into your own DDOS without any actual benefit. You're on the losing side of the numbers game, and in the immortal words of W.O.P.R., "The Only Winning Move Is... Not to Play."
And similar to how most people running mail daemons are using blackhole lists nowadays and are keen to not end up on there, maybe ISPs and web hosters can use the AbuseIPDB to sort out their customers.
Just doing nothing doesn't appear to stop the scans hammering my poor Raspberry Pi serving my few Git repositories.
Now in 2026, running a "public proxy" doesn't take an administrator. You don't even need to be aware. Most victims are unknowing victims. They simply subscribe to an ISP and they have their own devices. They are being exploited for that innocence and ignorance. Most victims have no visibility to even detect that they're being used as a proxy. Most victims couldn't stop it, even if they wanted to.
I challenge anyone with a home router to list the processes running on that router, and list all current open connections on that router, and list all open, listening sockets on that router. I bet you can't do it. There are no consumer router OS that lend themselves to being secured, or even diagnosed. Malware can easily be planted on any of them and run, completely invisibly.
A residential proxy server could run on routers, could run on a switch, could run on your "Smart TV" or a smartphone, or a notebook computer. It could be anywhere in any form. Perhaps you consented to it, perhaps you didn't notice.
In no way is this the same as an SMTP open relay situation. If you wanna play "whack-a-mole" with a "blackhole list" you're simply going to overwhelm those lists with false positives and collateral damage. The residential proxies have long since moved on. You won't even find the culprits using those addresses you just blocked. You're just clogging up your own machines. It's a total self-own.
How many times do I have to hurt them before they decide to buy a different smart TV?
Seriously, that's like saying "if you try to stop your neighborhood rodent problem by getting citations sent to people with cat food on their porch, you're just hurting the innocent outdoor cat owners". They're participating, whether they know it or not. We can and should PSA and shame and regulate away residential proxies on the supplier side, but we can and should also simultaneously discourage them on the end-user side as well.
Using shared ip banlists is the best solution so far, like cloudflare. Sure maybe they hit your server for 5 seconds and then desist, but they'll attack someone else, and they'll eventually rotate.
I'm not sure if Anubis has a feature for centralized banlists, but I'm assuming since it's OS and privacy oriented, there isn't.
There's a tradeoff between privacy and abuse, you want privacy? You get abuse, you want to battle abuse? Gotta sacrifice privacy. Worth noting that unmarked vpn users (residential proxy or residential vpn users) use these proxies for privacy, and therefore give a reasonable alibi to abusers.
If you know CloudFlare as anything else, it speaks to how successfully it has grown and marketed itself into other areas.
Is that really the case? It was my understanding that LLM-based agents were explicitly allowed as long as their users follow certain guidelines [1]?
And more generally: Somehow the theory of "essentially all bot traffic is AI labs crawling the Internet for LLM training data" doesn't make sense to me at all. There are at best dozens of labs capable of running their own crawl at Internet scale, but hundreds of millions of people using LLMs to answer their questions. (If my personal LLM usage is any indication, firing off dozens or hundreds of web fetches to answer a single question is not unusual.)
While I understand that many existing projects have been resourced only for human readers and might as a result be struggling due to this, this characterization sounds a bit dishonest to me.
And unfortunately, for this use case (i.e. ephemeral queries in a context possibly lacking storage or git access), forking the individual repo to answer a handful of string match queries against it might just be more expensive than to run that query against a web search index and then just fetch those results over HTTP.
The solution would accordingly also look very different, as caching at the inference layer is significantly harder than at the training one (where it's most likely already widely done as that seems like a no-brainer).
The people that Anubis tries to block have tons of compute to spare.
For example at runpod, if you rent a container with just one B300, you get 32 cpu cores and 250 gbs or ram that are essentially just sitting there while the gpu does all the work. If you think you can impose a compute cost on these people without blocking everyone else, well think again.
Or in my case, I am running qwen3.8 at home on a couple of gpus, these are attached on 32 core epyc server with 128gb of ram I am pretty sure i have more compute than the typical dev laptop.
I am of course nice, and don't aggressively scrape peoples services.
(Each new tab is isolated, unless opening a link in a new tab. Same as Safari in private mode)
We’re talking about privacy-preserving proof of age, but as we see here the real utility of such a system will be proof of humanity.
How do you ensure it includes every human?
How do you ensure it doesn’t include every non-human?
I’m not even asking about computation or algorithms. I straight up don’t think you can make a definition that isn’t a tautology or an approximation. Both of which are useful, but neither of which can fit a _proof_.
Let's say that 95% of individual humans can pass it and only 2% of bots. For someone maintaining a website, who has to decide between using this system and shutting down their site because of the increased costs, that may very well be good enough
I want to take my existing national digital ID and use it online basically. BUT I don’t want the websites to know it’s me. Just that I’m human (or perhaps over a certain age). And I don’t want the ID issuer to know what site/service asked whether I’m a human or I’m 18 etc.
I actually doubt that! And in places that do I doubt it’s perfection. Citizenship is mostly defined legally. Residency has a legal definition that may or may not track with reality. Most legal regimes I’m aware of (IANAL) define personhood; whether humanness is defined I’m less sure of. Things that are more nebulous and subjective…sometimes they’re defined, but necessarily subjectively so.
And that gets you into dangerous territory if you’re at the margins.
A fun example: how do you define “alive”? Does the person have to have been born? Are there conditions on that? Would they exclude some people we think of as alive? Does the person’s body need to be self-sufficient or can it rely on mechanical or other intervention? Is there a line? Is the person alive if their body is functioning but their brain is not? What about vice-versa? And even if the answers to all these questions are currently perfect, what happens if and when the definition shifts?
Being alive and human is messy business. And this is setting aside the discussion of making government issued records safely usable without leaking, which they already aren’t.
Of course the debate would revolve around ”is it fine to have a national government issued ID be the ultimate proof of your identity/humanity?”
It’s problematic - yet I think it’s better than the alternative.
If you're reading this - go ahead and see HOW Linux Foundation spends their money.
The horror.
For some reason, like with fraud via email, text, and phone, we don't do anything about it. All this brazen crime and government does nothing; we don't even imagine government doing anything.
You have the power, and should exercise it, to rate limit bad actors
It's well known, and it was the point of the OP, that we don't have that power. If you know a solution, please share it - the world would love to hear it.
This is not a hard problem. It just requires that someone build a certification wall. The IETF should have done this long ago, right?
- Using a terribly inefficient way to redownload the same commits as e.g. HTML diffs, possibly the most inefficient.
- Putting in tons of CPU cycles to surpass the Anubis PoC.
- Putting in other kinds of active effort like reworking access methods and buying "residential proxies" that are probably illegal in most jurisdictions.
This sounds more like escalating DDoS than AI scraping.
AI scraping might be bad, but if a particular case that's actually a DDoS becomes the cause celebre against AI scraping, it will weaken the argument, not strengthen it.
These scrapers aren't LLM agents. They're distributed classical programs that harvest data which is later used to train an LLM. The LLM doesn't write the scraper or respond to individual scrape events. The entity training the LLM contracts someone, who contracts someone, who contracts someone to run a web scraper and send them the data.
(I'll note that while these are generally attributed to AI data gathering because of the timing of when they took off, it's not actually obvious who's running these bots. The big players all have crawlers that identify themselves and are reasonably well behaved, but I don't know if anyone has managed to positively attribute these other ones to any particular group)
This is fascinating to me. It's a large enough phenomenon that it's affecting the entire Internet and yet nobody seems to know yet who's actually doing it. Which isn't surprising, of course, it's hard to trace back to a source through all these proxies and it's probably a bunch of distinct groups anyways, but still!
Personally I have to wonder how much of it is "scrapers for training data" vs just tool-use LLMs. Even if you use chatgpt in thinking mode you can clearly see it searching and visiting a bunch of different websites to answer a question, presumably faster than any human would. That's got to add up. It's got me wondering why everyone seemingly discounts that as an option
GPTBot → model improvement / training crawl
OAI-SearchBot → search index / ChatGPT Search
ChatGPT-User → fetch initiated by a user's ChatGPT interaction
OAI-AdsBot → advertising landing-page crawl
(No second chances, I blocked their whole IP range.)
Maybe I'm not understanding how many requests at a time bots are sending to kernel.org (or how larger kernel is), but couldn't they have a local cache system too, where all it has to do it serve up dumb .html pages, needing next to no compute cycles.
Honestly, the answer is simple: sue.
It'd be hard to argue that it's not a DDOS.
IMHO if I visit your website and it intentionally starts wasting my electricity for no other reason than to cost me money, with no opt in, it's hostile and malicious.
muahahahahah
Sorry, just the thought of it
But when they do… boy do I have a trap waiting for them.
My wife calls me The Genius. I’m the guy she calls when her battery dies or when her instagram breaks like when it shows that random guy in her DMs, stupid bugs LOL
I digress. Alas, when a user lands on my page. My page wants to know exactly 2 things:
1. Why are you here and who are you
And 2. Can you produce a working solution to Pharoah’s Fortune
…those of you aren’t familiar Pharoah’s Fortune is an old chestnut little poem, a riddle if you will I like to ask candidates and so far nobody’s solved it
And the reason nobody has solved it is Pharoah’s Fortune is a very tricky problem. It’s not something you can “solve” per se it’s more like you arrive there.
So far no one has solved it. They all fall for the same trick! It is of course what separates those who write elegant C versus those write poor quality JavaScript.
So I always say to my students to keep an open mind because you never know who - or should I say where you’re talking to.
I’m bookish.
And none of them are smart enough to realize they could just `git clone` all the content and use it offline?
That is super interesting, thank you!
> At that price point, its actually very affordable to many thousands of organizations to get their own copy.
I'm still confused as to who is actually doing it though! Maybe it's affordable to scrape and store, but training a competitive AI model is going to cost much more, right?
turning it into a problem of ROI is one of the stupidest things you can do to prevent bots.
use https://github.com/danieltanfh95/continuity-auth instead
I must be missing something. How can using peoples' TVs as bot farms be even remotely legal? Especially when the purpose is to avoid IP blocks?
Then again there is the conspiracy theory about cloudflare sponsoring the scrapers
I feel like I'm at work.
We had some web crawler using Selenium to make queries and scrape the data instead of just downloading the whole file.
Every day it seems like we have some people that know just enough to be dangerous creating things like that. And then of course it's our fault that things are slow, or we won't give them infinite system resources, etc
For a shopping website or user generated content website, I might understand the terrible load of crawlers that are trying to "steal" the data.
But for the kernel, what's the purpose? Are you that "no human" are seeing your page or its content? Maybe we should investigate more the usage being this "bots".
I don't buy the explanation that there are millions LLM that are constantly trained on redownloaded data from kernel.org. What would be my better guess is that it is not training, but users are actually accessing this content through chatbot and co. Like when you ask why your sound is suddenly not working anymore after an update or why your wifi driver is constantly disconnected after leaving sleep, it might be possible that the "LLM agent" is requesting the commit contents to "understand" or refer or explain them. Is it a bad thing if it helps users?
But actually, regarding this article, I'm quite amazed that with all the advances of the linux kernel, and server softwares, and that the C10k challenge is solved since a long time, still such a basic traffic is such an issue.
> At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html. 14 cpu looks nothing to me. It's like you have 1 iphone and 1 raspberry pi active in a corner of a room. Counting in "seconds" of activities, easily shows meaningless huge numbers. Do you want to know how many breaths I take per year? 8 to 9 millions!
Most certainly, the usage of this shitty Anubis has ruined the climate million times more only with the wasted cpu resources of legit users...
But moreover, by definition the git commits are not supposed to change, ever, so can someone explain to me why the fuck do kernel.org "re-render" the commit to html each time someone is accessing it instead of using a cache or a static version of the html of this commit?
> oh, several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits Again, reading that, my immediate thinking is that it is a shame that such talented people would not be able to have a proper optimization, so that getting the 922 duplicates are just costing a fraction millisecond more after the first person retrieve the first page.
This has nothing to do with that.
Any Node.JS application will happily accept 100K connections. They'll all wait for the under-resourced database behind it. That application "solved" the C10K challenge, but it's still overwhelmed.
> it might be possible that the "LLM agent" is requesting the commit contents to "understand" or refer or explain them. Is it a bad thing if it helps users?
The article describes random algorithmically-generated traffic arriving in batched waves from laundered residential proxy IP addresses, a few unrelated hits in a group then gone. That's not the pattern you'd see if end users were asking their agents for help.
> it is a shame that such talented people would not be able to have a proper optimization
It's mostly not static content in the sense that you're implying.
Routes that access a single commit can be cached. But most of the routes scrapers are hitting are e.g. computing diffs between arbitrary pairs of commits, or other computed-on-the-fly views into history.
I'm sure they're already caching their useful-to-real-humans data. As the article said, the vast majority of their traffic is bots hitting those arbitrary, permuted URLs. So whatever cache they're using is probably a) missed almost every time, and b) constantly getting evicted to make room for data served to bots (unless they eschew caching to avoid this--fair--and are thus back to the original issue regardless).
There is no "proper optimization" here. It's not slow to go compute the diff between a random pair of refs, render that into pretty HTML, and serve it. But it costs something more than a cache hit, and doing that dozens-to-hundreds of times a second constantly consumes resources.
> But most of the routes scrapers are hitting are e.g. computing diffs between arbitrary pairs of commits, or other computed-on-the-fly views into history.
Looks like to be wrong based on the article: Today, git.kernel.org receives about 6M daily requests demanding to see random commits.
Also, in "how bad is it", there is a bar graph of the tasks requested. And only diff can't be cached, and it is clearly not the most important one.
> batched waves from laundered residentail proxy IP addresses
Again, I don't see anywhere anything stating that there was "grouped" requests that are directly related, except in something not related to AI: The thing that usually takes us down are not scraper bots, but poorly designed CI systems that try to do something stupid like shallow-clone stable.git from 20 different nodes, all at the same time. (Shallow clones are awful. Run your own damn mirror if you're going to do something nasty like that.)
Just "waves" of requests coming from the same "residential proxy IP". But that makes sense because they might come from a same service without being a single "source" trying to scrap for training.
Let's say that I'm chatgpt, or Claude, and I have a lot of user related requests to do things based on the commits there, each one will be a different task, but they will all go through the same pipe, that would be a proxy with the residential IP if the provider notice major websites like this one blocking requests for bots.
Ah yes, the stereotypical Linux kernel developer.
> Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you. You just needlessly ballooned your firewall ruleset by adding IPs that would never be back.
Without something like cookies (which are almost certainly tossed after the IP is rotated) or some other persistent identifier, you are stuck have to apply mitigations that scale with the load you're encountering, which means longer challenges for everyone or degraded functionality, like removing some of the fancier cgit features.
Wait what? I never heard of that. I call that a botnet
I'm strongly convinced that these aren't "AI crawlers"; they're just plain DDoSes done by those who have interests in turning the Internet into a dystopian walled garden with "security", and now they have a convenient scapegoat to blame. Don't you find it too coincidental with the rise in identity/age verification and other attempts at silencing free speech on the Internet?
It's widely known there are questions that LLMs can't solve, and once in a while an obvious example appears, so a simple CAPTCHA-like challenge with an HTML-only form would be the logical "defense". Instead there's a huge interest in pushing JS-required proof-of-work (as others have pointed out, these attackers have far more compute than the average user) and remote attestation (there are already providers with huge farms of mobile devices that can defeat this easily).
Things just don't add up.
...and how convenient that pointing out the inconsistencies and this massive scheme of propaganda and manufactured consent gets you downvoted. "Truth does not resist questioning."
Is it foolish of me to have expected more from a blog post on kernel.org?
The predicted consequences have now had time to manifest, and have not done so. This makes the claim either false or overstated. Perhaps there will be issues in the future, but to date there have been many claims that AI development will stall (for a variety of reasons). If they were the critical weaknesses they have been portrayed as, models would not have advanced to the level they are today.
If you have a hypothesis, make a clear prediction based upon it. If you start pushing the date forward after each failed prediction, you end up looking like a hapless doomsday cult. If your hypothesis is correct however, your prediction should actually happen. Then provided you have not made so many predictions to get one right by chance, people will take what you have to say seriously.
Seems like this chaps problem was that the actions were based upon a different hypothesis than the one he stated. X will cause Y by the end of the year is considerably different to X will eventually cause Y.
Learn how to cache your application and your cpu usage should be almost zero. In fact basically any read should cost nothing in comparison to writes.
> Us PHP developers
I can tell.
Imagine if WooCommerce had a route "/product/<sku1>/compare/<sku2>" which displayed an auto-generated comparison between any two product pages.
Now imagine running a million-SKU WooCommerce site, where each product page was 100kb of text.
Now imagine scrapers are permuting those URLs.
How would you cache that?
That's what cgit/kernel.org and many other Git forges are dealing with. These aren't static websites, even if the underlying Git repo is largely static; they're rendering arbitrary diffs and other generated-on-the-fly views into Git history. The ability to do that is a large part of the value of a Git UI.
The following from Claude: """ A diff between two randomly chosen commits usually spans years of history, so it's not a few KB — the tree itself is ~1.5 GB of text, and a multi-year span rewrites a large slice of it. Call it 100–200 MB per pair on average: 8.5×10¹¹ pairs × ~2×10⁸ bytes ≈ 10²⁰ bytes, or ~150 exabytes """
“[…]because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git.”
> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
But if bots a crawling diffs between all possible commits it's crazy.
CDN will do nothing, because it's new urls each time. You can maybe find a CDN provider that block bots.