How to Block Some of the Bots
nochan.net
nochan.net
WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that.
Are you using lambdas or something like that to serve a blog?
Is it basically just OCD? Vulnerability protection? Old habits from a time where crawling really did impact service?
is the goal for your content to just be invisible to this growing population? is the expectation that all of this is just a fad, which will fade away? what is the end game to the tactics you have outlined? what is the strategy?
For smaller higher trust communities I don't do any of this. Rather we use basic authentication to keep bots and strangers off the services all together and then regular user accounts on forums and such. That is where strategy comes into play and has been working well for some time.
how do you define a user agent? is it desirable or not for users to be able to discover and access resources and communities on the internet using the tools, formats, and workflows that they prefer?
are search bots desirable?
the internet wayback machine?
how do you feel about browser extensions and greasemonkey scripts?
I have no need for search bots personally. If I had a complex site I would build my own search feature. If it was a commercial site I would pay for advertising on popular sites that were of related categories.
Wayback is fun to play with but it's just a toy to me. There is no concept of domain ownership. Most of the snapshots of the domain I am using were from when someone else rented it. They claim to respect robots.txt but that is not entirely true. They crawl and save content even if a site says not to and as soon as the site is offline and robots.txt is no longer accessible they will display all the archived content. This means a person has to park their domain on a server containing a robots.txt that matches their intentions.
Browser extensions are mostly invisible. What is obvious to me is when people are using "reader" applications as web clients. They should be concerned more than me. They are reading random sites run by strangers using apps that may or may not have been battle hardened, reviewed by third party penetration testers and so on. Some of them are vibe coded in unsafe languages.
More interesting to me was that Alexa on Amazon Prime answered all my questions without hesitation or disclaimers but once I have my answers I close that tab.
Whoever using purely agentic worflow is not my concern. Just like people who read only physical book, learn only from podcasts or watch only netflix movies.
There is zero reason for the rest of us to worry about filling free data to companies that try to make themselves monopolies.
If the bots are eating your site traffic and users, what's the point of allowing them in?
I could improve this by moving most of the blocking to a demo site so the article is easier to access. I will do that. I will also update some of the nginx ACL's to what I am currently using that explain to the botters what is out of place.
Yes we know - I still had my site knocked out a couple of times by overeager bots. They can be really abusive.
Over the weekend I got round to turning off some old viewvc (CVS and subversion) and hgweb (Mercurial) web interfaces. They'd been running fine for a decade or two with basic security updates etc, mostly only occasionally visited for real purposes, but that's fine!
But getting 2.7 million requests/day from residential proxy IPs was a bit taxing for the old uwsgi/cgi programs (that's average 30/s), making other sites on the server suffer. It was also getting close to 1TB/month which is the VPS quota.
So I ended up turning them off, one more cut towards the centralised internet. Sure I could put a cache in front, but there are probably millions of combinations of dynamic urls in the VCS tree so caching mightn't help much anyway, not worth the effort. I'm sure the web server could be tuned better too, but again it's a waste of time.
Typical traffic:
https://hg.ucc.asn.au/dropbear-fuzzcorpus/annotate/default/fuzzer-preauth/23f6b74342011f35e657f06e9398ad339360d7e9
https://hg.ucc.asn.au/dropbear-fuzzcorpus/comparison/e67e5df225e5/fuzzer-verify/b5f57a2bb85ab7ef957e4979fd202b091de27bd8?revcount=8
https://hg.ucc.asn.au/dropbear-fuzzcorpus/log/72337e3dcccf/fuzzer-preauth_nomaths/6b1f390d26b08759983937727e10ab90aa7b03f2?revcount=15
https://hg.ucc.asn.au/dropbear-fuzzcorpus/log/4cdb539e4b20/fuzzer-preauth_nomaths/713e5cd4f73276f00a5d6685cc16d4565ad3815a?revcount=96
https://hg.ucc.asn.au/dropbear-fuzzcorpus/diff/tip/fuzzer-client_nomaths/540778ea81a915af6e898fd9658c5d9d6594c40a?revcount=128
https://hg.ucc.asn.au/dropbear-fuzzcorpus/diff/0ef1e51e583f/fuzzer-client/65ac44bdf4868866def70f40542dd1b5ca447310?revcount=30
Google couldn't be bothered to cache; I would guess this before anything else.
https://drewdevault.com/blog/Google-has-been-DDoSing-sourceh...
Dumb! When was this fixed and don’t say never :)
Switch VPS providers, you can easily find higher numbers or "unlimited".
I have it guarded by a simple cookie check now, which seems to work well enough. Only a small subset of bots pass it. It likely kills search engine discoverability, but that's the tradeoff I have to make.
Yes, in the grand scheme of things, waiting a few additional seconds to check my local imap server (because my DMZ gets bogged down) isn't a huge deal, but that doesn't mean I have to like it, or allow it to continue.
> An actual proper static site would likely 10x that.
A lot of the stuff I host isn't static. Any non-static content also gets overloaded pretty quickly. Hosting something like a Git server can turn pretty resource intensive if shitty bots start diffing every commit with every other commit.
I'm generally lazy when it comes to blocking IPs, but when I do run into an issue, I just blanket block IP ranges. When that doesn't work, ASNs get blocked. So far I haven't had to take most measures mentioned here but I probably will once the broken bots start coming from botnets using residential IP addresses.
Also, they hit so hard that nginx returns 502s and times out for real user half of the time, and when the site does load it takes a few seconds to load.
This is all on a high performance single core vultr server, not the cheapest vps but not the fastest, but should be plenty fast enough.
Block on traffic volume and request frequency if that's causing a problem.
(And yes, I can't access the site either. No, I will not conform. But I bet anyone determined enough will still get through, just like with DRM.)
On the topic of browsing with any kind of browser, I would be extra careful using things people have vibe coded or more specifically have not been battle hardened to protect against malicious sites. The reader apps people use I think would be especially at risk of malicious servers if they have not gone through extensive third party code reviews by penetration testers.
I don't remember what it was called and I don't use Chrome to test it. It sends a header with a long hex string that supposedly proves it's real Chrome.
It is more likely they were looking for a DNS server with a big response packet for use in amplified DDoS attacks.
There are loads of big TXT records on many domains that can be used for amplification attacks. [1] These are just a few by the way, I transparently filter many TXT records on some DNS servers for this reason.
Blocking http 1.1 has a higher risk of blocking real users if they are using old browsers.
> One class of bots always claims to come from the root page of google. Obviously this is a lie.
This isn't just bots. Some browsers won't include the full url for cross-origin requests, so that is the referrer if the request came from google, so you probably don't want to block that unless you don't want any traffic from google search.
I will keep the blocking of http/1.1 at the risk of blocking ancient browsers and API tools. That's just too easy for people to update these days and they are not missing much by not visiting cranky ol' me. If we were talking about proprietary code running in an old financial system I would totally get it, but the public internet can and should update for their own sake.
Any request coming to me claiming to be from Google I know is a lie as I have been blocking Google for a fairly long time. If anything Google would say I am malicious for blocking them, or at least they used to. I cycle by blog through many random domains to break any associations and related snapshots to disrupt birds of a feather associations. I try to control how people discover my silly ramblings.
I also got a free pentest. I've expanded since. Takeaways: my mitigations and pipeline are solid. If the people on HN pleading for the poor LLM users and google search have any relationship with the people providing the free pentest... I mean, they don't? Srsly.
As for blocking IPs... the survival pressure has forced 90% of the bot traffic to VPNs. Man, they're lighting up the VPN endpoints like christmas trees. Little points of light, very pretty. Thanks for the fish. I could make feeds of the one-shot wonders available, my infra is solid, you'd need to be properly vetted and I'd need to approve the use case.
This is fun.
Blocking http1.1 is reasonable imo, its been over a decade since just about every browser has supported it. Anyone on a browser that old has most the mainstream web broken to them at this point, a broken (personal) site wouldnt be an exception to them, but the norm.
https://i.ibb.co/vCDH79d0/IMG-0303.png
…and not a bot… hoping not to turn off iCloud Private Relay to read, well, anything. (Maybe Google Scholar if I really needed it, the most mainstream thing I know of with a complete and effective iCloud relay ban.)
*edit: per author’s reply elsewhere, as a test site, rather a good implementation! But other webadmins, please don’t adopt all methods if you can avoid it :)
I could some day split it out into a blog and a demo site.
Instead of learning something I get a blank screen, for the crime of being interested and living somewhere you deem block worthy. Cool.
Our situations to be fair aren’t equivalent. If my block were due to my region instead of an optional paid service, that would feel worse. Maybe it wouldn’t have been the first time either. Whole continents written off!
Let’s certainly loudly complain if someone implements all these methods and inadvertently blocks us when they should’ve known better or don’t care about collateral damage. (If they’re apologetic, between a rock and a hard place with receipts, would require further discussion. e.g. is their choice really between serving us and closing shop)
Edit: but title could add “(but don’t)” at the end
*I think they do something manually sometimes, like logging into a The Information Pro account and mirroring popular articles. (Unless they can use a visitor’s account somehow, like with some browser extension…) And suspected without evidence maybe The Information did some watermarking to ensure they could quickly ban accounts used like that.
And thank you!
They were the reason I started blocking data-centers. I wanted to see if I could block all of their addresses which turned into quite an exercise. Initially I was able to block them by blocking TCP SYN packets with an MSS of 1380 and 1300 but talking about it here was a mistake as their admin is on HN. They fixed that and made their MSS 1460 like a nominal home internet connection.
The next thing I had to block was the Russian Federation. That seems to be where the controlling nodes are, I think. That made me even more curious who they were so I just kept iterating through the blocking process. I ended up blocking a number of ASN's and all of Russia. They are all in the linked archive file in the article but I don't specifically call out the ones that are archive.is.
Most on HN think they operate on a shoe-string budget but I have my doubts. I suspect an asset of a technical arm of an intelligence agency but to what purpose and what end I have no clue. They are very clever whoever they are and certainly earned my respect. Either way I think their site is useful, I just wanted to see if I could block it.
For some reason I find this very amusing.
Nothing to read, nothing to see, I move along. (Yikes, the modern web sucks!)
People can see a few headers here [2]
[1] - https://caniuse.com/?search=sec-fetch-mode
[2] - https://nochan.net/.env
load average: 0.00, 0.01, 0.00Nice, I will have to take a look at some point.
On the other hand, tools like Cloudflare and Anubis can create serous accessibility issues; some more than others.
I prefer rate limiting; it's elegant and avoids accessibility problems. For temporary issues, short-term IP bans can also work well.
Personally, I use fail2ban to parse HTTP logs and temporarily ban IPs for N hours if they request forbidden URLs (e.g., those blocked by robots.txt or paths like wp-login.php). Same if a IP trigger the rate limit much to often. And currently, I'm testing Anubis on my Git web UI.
If it is at all helpful most layer-7 load balancers have an option to enable a header that contains the real IP address, then one need only reconfigure their web daemon to log that header. It's very similar to how CDN's send the real IP in a header.
This is a ubiquitous security concern, like IDOR and memory safety. Every time you have a separate validation step for anything, you have the possibility of a parser differential vulnerability.
If you don't know it yet, look up "HTTP request smuggling" - pretty interesting.
the real humans you are seeing, are bots.
brightdata is one vendor, they have 150m genuine residential ips, aquired through standard business. actually, since this was kind of an invitation, i scraped it. the volume is so small compared to the ai corpus, this volume does not cost anything. as i suspected it was a 100% success rate. the only page it did not reach was the one needing authentication. the tests, they were fine. of course, ai knows ascii from this type of data but it wouldn't be able to read the page unless you use a screenshot.
the bots are not interested in the headers. they never see them. imagine a normal person at home using their browser, that's the bot. the reason this is the case is because the bot ip is someone's house (at least for premium proxy), and the browser fingerprint is a recording of someone's browser. oversimplified of course.
the 'bots' are the public facing corpo scrapers, and the 'users' are bots run by data vendors.
to test this, there is a ranking on 'foil' bot detection. to actually detect them they need 0.05 usd per check, and they are more interested in fraud via automated interaction and account creation.
grab a free account, you can very easily collect such data.
The added explanations by the author in this comment thread are hilarious. You sir are a good writer.
Exactly! Like any ecosystem, the internet needs variability in order to be healthy and survive long term.
Monocultures are not healthy.
Response is brotli compressed, Accept-Encoding: is ignored
Overall, easy to make requests outside the browser, no need for Javascript
Distinguished name on the x509 certificate is obscene
Response headers are full of profanity, plus URL pointing to this video:
https://www.youtube.com/watch?v=wAk6BGgaSI4
Bananas is misspelled
But there are some fun things to read there in any case.
That is both funny, sad and oddly appropriate. Indeed, decades of running forums, chan and chat servers. I came here to take a break from all of that. It's been a nice vacation so to speak, letting someone else do all the management of a user-content driven site. Now it's all just tinkering and experimenting for me.
Actual bad actors rent residential-classified connections. Good bots don't.
Voluntary malware? Funded malware facilitation?
I'm not against crypto, but it would sound more legit if you removed the word crypto from your sentence. If it's exclusively the ability to be paid in crypto, as opposed to local currency or USD, then I'd guess it leans pretty heavily to shady-ness.
I pretty much think all residential proxy traffic is undesirable - otherwise it wouldn't need to use a residential proxy (they're used to get around erstwhile blocking, which someone has actively chosen to put in place - rightly or wrongly).
We've already established that you wish to block basically the entire internet including two actual continents, no wonder someone who was really interested in your site would try to view it with a residential proxy.
How about phonecalling someone with an unblocked connection and having them send pictures of the pages. Would you block that if you could?
"ai" ...
response length 68
edit:
>"I can not see votes or karma and do not require social validation. "
>proceeds to downvote
classic
Every step is optional of course. There are ways to make curl work on my site but I choose to add friction as some botters abuse libcurl. I doubt anyone else will do what I do. They could spoof the user-agent but most botters seem to not know how to do that despite the myth that they all do.
For what it's worth if I had a site specific to sharing code artifacts or archives I would not block curl and I would also enable native rsyncd.
This has always struck me as dishonest, but recently I've been blocked from accessing the docs of a service I pay for.
Although @Bender's unique answer is better than my obvious observation.
Plenty of people would say that I shouldn't block residential proxies, but I would 100% block residential proxies if I could.
I think most people responsible for blocking traffic would actively manage their rules if it was costing them something valuable. If they don't notice, then, well no harm no foul.
Blocking vpns is the same kind of bad... with governments requiring you to show your ID or do a face scan just to jerk off, vpn usage will go up, and you'll be blocking all those people for no good reason.
Sure, you can block whatever you want, and i can and will hate you for that.
> if ($http_x_forwarded_for) {....
This may block schools and libraries that use content blockers. Often the internal client is left to make abuse tracking easier (or because the overworked admin didn't know they could turn it off).
Oh, well that is ok for me I suppose. I add RTA/adult headers that hopefully they also look for and block using parental controls as adult content should not be viewed in a school or library. I could add a note suggesting to skip that step if one wishes schools and libraries that may be using a proxy to view.
I think it's a great article to be fair. We need more of this cheap and quick bot blocking. The fact the solution to unwanted traffic is often "use Cloudflare" is _not_ great for the internet, and nobody really actually likes deploying or managing ModSecurity. Its a nice middleground.
security theater at its finest. not to mention wasteful.
Yes, of course, if a real threat actor is doing things properly they'll be using residential proxies and legitimate user agents, but we're talking here about blocking "some of the bots" - mainly the script kiddies.
If you don't have LaLiga in your country, that doesn't mean everyone is so lucky. Blanket IP range bans, in this day and age? Basically a proclamation of incompetence.
I understand your frustration. Normally a blog site would not have any or many of these measures enabled. This is more of a test site to show what could be done on any other type of site. People can pick and choose which methods to use. Some may wish to block VPNs and proxies especially on forums.
I should make an article that shows additional ways to detect VPN's, including residential.
I disagree. If I'm getting problematic traffic from even a few of your IPs and they're in a DC/VPS range, I'm blocking it and moving on with my day.
But I suppose none of my clients services typically ever need to be accessible from such countries anyway.
Permanently blocking every IP that some random guy didn't label as residential: bad