We collect signals that help determine good vs bad networks. For example, large amounts of requests to .php endpoints, large amounts of empty accounts from the same /24 subnet, etc etc. All these signals let us automatically determine risk, and then put up a challenge. Authenticated users never see the challenge even if they are on a risky network (VPN 99.9% of the time), unless the network has been identified as 100% malicious, then it gets a full block.
Here's a small snapshot of the dashboard:
https://cos.ridewithgps.com/screenshots/6a7c54d0-12Aug26-358...
This was probably a total of 3-4 days of work, spread out over a couple months of iterative claude led hacking. I didn't know exactly what to build, but had some of the key architectural ideas in my head. Opus+Faable made easy work of it all, and ended up guiding some really slick improvements for performance.
I would say this has dropped about 20% of all traffic to our service, though it turns out turnstile is a massive target for bots, so replacing that with something custom is next on the list.
You know all these guys just switch to residential proxies if they detect a site is blocking data centers, right? Because that's a very common thing to do.
As for the latter comment....not sure what your implication is. Yes, bot/spam mitigation is whackamole, but there are consequences for not playing the game of whackamole. Luckily residential proxies are few and far between so far, but they will grow in popularity. When they do, and I can't get by with the occasional individual residential IP ban, we'll come up with other methods to handle.
Luckily the signal is strong with vulnerability scanning, which makes it pretty easy to automate. The only reason to put up whole ASN mitigation (captcha/turnstile, outright bans) is just efficiency. Nothing stopping individual IP banning. The scrapers are the tricky ones, since they more easily hide in legit traffic. However legit traffic has patterns that scrapers do not emulate (at least for a service like ours with millions of pieces of user generated content that's easily walkable), so you can still pull out the signal. It's just a little trickier.
Definitely a continual arms race though.
From a technical perspective, all this "china/russia" attribution is built on a quite shaky foundation. As a sysadmin you'd never know if it would be the British crown attacking your European company instead.
Not minimizing nation state cyber crime here, but the packet goes through many hands with different incentives.
My traffic from Germany passes through a British-owned hop on its way to Switzerland. My German ISP is British as well so either way it wouldn't make a difference, they basically have all traffic twice.
No, because legitimate users do not just use residential and "commercial" IPs. Like me, right now
I am going to move full blocking to a test node that people can play with but I have to finish working with Claude to revise someones repo is is no longer maintained because one does not simply put an anonymous chan board on the great wide open internets without some critical thinking.
The complete hysteria people go to over the near-non-issue of bots is, well, completely hysterical. Don't be that guy.
If you have fail2ban or NGINX logs, you can use our CLI to summarize those IPs and identify the ASNs you want to block. But before you block entire ASNs, make sure they are not classified as "ISP" type. For that, visit our website's ASN page first.
I have quite a few community posts around this approach. https://community.ipinfo.io/
If you have raw logs, you can send them to me as well, and I can review them and provide some guidance.
Blocklist download and configuration: https://knock-knock.net/blocklist
Honeypot dashboard, where you can see attempted attacks in realtime: http://knock-knock.net
It never ceases to amaze me that these ISPs don’t bother to shut down the botnets. They could do so very easily. For example, they could identify the IP address of every bot that hit this honeypot with their ASN with one API call: https://api.knock-knock.net/check-asn?asn=<asn number>. (See https://knock-knock.net/api). They just don’t care!
You take IP down, you kill the cancer but you also end up killing the patient.
I'd say its probably an acceptable casualty in the battleground that is the internet especially for little one-off sites hosting blogs, forums, chat servers, etc... For a bigger site I would expect that person may have to open a ticket with the platform such as Amazon accepting that some CDN's and firewalls may be harder to get the block removed. This is why we can't have nice things.
IP Blacklists, no matter how good can't stop this. You have to start using stats or deep-diving telemetry.
So appeal to emotion doesn't fly with me. If grandpa is 76 in the year of our lord 2026 that means he was 50 when the internet was getting popular and 59 when cell phones became very popular on the internet. He's not much older than I. He knows what's up.
God help the makers of that television if he finds out it has been spying on him and dorking around with his traffic. If they are lucky he will just take a baseball bat to it. If they are unlucky he will fly to their headquarters and end up on a viral bodycam video likely with a lot of supporters that will bail him out of jail.
IP Blacklists, no matter how good can't stop this. You have to start using stats or deep-diving telemetry.
I use a myriad of methods including IP blacklists. That's my choice and every site operators choice. I do not have to use deep-diving telemetry but you are free to do so.
However, I can see the argument for giving the customer 24-48 hours to resolve the problem.
[1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...
Edit / Update: It was Apple's Private browsing mode that causes it not to respond. I can now see it when this is disabled.
> block http 1.1, real users only use 2.0
Chrome on android and Firefox on linux both appear to use 1.1 still...
There are some reader apps that act as a proxy that only support http/1.1. Be careful, some of those are not just readers and do not trust what they claim to be the source code. Some of them are created by cute and fuzzy bunnies.
There are a number of botters on HN, some that control residential and phone browser-hijacked systems. One was sending me playful messages the other day. I enjoyed the bot block-jousting with them.