How the FSF sysadmins block botnets with reaction | Hacker News Reader

How the FSF sysadmins block botnets with reaction

fsf.org

168 pointspseudolus79 comments

My personal preference is to 'ip route add blackhole ${net}' as it has the lowest CPU overhead and I can add hundreds of thousands of CIDR blocks with no noticeable impact. The only downside is that it won't stop UDP packets from getting to a UDP listener. There will not be a response but the application will still see it. For my TCP daemons it's great.

    grep -m1 -E ^Tot /proc/net/fib_triestat ;ip route | grep -Fc blackhole
    Total size: 56735  kB
    426951

Those 426951 blackhole routes include data-centers, VPS providers, botnets, AI datacenters that ignore robots.txt, search engines, abused CDN's, known bad residential nodes and much more. I still see a few residential proxy bots that do a halfway decent job of pretending to be real people at times but the feds are playing whack-a-mole with them. The bots self report to my silly blog so I can block them elsewhere on systems I might care a little bit about. Happy to share them if anyone is remotely interested.

I also use a couple generalized rules in nftables raw table that keeps a lot of beyond poorly written bots away including hping3 tcp floods and masscan. My rules to port 443 are stateless. One must not taunt the state table.

The problem is when you block those "residential proxy bots" you also block real people who just happen to have a dumb teenager on their network playing some free games that are monetized by proxies.

The only real solution to bots is making users log in. And even then you have to fight registration bots.

I get what you mean. It happens all the time when some clown trashes an IP's reputation and Cloudflare or Google will send the next lease of that IP into crosswalk fire-hydrant bus traffic light purgatory.

That's why I eventually let those go usually after a kernel update and the git repo for FireHOL gets updated often. The kernels get updated often. I only perma-ban the data-centers which is fine for my silly blog and probably for some peoples hobby sites. People can chose which methods to apply, how to apply them or which ones to skip entirely.

Excellent username btw. Those SNL Celebrity Jeopardy episodes are unforgettable. [1]

[1] - https://www.youtube.com/watch?v=bEghu90QJH4

I would like to learn more how you maintain your table of IP ranges (or CIDR block). How do you decide when to add/remove a range?

I'm most concerned about blocking innocent users, currently I use Cloudflare to block known bad ASNs using a list I found on GitHub.

How do you decide when to add/remove a range?

The only IP's that come and go are the Tor 30 day blocklist and a couple FireHOL attackers from a repo though I will sometimes leave the last entries live until reboot. I do not really need to block tor but I use this silly blog as a testing ground. Tor and some known abusers come from a git repo I refresh periodically.

The data-centers, VPS providers, CDNs, known botnets are perma-banned. For my hobby nodes I personally find this acceptable. I would not do this in a professionally managed data-center. There are better methods for those cases especially for B2B corporate arrangements. Regardless of what daemons I run I never have external dependencies that need to be accessed from my node or from the client with exception of stratum-1 time servers.

I do have to periodically update the CIDR blocks for given ASN's. I have not automated this but I probably should some day. It's not hard to automate, I am just excessively "efficient". I was told to stop calling myself lazy, but I am.

Methods 2, 3 and 5 are the ones I talk about here. [1]

[1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...

Thank you for including a link to your blog, very useful read.

What does all of this give you? For a static(?) site burning a few cycles unnecessarily, saving what, 30 cents of power per year?

Peace of mind? Fair enough but I'd be more wary about blocking legitimate users. VPS providers are often used for VPNs etc.

What does all of this give you?

Good question. A playground to test things. A place for bots and their kin to report themselves to me so that I can use this information for sites I actually want to protect a little bit. A place to share some ideas with a small handful of like minded people. I do not consider power savings for a device unless I have it running on one of my inverters or if I am doing that to constrain a potentially malicious node.

Blogs are throw away for me. After some time I delete the VM and edit articles offline for 6 to 18 months and then put them back up on another domain when there may be a need to share some old articles. This method disjoints the archive sites and breaks any filters botters have set up to ignore me. That also allows me to change the CSS. I try to make it smaller each time.

I also find it easier to put long form content on a blog of sorts instead of HN comments in the unlikely chance that YC removes HN due to future ID/Age verification constrains or other unforeseen reasons that we hope never happens.

> The only downside is that it won't stop UDP packets from getting to a UDP listener. There will not be a response but the application will still see it.

Try:

  ip route add src ${net} blackhole

You could configure reaction to add and remove `ip route` commands in that format.

> This software is gay, trans and anticolonialist. If you're uncomfortable with that, please don't use it

Weird message to include in AGPLv3 licensed software (which explicitly allows people to use software however they like, regardless of their beliefs or feelings).

You can have preferences while not restricting legal rights.

You can, but if the exact quote in the GP is correct the claim is claiming the software is "gay, trans and anti-colonialist" and asks you not to use it. Why use a license that is designed to be politically neutral and then ask some people not to use it?

What I can see is a fairly clear indication that they do not want contributions from people whose politics differ from theirs. I would also question whether government funding of a project with political policies about who can participate is appropriate. The political stance is also rooted in a particular culture so is unwelcoming to people from other cultures.

Of course people can political views and preferences, but they presumably have some aim in mind when making that statement in the README. What is that aim?

A license designed to be politically neutral?

The GPL variants are the antithesis of politically neutral.

Only insofar as political neutrality is interpreted by the modern polarized political factions as a political stance.

The GPL "tolerates" conflicting political opinions, which used to be closer to the norm and thus not political.

The GPL mandates that if you modify and distribute GPL licensed software, your derivative work must also be free and open-source. This ensures code cannot be made proprietary.

Other than that, there is no expectation of political stance.

Only in the current ultra-modern age, has politically neutral devolved to meaning that you must agree with me on 99.99%.

>Why use a license that is designed to be politically neutral and then ask some people not to use it?

Because you can have preferences while not restricting legal rights.

If $CONDITION, then please don’t use this software might be construed as a legal restriction.

> What I can see is a fairly clear indication that they do not want contributions from people whose politics differ from theirs

This is the same FSF that in the past has refused contributions from people whose politics include "I would like this software to run on my windows/apple/other proprietary platform". They're extremely political.

When did this occur? I am a GNU maintainer, and have never heard such a thing from the FSF. The GNU Coding Standards and other similar texts leave the decision to support non-free platforms up to the maintainer.

It was an emoji solution for Emacs that only worked on Mac. RMS basically said:

"If the point is to promote Free Software, we do ourselves a disservice by making our Free Software work better on a proprietary system than a Free system. Let's include this support when it works on the Free platforms as well."

FSF is political, but only about Free Software. Their goal is to promote it, and I think RMS has shown very clear thinking in this regard. I don't love the decision/outcome (someone did work to make something better and it was rejected), but I get it in service of the larger goal.

To flip the pschology, you could imagine a world where Emacs did way more awesome stuff on Windows and Mac, and the ensuing HN discussion where the obvious snipe appears a dozen times: "lol they keep talking about free software but their own products work better on windows lol".

FSF is not political in the sense that they won't forbid you to use their software or otherwise blacklist you even if you are against free software, or if you're far left or far right. They are only political in the sense that they have a goal. If that's how you define "political" then the word is somewhat meaningless.

They won't take contributions not aligned with their goals but won't block you from sharing your contributions in the spirit of free software.

It was a while ago (20-ish years?) and I'm forgetting the details, but it was RMS and I think the package was emacs.

But... that software (reaction) is not written by the FSF? They just use it

1. GPL is extremely and intentionally political 2. anti-colonialism is not a trait of culture 3. the state of being gay/trans is not political 4. culture != politics 5. is != ought

Of course people can have their preferences, but presumably that reaction implies "gay/trans anti-colonialist" ought to be a problem for some "political/cultural" groups?

What is the aim of that?

My read is that the authors are signalling their community is a "safe space". I don't think they're trying to exclude particular cultures, trigger people, or otherwise cause problems.

You can always ask anyone not to use anything. Doesn't mean they have to listen, but you can still ask.

Because signalling has replaced real virtue.

I'd like to suggest that this messaging is actually the inverse of virtue signaling, I.E. "icky signaling" for snowflakes. Keeps people at bay who are icked by the message.

The aim is to reduce the number of users of the software who are uncomfortable with those who are gay, trans, and/or anticolonial, probably because dealing with such people is a heavier burden than the other kind.

You wouldn't know my stance on the matter based on a pull request, especially not if some author didn't plaster it all around their profile.

Then you don't care. People that care tend not to do a PR.

How would a you even know whether a user was uncomfortable with any of those things? Why would someone with a particular political stance be a heavier burden on maintainers? How would you even know how someone felt - if someone reports a bug it is highly unlikely they are going to add something like "I am uncomfortable with gays" are they? Nor is it going to be in the comments in contributed code. It sounds more like that the maintainers are uncomfortable with people who are not like themselves.

You seem to be drastically overcomplicating this. They are asking people who are uncomfortable with it to not use it. The people who are uncomfortable with it are the ones deciding, nothing at all in there implies that they are deciding who is uncomfortable.

Seems like it might be working!

My Gen Y brain can't read the phrase "[inanimate object] is gay" without interpreting it as disapproval.

My Gen Y brain can do this no problem, and I'm among the oldest of the generation. Anthropomorphizing is fun!

Functions are colored. Software has sexuality. And we aren't even talking about AI!

That's interesting. I haven't used fail2ban for a long time, but reaction is worth evaluating. Unfortunately, that post does not describe their full configuration. Maybe it's on purpose, so that attackers can't adjust to fit.

My experience is that modern web scraping had no obvious pattern, since it is proxied through many IPs. The last time a server was failing to handle the pressure, we decided to temporarily ban IPs from some Asian regions. How does the FSF decide to ban an IP?

Why do they use iptables + ipset instead of nftables? Is there a technical reason or is it just legacy? AFAIK, Nftables is more performant, and IMO simpler. And it has native sets, see https://wiki.nftables.org/wiki-nftables/index.php/Sets

> The last time a server was failing to handle the pressure, we decided to temporarily ban IPs from some Asian regions.

This is something we've been forced to do at work, a LOT. Some weeks it's Huawei Cloud, Tencent, and Alibaba. Other weeks it's all China Telecom. We're using Anubis where possible, but a lot of it is just whack-a-mole with residential proxies. I looked at Datadome and HUMAN, but they would be hundreds of thousands a year at our traffic scale, and I suspect may also have false positives. We abandoned CrowdSec for that reason as well.

I'd love to find a decent k8s native solution to this problem.

Prosopo could cut out a lot of the residential proxy nonsense for you. We integrate with lambda@Edge / cloudfront workers / server side and perform analytics to detect residential proxy networks - at far less cost than DD or HUMAN.

> I looked at Datadome and HUMAN, but they would be hundreds of thousands a year at our traffic scale, and I suspect may also have false positives.

I can confirm. DataDome has been making my life a living hell. It thinks my phone is a bot, so I can no longer use PayPal and other quasi-monopoly services while on the go. DataDome will reliably block me on the first request.

Fuck DataDome.

iptables has been mostly a wrapper for nftables for some time now. The choice of iptables + ipset with reaction is the difference in their configuration. Compare restart performance between the ipset and nftables example configurations with lists of greater than 1 million IPs.

It's somewhat interesting to see the FSF's approach to this. From what I understand they can't really use something like anubis since they want their websites to be accessible without javascript:

https://www.gnu.org/philosophy/javascript-trap.html

Users can't consent to running a page's javascript the way they can consent to running a program they've intentionally downloaded, so it's effectively "non-free" regardless of license.

> can't

Won't, they call it malware: https://www.fsf.org/blogs/sysadmin/our-small-team-vs-million...

Anubis does support the no-JS HTTP meta-redirect proof of work but few know about it and fewer enable it. And it may not block everything.

I indeed did not know about this. There seem to be some caveats:

https://anubis.techaro.lol/docs/admin/configuration/challeng...

I guess for the meta refresh challenge it's less "proof of work" and more "proof of patience".

The meta refresh challenge actually uses time as the proof! It makes sure you wait for at least 75% of the time the administrator configured and adds client side smear to ensure that browser temporal randomization doesn't trigger a false failure.

Does it still need a cookie though? Another thing I have disabled by default.

For what it's worth, Anubis supports LibreJS: https://github.com/TecharoHQ/anubis/blob/main/web/build.sh#L...

> We placed our regular expressions in fail2ban, and found that we were hitting the maximum rules that could be added to UFW firewall rules on our systems which showed degradation around 65,000 rules

Firewalld had a similar issue up until recently as well.

https://man.netbsd.org/blocklistd.8

>Popa botnet

It's no more of a botnet than ProtonVPN for example. Apps intentionally added the Popa SDK to their apps as a monetization method. This allows apps without ads and tracking to be financially viable. I would expect FSF to support apps being able to move off of monetization schemes that depend on tracking people so it is disappointing for them to put such alternative monetization technologies in a negative light.

This monetization scheme benefits the botnet controller and the developer who added the SDK and not the user who likely did not realize they signed up to become an exit node.

It allows as free versions of apps to be economically viable and compete with others. It helps users because they don't need to be spied on and shown ads to fund the development of the app.

The existence of an app brings users value, else they wouldn't use it.

Recruiting your users' systems into a botnet is not an acceptable way to make an app "economically viable" any more than, say, installing a rootkit on their systems.

Is Google Maps a botnet because all of the clients share location data to make navigation more optimized? Having multiple users connecting back to a central server does not make something a botnet. Users should be able to decide for themselves whether they want apps like this or not.

The difference of course is that Popa does not provide 1. informed consent to users and 2. actual useful app functionality. The user does not know when an application is using the Popa SDK and does not know their device has been compromised. It is bundled inside seemingly legitimate software without the user's knowledge. Therefore, the user cannot make an informed decision about whether to use the app. Also, unlike the Maps example, the botnet provides no feature that the user would want or choose.

Popa is an illegitimate and botnet[1] that no end user knowingly/willingly uses.

1: https://krebsonsecurity.com/2026/07/fbi-seizes-netnut-proxy-...

I would not have as big of a problem with it if it was a transparent arrangement that the app is free of cost, but the Internet connection would be shared with customers of RoboVPN. The issue is that the often invisible arrangement is not obvious and likely breaks the terms of service with the Internet Service Provider. If the user was intentionally running a TOR node, I would not have a problem with it. The current reality of it is absolutely unethical and breaking the Internet as we know it.

I agree that users should be informed, but if parts of the internet are breaking due to user anonymity it is hard for me to feel bad for those sites.

the fsf has never really been concerned with commercial viability. They're the worst audience for this sort of argument.

and I doubt these apps are really Free versions - do they support user modifications and access to the code? If they did support the four freedoms maybe the fsf would have something positive to say to balance it out?

You are not wrong. The FSF often takes an absolute stance on these things where they don't properly support partial steps towards more free computing.

"Monetization". What a horror.

I pay for some software services. The services I pay for have a billing page (or a donation page) and I pay via the banking system

I rigorously block every ad, every tracker, every thing that does "monetization"

The evil period of trying sneaky ways to generate money is, I am optimistic, coming to an end.

If you want my money, ask me. If you must have my money, demand it. If you are sneaking around "monetization" I will do everything I can to stop you.

Yeah but they don’t want your money. They want the botnet users money.

You got a genuinely free, to you, app.

"Many sysadmins know about fail2ban..." and many will now know about reaction. But why will the result be any different than fail2ban? It won't.

I identify features (which can be expressed as firewall rules) from log data; I write totals to a temporary store (Redis). I have periodic tasks which scan the temp store for patterns which exceed thresholds. When that occurs, fail2ban creates the appropriate rules. This occurs in depth and in concentric rings.

Et tu?

The difference between fail2ban and reaction is performance. If you are not hitting the ceiling of fail2ban, then you may not need reaction.

Do you have a blog post about your automated fail2ban rule generation?

Why should I have a blog post? I get amazing "performance" out of fail2ban, it does what it's supposed to do and I don't ask it to do more. I've given it "super powers".

Last year I blocked basically all of Brazil. No problems. Who knew?

are scrapers attackers?

I get they're DDoS; but take the mask off, and arn't they just the AI monied interests that fund the FSF? and a lot of them are just active inference, eg, the user is trying to ask about something and the AI monied interests setup a web scraper to go and get that data.

Just seems like no one wants to call out the hand that feeds them in a human centipede that's best described as the torment nexus.

instead of scraping then, they could pay the fsf for a dump of the site or some API access or something, right? why overload the servers normally.

That's what commoncrawl does.

common crawl pays the sites they crawl?

> AI monied interests that fund the FSF

Can you elaborate on who these interests are precisely?

I tried: https://www.fsf.org/patrons; the last FY listed is 2020.

I'm not entirely sure how those companies are related to "AI-monied" after clicking on their websites.

the point is i tried to answer and the page is 6 years out of date....

The point is that it is presumed that you must've gotten the info somewhere in order to back up your claims. If not this outdated website, where then?

So you made shit up.

The patrons page is updated regularly.

Anything which fills my logs with garbage is unwanted. Your cat could have fallen asleep on the keyboard, I don't care. If you want to use the internet as a giant petri dish, that's on you; but the cat box is elsewhere. I can feed you garbage or block you because your code is shit, or I don't like your style.

Reply on news.ycombinator.com