Just a Rant: Moral Hazard for Good (TCP Tuning as an Example) | Hacker News Reader

Just a Rant: Moral Hazard for Good (TCP Tuning as an Example)

6 pointsm30477 comments

MORAL HAZARD FOR GOOD: I say this and people think I'm summoning flying monkeys with horns.

The simplified formula for the successful delivery rate of a SYN/ACK depending on the loss rate (assuming the loss is the same for send and receive) is:

n_syns * n_synacks * (loss^2)

With a loss rate of 0.5 (and delivery rate of 0.5) 2 SYNs are sent and 1 SYN is received and 2 SYN/ACKs are attempted and 1 gets through:

2 * 2 *0.25 == 1

The default on this Linux box is 5 SYNACK attempts for every SYN received:

# cat /proc/sys/net/ipv4/tcp_synack_retries

5

That's 5x amplification during a SYN reflection attack!

Like I've said elsewhere, I've set it as low as 2 in prod, and I would consider 1. (People looked at me like the flying monkeys were gonna arrive at any moment, but with a little whistling we got past the graveyard.) The attacker has a geometric advantage. Emotionally thinking that you're gonna increase your SYNACKs for good is a fool's errand that only hurts you and some anonymous victim: you have to /geometrically/ increase it to offset the attacker. Thinking this way is the usual outcome of moral hazard.

Moral hazard for good is embracing hostility and reducing SYNACK retries to something like 2... if not 1... and assuming that somebody who really wants to connect will selfishly up their initial SYN attempts correspondingly. So the sender is prepared to send up to 4 SYNs losing 2 to attrition (and if they send them all at once without backing off or waiting for responses then they're jackasses and should be dealt with accordingly), and the SYNACK sender sends 2 losing 1 to attrition and the result is the same. So sad for that client having to send 4 SYNs instead of 2, my heart is breaking at the cruelty of teh netz!

It's that simple: moral hazard can be a force for good and you owe teh netz nothing. There's no mathematical reason to spread the risk around; all it does in practice in the face of an adversary is to cause more damage and spread that around.

If there is a question lurking here, it's whether tcp_synack_retries is for each SYN successfully received or whether it's for the connection. If I find some free time maybe I'll set up a VM and use Scapy to fire some SYNs at it.

People looked at me like the flying monkeys were gonna arrive at any moment

It might help to make a diagram for them using load testing and spoofing tools. Then attack a stand-alone version of your company code and show the differences before and after.

For what it's worth, I have always set synack retries to 0 on my web, smtp and dns servers. I leave it up to the client applications to retry. This is just my hobby crud but I agree 5 is a risky default that can amplify attacks. I have never run into problems setting it to 0 but I do not have the most all-inclusive set of applications to test. Most companies put their applications behind a load balancer that has some SYN protection enabled by default and that is probably why it's less of an issue.

As a side note I have also found that some of the syn-spoofing-flooding tools lack some TCP options that are very easy to filter within iptables modules such as missing or odd values for MSS in syn packets, odd tcp syn lengths, etc... One example from my raw table:

    -A PREROUTING -i eth0 -p tcp -m tcp --syn -m tcpmss ! --mss 1220:1460 -j DROP

Most home connections will be 1460. Some Cisco VPN's are 1454 or 1424. Cell phones using ipv6ipv4 gateways are sometimes 1280 or 1380. There are other less common variations such as people using jumbo frames ranging anywhere from 9036 to obscenely high numbers. The most common scanning tool "masscan" does not even set MSS at all. Same for hping3 and a few other tools.

Another obscure one in the raw table I use when I do not need or expect cell phones to hit a port and this happens because most cell phones enable TCP timestamps which usually bumps their TCP SYN length to 60. Why cell phones? Modern bot farms seem to be using many old cell phones to launch their attacks and I have no idea why.

    -A PREROUTING -i eth0 -p tcp -m tcp --dport 1:442 --syn -m length --length 58: -j DROP

As a more obscure note, there was a time I increased syn/synack retries beyond the default and that is when we used NFS Diskless in our production datacenters. But that's a much longer story and a much older kernel.

> syn-spoofing-flooding tools lack some TCP options

Four years ago my hobby server on a DSL link got attacked in a SYN reflection attack and it went on for a long time. I took the "usual measures" and it made no difference. (How much nutritional value are they getting here?) My upstream was aware, and had checked to make sure it wasn't inside their DC.

It became a diversion to investigate out of boredom. Finally, statistical analysis of TTLs revealed a pattern. I wrote a Python script to reveal the pattern (gotta love Scapy!) and make some guesses about the actual number of hops and sent that to my upstream. They sent it to their upstream. A few weeks later the Chinese arrested a few people.

A few weeks later the Chinese arrested a few people.

That's impressive. I've never been able to get their ISP's to respond to me, even after having coworkers write emails in traditional Chinese. The only time I had any success was contacting the CCP and suggesting someone was spreading democracy and talking bad about them and those networks would go silent but they would still never reply to me.

I won't crame credit, as it would be foolish to do so. If they were worth arresting, they were probably annoying people a lot more important than me. Nice coincidence though, and happy to help.

> Another obscure one in the raw table I use when I do not need or expect cell phones to hit a port and this happens because most cell phones enable TCP timestamps which usually bumps their TCP SYN length to 60. Why cell phones? Modern bot farms seem to be using many old cell phones to launch their attacks and I have no idea why.

I was pretty sure Mac OS enables TCP Timestamps by default. The only platform that doesn't tend to is Windows? AFAIK. TCP Timestamps are useful for Protection Against Wrapped Sequence numbers which is sometimes relevant, and often used to get round trip time estimates (although there's lots of other ways for that too, so IMHO, kind of a waste of bytes)

The only platform that doesn't tend to is Windows?

I believe that is correct. It is an obscure thing to even consider and prone to blocking legit systems but useful if one has Ansible scripts to enable "Threatcon Alpha" so to speak when under an attack and even then I only use that on specific ports. It's less common these days for any of my nodes to take on attacks as I mostly share my unpopular opinions here now.

Another Windows vs Linux/Mac default is the packet TTL. Linux/Mac/BSD is 64 and Windows is 128. This is only useful in TCP as it decrements in UDP but useful to see some bots that set really high TTL's. There is an IPTables module to make decisions based on TTL.

I'm not super familiar with Linux's syncookies, but if being used as for a reflection attack, you're probably into syncookie territory, and that gets you down to one syn/ack per syn anyway.

I guess if the attacker is sending a low number of SYNs to get the reflection, you could still hit the the 5 retries (is that actually 5 SYN/ACKs or the original + 5 retries?). But then again, if it's a low number, does it make a big difference? Maybe, but there's a lot bigger reflection vectors out there.

Reply on news.ycombinator.com