Conntrack tales – one thousand and one flows

Are the weird edge cases easily fixable through some conntrack source patches? Or will they require major surgery to address?

Good intro. I can relate, I had to debug conntrack dropping packets and performance issues and it's not easy. Cherry on top, there aren't two distributions taking the same settings.

Also have a look into the conntrack hashsize. The conntrack connection table is a hashtable and the hashsize is how many buckets it has. This affect collisions, memory consumption and cache hits.

Then conntrack timeouts settings, that decide when connections can be released from the tables. Connections can be hold by default for minutes even after close. The most trivial performance test can cycle a million connections in a minute and put down any system using conntrack.

I wouldn't have guessed that cloudflare relied heavily on conntrack. It's not that scalable or easy to use IMO.

The conntrack(1) utility also has some fun brokenness. For example, there's a TOCTTOU issue when you flush connection state, as Calico and other k8s CNIs do regularly. The utility queries the kernel state, builds a list in user space, and then iteratively deletes each session. But if a session expires between when its reported and when the utility tries to delete it (for example, hitting the standard UDP session timeout, or a TCP FIN, which happens all the time), the delete fails with ENOENT and the utility immediately exits without flushing the rest of the state.

Calico and other controllers resolve this by calling conntrack(1) in a loop until success or a limit (e.g. 3, 5, or w'ever magic number) is reached. But even then you regularly get one more error than the limit, triggering alarms.

I submitted a patch last year that added a command-line switch for suppressing certain errors, but the netdev mailing-list is high traffic and I never got any feedback. I never expected my patch to be accepted on the first round; not even the approach. I can think of several other alternative ways to address the issue, such as not exiting on error until working through the queue, utilizing exit status codes for reporting the reason, etc. But it's unfortunate I got no feedback whatsoever considering it's a substantial pain point in the wild and a clear (IMO) defect in the implementation--because the error is inevitable and to be expected given the TOCTTOU race.

Of course, if I were writing a controller like Calico I probably wouldn't be shelling out to conntrack(1), but instead making the netlink calls in-process. But I'm not most people, especially in this particular case where everybody else manifestly treats conntrack(1) as the de facto programmatic interface.