Obscure kernel settings to increase performance of Linux TCP servers | Hacker News Reader

Obscure kernel settings to increase performance of Linux TCP servers

tldp.org

47 pointsfosk17 comments

In general you don't want to screw around with kernel settings. You're just as likely to break something as you are to improve anything. You need to know what you are doing when you venture into this territory, and if you don't, there's not enough info in this post to really guide you.

Ideally, thats why you tune on a system that you can "break", in an environment that is replicating the bottleneck/issue you are trying to improve/fix.

Tuning can be fun. Almost as fun as fixing obscure problems with modem AT strings. ;)

The one I have been bitten by is the TCP retransmission timeout (RTO). This is the time TCP will wait for an ack before resending some data. This number has to be somewhat conservative because if the RTO is less than the normal round trip latency TCP will be constantly resending packets.

Normally the RTO is set to 200ms. This is conservative in case you have long distance TCP connections. However when your systems are in a same data center environment where round trip times are sub-millisecond the default is two orders of magnitude too high and should be tuned down.

Looks like incast. Here's an in-depth article http://www.snookles.com/slf-blog/2012/01/05/tcp-incast-what-... Check out these patches extracted from the original paper site for 2.6/3.0rc kernel lines https://github.com/vrv/linux-microsecondrto

I believe this list is up to date as of July 2002.

Worse than that even.

> The default is 0, since this feature is not implemented yet (kernel version 2.2.12).

2.2.12 was last modified in August of 1999. 2.2.14 was released by January 2000, so it's not like 2.2.12 was around forever.

Don't go screwing around with your kernel settings based on this document.

That might explain '/proc/sys/net/ipv4/icmp_echo_ignore_broadcasts', most modern Cisco routers now have 'no ip directed-broadcast' enabled by default, at least for the past 4-5 years, blocking this at the router level.

These all seem prime candidates for some kind of learning algorithm (at the very least a naive one). Like TCP's window size scaling, just for settings that are too complex to bother with tuning.

You'd have a risk of falling into a trap of a local maximum, especially with a naive algorithm. It's pretty much the cause of buffer bloat - everyone adjusting the buffer sizes to handle current load without losing packets. Which makes things worse globally of all systems do it. For a learning algorithm to work, I think it would have to be a distributed learning system in your whole network (or multiple connected networks).

At the same time, the default settings have been tweaked by domain experts over decades to be reasonably good for the general case. Not everyone might be interested in shoving a a complex, un-tested, un-debugged learning algorithm into their kernel. Still, interesting proposal . . .

You wouldn't necessarily have to run it in-kernel, as these are all configurable from /proc. With a sufficiently complete test environment, you could use machine learning to tune a lot of the params up safely, then just run the resulting config in production.

    > the default settings have been tweaked by domain experts
    > over decades to be reasonably good for the general case.

Yes, but look at default configurations such as those of Apache and Mysql - you'll find that the default settings have been finely honed for environments that were common ten years ago... Even settings need updating - optima are moving targets.

I think that happened: http://web.mit.edu/remy/

Thanks, very interesting!

As said elsewhere.... don't adjust the "richness" controls on your kernel unless there's an actual scalability problem costing you $$. Your special "tweaks" will end up using more memory or slowing performance most of the time. sysctls should be sensible by default on RH (cent/sci/cern/aws) and Ubuntu based kernels.

The two most common encountered problems are incast TCP collapse and slow clients.

https://everythingisdata.wordpress.com/2009/09/25/fine-grain...

http://www.pdl.cmu.edu/Incast/

http://conferences.sigcomm.org/sigcomm/2009/workshops/wren/p...

(For TCP testing: iperf... remember to account for protocol overhead, bits/bytes)

Handling slow clients is app-specific, so there's no general advice other than don't get clever. Clever -> !debuggable -> !maintainable -> !scalable.

I don't know where the poster's "performance" came from in the title, the TLDP doc just calls these "obscure settings".

Actually it seems someone just copypasted the whole set of sysctls from the kernel ip-sysctl.txt, as the list includes the most common settings such as net/ipv4/ip_forward, net/ipv4/tcp_keepalive_time and net/ipv4/ip_local_port_range.

This is worth noting apart from the crowd:

/proc/sys/net/ipv4/tcp_abort_on_overflow

because figuring out when you need it is awful.

Reply on news.ycombinator.com