Hacker News Reader

Comment by jeffbee | Hacker News Reader

Very nice round-up of techniques. I'd throw out a few that might or might not be worth trying: 1) I always disable C-states deeper than C1E. Waking from C6 takes upwards of 100 microseconds, way too much for a latency-sensitive service, and it doesn't save you any money when you are running on EC2; 2) Try receive flow steering for a possible boost above and beyond what you get from RSS.

Would also be interesting to discuss the impacts of turning off the xmit queue discipline. fq is designed to reduce frame drops at the switch level. Transmitting as fast as possible can cause frame drops which will totally erase all your other tuning work.

View on HN

Thanks!

> I always disable C-states deeper than C1E

AWS doesn't let you mess with c-states for instances smaller than a c5.9xlarge[1]. I did actually test it out on a 9xlarge just for kicks, but it didn't make a difference. Once this test starts, all CPUs are 99+% Busy for the duration of the test. I think it would factor in more if there were lots of CPUs, and some were idle during the test.

> Try receive flow steering for a possible boost

I think the stuff I do in the "perfect locality" section[2] (particularly SO_ATTACH_REUSEPORT_CBPF) achieves what receive flow steering would be trying to do, but more efficiently.

> Would also be interesting to discuss the impacts of turning off the xmit queue discipline

Yea, noqueue would definitely be a no-go on a constrained network, but when running the (t)wrk benchmark in the cluster placement group I didn't see any evidence of packet drops or retransmits. Drop only happened with the iperf test.

1. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/processo...

2. https://talawah.io/blog/extreme-http-performance-tuning-one-...

Does C-state tuning even do anything on EC2? My intuition says it probably doesn't pass through to the underlying hardware -- once the VM exits, it's up to the host OS what power state the CPU goes into.

It definitely works and you can measure the effect. There's official documentation on what it does and how to tune it:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/processo...

Okay, so it looks as though it only applies to certain large instance types -- presumably ones which are large enough that it makes sense for the host to statically allocate CPU cores (or even sockets) to a guest. Interesting.

I suspect that the web server's CPU usage will be pretty high (almost 100%), so C-state tuning may not matter as much?

EDIT: also, RSS happens on the NIC. RFS happens in the kernel, so it might not be as effective. For a uniform request workload like the one in the article, statically binding flows to a NIC queue should be sufficient. :)

Reply on news.ycombinator.com