Linux network performance parameters
github.com
github.com
- net.ipv4.tcp_rmem ~ 6MB
- net.core.rmem_max ~ 1MB
So.. the tcp_rmem value overrides by default, meaning that the TCP receive window for a vanilla TCP socket actually goes up to 6MB if needed (in reality - 3MB because of the halving, but let's ignore that for now since it's a constant).
But if I "setsockopt SO_RCVBUF" in a user-space application, I'm actually capped at a maximum 1MB, even though I already have 6MB. If I try to reduce it from 6MB to e.g. 4MB, it will result in 1MB. This seems very strange. (Perhaps I'm holding it wrong?)
(Same applies to SO_SNDBUF/wmem...)
To me, it seems like Linux is confused about the precedence order of these options. Why not have core.rmem_max be larger and the authoritative directive? Is there some historical reason for this?
once you do SO_RCVBUF the auto-tuning is out of the picture for that socket, and net.core.rmem_max becomes the max.
It's pretty clearly documented @ Documentation/networking/ip-sysctl.rst
Edit: downvotes, really? smh
2. It's always valuable to provide further references, but I'd guess that down-voters found the "It's pretty clearly documented" phrasing a little condescending? Perhaps "See the docs at [] for more information."?
3. "Please don't comment about the voting on comments. It never does any good, and it makes boring reading."
Oh I didn’t realize this. That explains the switch in limits. However:
I would have liked to keep auto-tuning, but only change the max buffer size. It’s still weird to me that these are different modes with different limits and whatnot. In my case, I was parallelizing tcp and capping the max size would have been better, and instead varying the number of conns.
I gave up on it. Especially since I need cross platform user-space only, I don’t want to fiddle with these APIs that are all different and unpredictable. I guess it’s for the best anyway, to avoid as much per-platform hacks as possible.
> It's pretty clearly documented @ Documentation/networking/ip-sysctl.rst
I guess I need to step up my doc grepping game, cause it was quite hard to even find this on Google. I ran my own experiments to verify.
> Edit: downvotes, really? smh
Fwiw not me.
If you want to limit the amount of excess buffered data you can lower TCP_NOTSENT_LOWAT instead, which caps the amount that is buffered beyond what's needed for the BDP.
Just one thing to add regarding network performance: if you're working in a system with multiple CPUs (which is usually the case in big servers), check NUMA allocation. Sometimes the network card will be in one CPU while the application is executing on a different one, and that can affect performance too.
https://blog.packagecloud.io/monitoring-tuning-linux-network...
https://blog.packagecloud.io/monitoring-tuning-linux-network...
https://packagecloud.io/blog/illustrated-guide-monitoring-tu...
Over the last year, I was troubleshooting issues with the following connection flow:
client host <-- HTTP --> reverse proxy host <-- HTTP over Wireguard --> service host
On average, I could not get better than 20% theoretical max throughput. Also, connections tended to slow to a crawl over time. I had hacky solutions like forcing connections to close frequently. Finally switching congestion control to 'bbr' gives close to theoretical max throughput and reliable connections.
I don't really understand enough about TCP to understand why it works. The change needed to be made on both sides of Wireguard.
BBR tries to find Bottleneck Bandwidth rate. Eg, the bandwidth of the narrowest or most congested link. It does this by measuring the round trip time, and increasing the transmit rate until the RTT increases. When the RTT increases, the assumption is that a queue is building at the narrowest portion of the path and the increase of RTT is proportional to the queue depth. It then drops rate until the RTT normalizes due to the queue draining. It sends at that rate for a period of time, and then slightly increases the rate to see if RTT increases again (if not, it means that the queuing that saw before was due to competing traffic which has cleared).
I upgraded from a 10Mb/s cable uplink to 1Gb/s symmetrical fiber a few years ago. When I did so, I was ticked that my upload speed on my corp. VPN remained at 5Mb/s or so. When I switched to RACK TCP (or BBR) on FreeBSD, my upload went up by a factor of 8 or so, to about 40Mb/s, which is the limit of the VPN.
This was obsoleted by fast retransmit which was standardized in the 90s and ~everyone uses, right?
(loss generally is still used as a congestion signal, but first loss is usually not)
In particular, since Wireguard is UDP, using vegas over Wireguard seems to me like it should be good (based on a very limited understanding, though :/ ), it is just a question of how well it would work on the other side of the reverse proxy since I don't think it can be set per link?
Please stop. BBRv1 is broken and should not be used on the open internet.
This sort of copy-paste cargo-cult performance tuning (just set a magical value and things will be better) is the exact opposite of what TFA is about.
Thankfully Google are upstreaming BBRv3 so this will be over soon.
It's easy to coax BBR into converging on using 20% of a shared link instead of 50% (cohabiting with one other stream).
The inverse is true and it's easy to get BBR to hog 80% of a link instead of 50% (cohabiting with one other stream). If you're happy for other people to steal bandwidth from you with greedy CCAs then go ahead and ratelimit yourself. I'm not.
E.g. if a service runs on a single server (no CDN) and you occasionally get users from Australia then the site will by a bit laggy for them but at least it won't be a trickle.
Congestion control works from the sender of data to the receiver. You don't need to switch both sides if you are just interested in improving performance in one direction.
Besides that, I agree to what others said BBRv1. The cubic implementation in the Linux kernel works really nice for most applications.
On desktops, other than disabling features, can anything fix the problems with i210 and i225 ethernet? Those seem to be the two most common NICs nowadays.
I don't really understand why common networking hardware and drivers are so flawed. There is a lot of attention paid to RISC-V. How about start with a fully open and correct NIC? They'll shove it in there if it's cheaper than an i210. Or maybe that's impossible.
The PTP timestamps would arrive as a separate DMA transaction after the packet DMA transaction. It very possibly could have been poor integration into the ARM SOC, but your PTP-specific issue on x86 makes me wonder.
https://www.google.com/search?q=i210+proxmox+e1000e+disable
Most people don’t really use their NICs “all the time” “with many hosts.” The i210 in particular will hang after a few months of e.g. etcd cluster traffic on 9th and 10th gen Intel, which is common for SFFPCs.
On Windows, the ndis driver works a lot better. Many disconnects in similar traffic load as Linux, and features like receive side coalescing are broken. They also don’t provide proper INFs for Windows server editions, just because.
I assume Intel does all of this on purpose. I don’t think their functionally equivalent server SKUs are this broken.
Apparently the 10Gig patents are expiring very soon. That will make Realtek, Broadcom and Aquantia’s chips a lot cheaper. IMO, motherboards should be much smaller, shipping with BMC and way more rational IO: SFP+, 22110, Oculink, U.2, and PCIe spaced for Infinity Fabric & NVLink. Everyone should be using LVFS for firmware - NVMe firmware, despite being standardized to update, is a complete mess with bugs on every major controller.
I share all of this as someone with experience in operating commodity hardware at scale. People are so wasteful with their hardware.
Many systems which cost more than a good car are still coming with the Broadcom 5719 (tg3) from 1999. They have a single transmit queue and the driver is full of workarounds. It's a complete joke these are still supplied today.
SFP would be great but I'd settle for an onboard NIC chipset which was made in the last 10 years.
If you're willing to potentially sacrifice 10-20% of (max local network) throughput you can drastically improve wifi fairness and improve ping times/reduce bufferbloat (random ping spikes will still happen on wifi though).
There's a huge thread https://forum.openwrt.org/t/aql-and-the-ath10k-is-lovely/590... that has stuff about enabling and tuning aqm, and some of the tradeoffs between throughput and latency.
I bought a PCIe I350. That's solved the problem.
Brendan's System Performance books provide nice coverage on Linux network performance and more [1]. It's already in the second edition, both are excellent books but the 2nd edition focuses mainly on Linux whereas the 1st edition also include Solaris.
There's also a more recent book on BPF Performance Tools by him [2].
[1] Systems Performance: Enterprise and the Cloud, 2nd Edition (2020)
https://www.brendangregg.com/systems-performance-2nd-edition...
[2] BPF Performance Tools:
https://www.brendangregg.com/bpf-performance-tools-book.html
But can anybody help me out, who tunes linux network parameters on a regular basis?
net.core.rmem_max net.core.wmem_max
e.g. wireguard-go will hit those limits of not executed with CAP_NET_ADMIN.
It's older so some details have changed over time, but the concepts are still relevant. It also has a lot of useful search terms to get you started.
There lots on networking in general, but I've had a hard time finding some on Linux specific implementation
I want to try developing a simple tcp echo server for a microcontroller, but most examples just use the vendor's own tcp library and put no effort explaining how to manually setup and establish connection to the router.