[flagged]
For example, look at these benchmarks from 2003[1]. Makes you wonder where the myth comes from.
The newest benchmark I could find[2] points in the same direction.
Does anyone have more recent data?
[1] http://bulk.fefe.de/scalability/ [2] https://matteocroce.medium.com/linux-and-freebsd-networking-...
Networking isn't a one size fits all thing - different networks have different needs, and different systems in any network will have different needs.
Userland networking is great until you start needing to deal with weird flows or unexpected traffic - then you end up either needing something a bit more robust and your performance starts dropping because you added a bunch of branches to your code or switched over to a kernel implementation that handles those cases. I've seen a few cases of userland networking being slower than just doing the kernel - and being kept because sometimes the what you care about is control over packet lifecycle more than raw throughput.
Kernels prioritize robust network stacks that can handle a lot of cases good enough. Different implementations handle different scenarios better - there's plenty of very high performance networking done with vanilla linux and vanilla freebsd.
This paper suggests, as I would have expected, that Google uses userland networking in strategic spots where low-level network development is important (SDNs and routing), and not for normal applications.
An example of how Linux TCP is unsuitable for datacenters is that the minimum RTO is hard-coded to 200ms, which is essentially forever. People have been trying to land better or at least more configurable parameters upstream for decades. I am hardly the first person to point out the deficiencies. Google presented tuning Linux for datacenter applications at LPC 2022, and their deck has barely changed in 15 years.
Later
Also, the idea that QUIC is a concession made to intractable Linux stack problems (the subtext I got from that comment) seems pretty off, since the problems QUIC addresses (HOLB, &c) are old, well known, and were the subject of previous attempts at new transports (SCTP, notably).
Google/Amazon "etc;" are likely happy to pay the cost because it really is "good enough" and the benefits of Linux over FreeBSD are otherwise quite considerable.
Google in particular seems blissfully happy to literally throw hardware at problems; since hardware is (for them especially) fundamentally extremely cheap.
Even multiple percentage gains in throughput are not necessary for most applications, and Linux is decent enough with latency if you avoid having complex IP/NFTables rules and avoid CONNTRACK like the plague.
as u/jeffbee says anyway, most of the larger tech companies these days are using userland networking and bypass the kernel almost completely for networking.
This is true, to a point.
Counterpoint: Windows Desktop Experience.
EDIT: that comment was glib, let me do a proper counterpoint.
Common area's are some of the most least maintained in reality; I can think of meet-me-rooms or central fibre hubs in major cities; they are expensive and subject to a lot of the whims of the major provider.
Crucially, despite large amounts of investment, the underlying architecture or infrastructure remains, even if the entire fabric of the area changes around it. Most providers using these kind of common areas do everything they can to avoid touching the area itself, especially as after a while it becomes very difficult to navigate and politically charged.
Fundamentally the architecture of Linux's network stack, really is, "good enough", which is almost worse than you would originally think since "good enough" means there's no reason to look there. There is an old parable about "worse is better" because if something is truly broken people will put effort into fixing it.
Linux's networking stack is fine, it's just not quite as good an architecture as the FreeBSD one. FreeBSD one has a lot less attention on it but fundamentally it's a cleaner implementation and easier to get much more out of..
You will find the same argument ad infinitum regarding other subjects such as Epoll vs IOCP vs kqueue (Epoll was abysmally terrible though and ended up being replaced by IO_URING, but even that took over a decade)
Linux is not your "random on the side" feature that is good enough.
Especially since you don't even know what you're attempting to optimise for.
Latency? p99 of linux is fine, nobody is going to care that the request took 300μs longer. Even in aggregate across a huge fleet of machines waiting an extra 3ms is totally, totally fine.
Throughput? you'll bottleneck on something else most likely anyway, getting a storage array to hydrate at line rate for 100GBPs is difficult and anyway you want to do authentication and distribution of chunks and metadata operations anyway? right?
You're forgetting that it's likely an additional cost of a couple million dollars per year in absolute hardware to solve that issue with throughput, which is, in TCO terms, a couple of developers.
Engineering effort to replace the foundation of an OS? Probably an order of magnitude more. Definitely contains a significant amount more risk, and the potential risk of political backlash for upheaving some other companies workflow that is weird.
Hardware isn't so expensive really.
Of course, you could just bypass the kernel with much less effort and avoid all of this shit entirely.
Meanwhile, Google had a building full of people maintaining the Google kernel (eg, maintaining rejected or unsubmitted patches that were critical for business reasons), and it took many months to do a kernel release.
However there is some public information on some components that has been shared in this thread which allows you to draw your own conclusion.
And making that assumption when there are many public examples of Google doing this in other areas such as gVisor and Netstack?
Which is one of the arguments of why to do it that way; instead of using general purpose networking.
I haven't the time or inclination to find anything public on this, nor am I interested really in convincing you. Ask a former googler.
Per the Snap thread above: if you're building a router or a load balancer or some other bit of network infrastructure, it's not unlikely that there's userland IP involved. But if you're shipping a normal program on, like, Borg or whatever, it's kernel networking.
I continue in a similar position today and thus my NDA is still in complete effect which limits what I can say if there’s nothing public.
I have not worked for Google, just very closely.
Maybe it's all changed in the last year! But then: that makes all of this irrelevant to the thread, about FreeBSD vs. Linux network stack performance.
Which is why I said you "probably read it wrong".
Google is much happier to throw hardware at the problem in most cases, only when it really matters and they would have had to rearchitect the kernel to improve a situation any further do they break out the user-space networking.
The point I was driving at was that it's more common than you think.
Your base assertion that it's ubiquitous is very obviously false because Chromebooks are pretty common inside google offices and those are running stock chromeOS (except in the offices that are developing chromeOS)
That's all I'm here to say.
As these things go, in the course of making your argument, you made a falsifiable and, I believe, flatly incorrect claim:
Most of the larger tech companies these days are using userland networking and bypass the kernel almost completely for networking
At least in Google's case, this isn't true. People doing custom network stack stuff totally do bypass the kernel stack (sometimes with usermode stacks, and sometimes with eBPF, and sometimes with offload). But the way you phrased this, you implied pretty directly that networking writ large was usermode at Google, and while I entertained the possibility that this might be true, when I investigated, it wasn't (unsurprisingly, given how annoying user mode networking is to interact directly with, as opposed to in middlebox applications).
To answer: "Most of the larger tech companies these days are using userland networking and bypass the kernel almost completely for networking"
You could read "almost completely" as in "almost across the whole company" which would be a weird way to read it, but you seem to have read it this way.
I intended it to mean: when they bypass the kernel; it is a near complete bypass.
Since there actually is still a network connection going through the kernel (the host itself will still be connected of course), which is of course the inverse of what you seem to have taken away; in that user-mode networking is used even less than entirely even on a single node.
edit: In fact, I stated multiple times in my post: "Google is mostly happy to just throw hardware at this" which you seemed to just.. ignore? Google are absolutely happy to throw hardware at issues until they can't anymore or the gains are too enormous to avoid. I thought I was extremely clear about that.
Your point about user-land networking in FreeBSD is just a nonsense one to make -- and not the point we were discussing anyway, like suggesting "if it did rain beer, would we all get drunk?" it's completely hypothetical and not based in any subjective reality or objective truth. You have absolutely no way of knowing if FreeBSD could do those things, the statement that the architecture permits is is shown somewhat in FreeBSD's use in Netflix, which has been commented elsewhere in this thread to achieve close to 0.8TiB of data transfer, but knowing what google would have done with freebsd would require seeing into alternative realities.
I don't know where you work, but as far as I know: nobody has managed to perfect that technology yet.
This is a weird cursed thread that started out with a pretty silly† claim about FreeBSD vs. Linux network stack performance (neither of us started it; it's a standard platform war argument). Someone made a comment that the hyperscalers all depend on the Linux kernel stack, to a far greater degree than they do on FreeBSD. That's a true statement; at this point, you and I have both agreed on it.
When that point was pressed earlier, you and another comment brought up kernel bypass (usermode networking, specifically) as a way of dismissing hyperscaler Linux dependencies. But that's not really a valid argument. Hyperscalers do kernel bypass stuff! Of course they do! But they're doing it for the things that you'd formerly have bought dedicated network equipment for, and in no case are they doing it in a situation where deploying FreeBSD and using the FreeBSD stack would be a valid alternative.
The disconnect between us is that I'm still talking about the subject that the thread was originally about --- whether Google using the Linux stack is a valid point backing up its fitness for purpose. I think it pretty clearly is a valid point. I think the usermode networking stuff is interesting, but is a sideshow.
† "Silly" because these kinds of claims never, ever get resolved, and just bring out each side's cheering section, not because I have low opinions of FreeBSD's kernel stack --- I came up on that stack and find it easier to follow than Linux's, though I rather doubt the claim that it has a decisive performance advantage.
I agree that we agree on many points; but I think where we diverge (and perhaps fundamentally) is in the base assumption that "because google uses it, it must be the best, because even if it wasn't google would make it so" (at least, this is my interpretation of GPs comment).
I have little doubt that the low hanging... mid hanging and perhaps even most of the high-hanging fruit has been well and truly plucked when it comes to linux throughput at the behest of the large tech companies; because a few percent improvement translates a lot at their scale. -- However I am reminded of an allegory given in a talk (that I can't find) regarding bowling.
In the talk the speaker mentions how they got "really good" at bowling with completely the wrong technique; but it worked for them, up to a point in which they could not improve no matter what they did. They had to go back to the basics and learn proper technique and become much worse before they were able to overtake their previous scores with the bad technique.... but after that point there were further improvements to be had.
My argument that this is the case is merely: doing an architectural re-write of the linux kernel to be more scalable in the way FreeBSD's is would be very punishing for too many people, and additionally that the economics are not favourable when, if you do get to a point where you cannot scale due to the kernel, you could just break out into userland. -- then simply suffer the adequate but not insane performance everywhere else where it's not needed anyway.
So, to summarise my points:
* Because a big company uses something does not mean it is perfect in all areas
* That Linux has a lot of attention on it does not mean necessarily that it has the most potential: though I don't doubt that the majority of it's potential has been reached.
* Diminishing returns means once it's "good enough" people will try to get performance elsewhere if they need it.
* Rewriting the network stack in Linux completely would likely be harmful to many and subtly so, I haven't seen people moving towards this idea either, this feels like it could be political as well as technical.
* Hyperscalers will often trade convenience over performance: regardless, CPU time is much cheaper for them than it is for us.
Yahoo switched to Linux because having a single server OS is nicer than having two, acquisitions were nearly all running Linux, and Linux was at least good enough (although having all of Overture crash at the same time because of a leap second bug in the Linux kernel, twice, wasn't great), and maybe something about easier to hire for kernel engineers.
WhatsApp switched to Linux because the server team was mostly ex-Yahoo and had seen how much strife running a different OS than your acquirer in their datacenters causes. There was enough non-negotiable strife from all of the other tech and philosophy differences, that accepting the acquirer kernel that is at least good enough was worthwhile.
I'm not saying FreeBSD or Linux has better performance. I like the reputation FreeBSD has for network performance, and it certainly has good performance, but is it better than Linux? I don't know, I never ran apples to apples comparisons, because whenever I was involved in a switch, the hardware was going to be very different, and the changeover was a policy decision rather than a technical one.
Personally, I run FreeBSD when I can, and Linux when I have to. FreeBSD's development model and smaller team lead to less churn, and I value stability and consistency. Other people have different values, and that's fine too.
I would expect in the 1% to 10% bandwidth utilization, on average. From my vague recollection, that's what it was at FB when I was there. They put stupid amounts of network capacity in so that the engineers rarely have to think about the capacity of the links they're using, and that if their needs grow, they're not bottlenecked on a build out.
To answer the original question, it's complicated. I have a weird client where freebsd gets 450 MiB/s, and Linux gets 85 with the default congestion control algorithm. Changing the congestion control algorithm can get me between 1.7 MiB/s and 470 MiB/s. So, better performance... Under what circumstances?
Talking generally about "network performance" is approximately as useful as talking generally about "engine performance". Just like it makes no sense to compare a weed-eater engine to a locomotive diesel without talking about use case and desired outcomes, it makes no sense to compare "performance of FreeBSD network stack" and "Linux network stack" without understanding the role those systems will be playing in the network.
Depending on context, FreeBSD, Linux or various userland stacks can be a great, average, or terrible choices.
Linux is a networking swiss army knife (or maybe a dremmel). It can do a lot of stuff reasonably well. It has all sorts of knobs and levers, so you can often configure it to do really weird stuff. I tend to reach for it first to understand the shape of a problem/solution.
BSD is fantastic for a lot of server applications, particularly single tenant high throughput ones like mail servers, dedicated app servers, etc. A great series of case studies have come out of Netflix on this (google for "800Gbps on freebsd netflix" for example - every iteration of that presentation is fantastic and usually discussed here at least once, and Drew G. shows up in comments and answers questions).
It's also pretty nice for firewalling/routing small and medium networks - (opn|pf)sense are both great systems for this built on FreeBSD (apologies for the drama injection this may cause below).
One of the reasons I reach for linux first unless I already know the scope and shape of the problem is that the entire "userland vs kernel" distinction is much blurrier there. Linux allows you to pass some or all traffic to userland at various points in the stack and in various ways, and inject code at the kernel level via ebpf, leading to a lot of hybrid solutions - this is nice in middleboxes where you want some dynamism and control, particularly in multi-tenant networks (and thats the space my work is in, so it's what I know best)
Please bear in mind that these are my opinions and uses/takes on the tools. Just like with programming there's a certain amount of "art" (or maybe "craft") to this, and other folks will have different (but likely just a valid) views - there's a lot of ways to do anything in networking.
If you are looking to hit line rates with UDP, or looking to head well above ~1-10gbps with TCP, you're fast headed into territory where you likely need to move away from POSIX. (For a super dumb benchmark, oversized buffers amortizing syscall overhead might get you to 10gbps on TCP, but in a real application everything changes)
Once you're headed over 10gbps you'll quickly run into a hard need to retune things even for TCP, earlier if you're talking to non-local hosts. Once you're over 25gbps you're headed into the territory where you'll need to fix drivers, fix cpu tuning and so on. For a recent real world example: when we were doing performance analysis of our offloading patches for Tailscale we identified problems with the current default CPU frequency scaler for Intel CPUs on current kernels, and reached out to the maintainers with data.
for example:
- SO_BINDTODEVICE
- an option for not-symmetric NAT