A pair of memory-allocation improvements in Linux 5.13 | Hacker News Reader

A pair of memory-allocation improvements in Linux 5.13

lwn.net

114 pointschmaynard11 comments

This patch brings XDP RX to 4.2Mpps which is nice.

In 2015 high-speed networking was targeting 10G line rate, i.e. 14.8Mpps. Today the NICs are 100G (42Mpps), next year it will be 400G.

I'm not sure how the kernel is expected to follow with the pace. Those NICs will offer offloads, so static network functions should be accelerated without involving the CPU. Still, it seems software-based networking will never be able to catch up.

DPDK and userland solutions in general are able to scale better. At least for DPDK, depending on the workload an L2 forwarding function can do 14.8 Mpps on a single core, and supposed to scale linearly (so 4 cores for a 100G L2 fwd?).

However I've yet to see a TCP implementation able to scale linearly. DPDK in used generally for middleboxes, not endpoints, where scaling TCP has different challenges (handling a socket state with a single RSS dispatch vs. two RSS dispatch I guess).

So what does it say about the future of SDN? Will it be able to keep pace with hardware network functions? A pure kernel-based approach would be nice but it seems impossible. On userland side the industry is not there yet in open-source, as far as I know. There are probably proprietary stacks able to do this, but I don't know them.

There is nothing that user space solutions do that can not be implemented in kernel space. From what I understand, the Linux kernel networking stack is not optimised to process packets in bulk, which is the only way to deal with high speeds nics https://lwn.net/Articles/629155/

True, but we have a kernel networking stack with a certain interface and a rich feature set. The kernel cannot break that API.

In contrast, a new userspace stack doesn't need to worry about backwards compatibility or supporting every use case.

So yes, the kernel can do just as good, but it's harder.

This is probably the only way to have fast software in today's hardware.

Unfortunately the popular languages are all about single element at a time, and pointer chasing with the compiler having a hard time vectorizing operations and reorganizing data structures. I think most compilers don't/can't even do this because there are not enough guarantees. The only systems that actually do this are DBMSs.

> compiler having a hard time vectorizing operations and reorganizing data structures

Reminds me of a recent blog post on exactly this topic. Apparently the Jai language might make some progress on this point.

• https://blog.royalsloth.eu/posts/the-compiler-will-optimize-...

• Discussion: https://news.ycombinator.com/item?id=27010965

I hate to nit-pick, but 100GbE line rate is 144.8Mpps, and 400G line rate is 579Mpps

You're right to nit-pick! It is rather embarassingly wrong (10G -> 100G should be a pretty obvious 14.88Mpps -> 148.8Mpps...). Unfortunately it seems I cannot edit the comment so I'll leave the error for posterity ^^. Still, thanks!

Writing or using an existing custom allocator is common enough. Networking developer Jesper Dangaard Brouer must have looked into that approach and found it wanting for what the article describes as “The networking code could then grab a pile of memory, and quickly hand out pages as needed.”

What does the improvement described in the “Batch page allocation” section add that a custom allocator would be unable to? Or is it a widely-used pattern that many custom allocators can take advantage of? In that case, why put it in the kernel?

Etc.

The kernel allocates memory too, hence an improvement at that level benefits allocations across the board.

So the use case described by Brouer was for networking code in the kernel? The article explicitly mentions his use case as the ‘original’ motivation, regardless of whether that’s the primary use case.

Is the kernel then the primary place this is used now? Which places in the kernel need and take advantage of it?

The Sun RPC code had a loop where it allocated one page at a time and filled an array with the returned pointers. The article says there was a 500% performance improvement after replacing the loop with a single call to the batch allocator. (While RPC can be thought of as networking code, it is more than just network packet flow.)

There are probably other places where this will help. Now that the interface is available, developers can start trying it.

Reply on news.ycombinator.com