mTCP: A high-performance user-level TCP stack for multicore systems

I should be more clear: From my perspective as someone following this closely little progress has been made. With the notable exception of io_uring.

Which is exactly what Mellanox and others are doing. At QuasarDB, we’re using their drivers to bypass the kernel entirely, which (at scale) saves tremendous amounts of CPU load.

These user-space TCP stacks usually use an user-space driver or a driver that minimizes system calls with packet aggregation. For example DPDK.

> But if you’re willing to do that, then the application is basically acting as its own kernel... so why not just admit that, and instead of writing a user-space process that has half the features of a kernel, just either 1. write your logic as a Linux kernel driver, or 2. compile your program into a unikernel framework?

Because the kernel is still a lot of other things for you other than networking - I think it's a stretch to say that all user-space networking makes your work "half" of that of a kernel. And, you're not necessarily the one doing it. You may be an application, and your user-level TCP (including kernel bypass) may be a library from someone else. But to your general point of now you are now well past the city walls, and may run into trouble, I agree. I assume that this sort of thing is only done by a small number of people.

Or BeOS or DragonflyBSD.

I'm curious too, as the ones I've seen show Linux outdone by Dragonfly, matching throughput at much lower latency.

https://leaf.dragonflybsd.org/~sephe/perf_cmp.pdf

As I recalled, you wrote channel instructions, or chains of channel instructions, as "CCW"s (Channel Command Words), which you dispatched to the channels via a "SIO" or "SIOF" instruction (Start I/O, or Start I/O Fast).

Once the operation started, the SIO(F) returned to you and the channel operated asynchronously from the CPU, moving the data directly in or out of memory. I think SIOF returned as soon as the channel received the instruction, whereas SIO waited for the instruction actually began.

So the channel actually added as sort of co-processor with the same access to memory as other CPUS. I'm pretty sure the channel could use virtual addresses, but the memory of course had to be fixed (pagefixed) during the operation.

I don't know much about GPGPU-land, but all of the difficulties I foresee have to do with multiprocessing/sharing/context switching... my guess is that only one logical program can use the scratchpad at a time?

> Why can't I have fast, low instruction cost communication between any two CPUs, regardless of which core it's on?

Fizzics, specifically a mixture of C and some electrical crap about signal propagation which I don't unnerstand. Einstein, he say no.

> The RISC revolution was in part

true I suppose. Even dumber things were reputed to have been done in the design of the SPARC chips.

> Is it really worth sacrificing an entire cache line to do it?

First show me where this is mentioned in the doc. Second, yeah, why not. If you can pass 512 bits of info between threads in a few cycles, sounds ok to me. If you want another mechanism, be prepared to pay for it. Memory has been around for ages, it's well understood, and bloody fast. It'd be stupid not to use it.

Look, you can't just have stuff cos you want it. You've given no reason for needing it, or that you understand the tradeoffs at the hardware level, or any appreciation of the blinding speed of current chips. How would faster XYZ improve your life?

Go and read <https://www.amazon.co.uk/Computer-Architecture-Quantitative-... then maybe you'll understand your plaint is falling on uninterested ears (mine anyway). Suggest something constructive based on the realities of hardware then I'll talk.

> Why can't I have fast, low instruction cost communication between any two CPUs, regardless of which core it's on? In other words, why do I have to do that bookkeeping to have super fast communication between threads/actors?

L1 cache is literally inside of the core of modern chips. You can't beat its speed because its physically the closest speed possible.

Moving cache further away (ex: L2 or L3) means more wires, more capacitance, and slower electricity. Electricity doesn't move instantly: it has to fight the capacitance of wires. The wider the wires, the more electrons you need to put onto the wire before its voltage changes. There's also inductance: the longer the wire is, the longer you need to hold a voltage before electrons actually decide to move.

Second: L1 caches do a LOT of book-keeping for you. All cores on modern systems (anything more modern than DEC Alpha from the 90s) will have cache coherence automatically.

That's downright huge, and modern multithreaded programming would be damn near impossible without the ordering guarantees of basic cache coherence. (yes: even ARM / MIPS and their "relaxed memory model" are more coherent than chips from earlier eras).

So first of all, be gracious about the level of memory ordering available to the modern programmer.

------------

So how are caches coherent? Most modern caches use some form of MESI to communicate with each other (invisibly: this is "below" assembly language and is happening at the hardware level).

When one cache claims "exclusive access" to the data, no other cache will write to it (although they may read stale versions of that data from their own cache). When that core / L1 cache is done, it will "invalidate" the data.

Modern cache coherence protocols can go faster: MESIF for Intel means that writes may be (F)orwarded to other caches automatically (which is slightly faster than (E)xclusive -> (I)nvalid -> (E)xclusive in the 2nd core).

Again, this is all done automatically, because programmers want to have their data look the same from all cores (for some definition of "looks the same").

This does NOT mean that the data is in the correct order however (!!). L1 caches can change the order of reads and writes before the data is shipped to other cores. Indeed, this is the very POINT of L1 cache. If one core is doing a while(true) iter++; loop, its more efficient to merge the writes together and "batch" them out slowly. Ex: Core#2 may see "1, 10, 33, 600" each time it reads the value of "iter".

Indeed, the most efficient implementation of this while(true) iter++; loop is to have "iter" be a register private to a core that has absolutely no communication to the outside world at all.

Why do you need faster communication mechanisms? Unless you're "burning" an entire core via busy waiting which costs some performance you will never reach the latency limits of the cache. Your bottleneck will not the communication mechanism, rather the bottleneck will be that the receiver isn't processing the messages fast enough. If your problem is really that time critical then using a CPU was a huge mistake in the first place. What you want is an FPGA.