Scipio: A Thread-per-Core Crate for Rust and Linux

Serial access to hot keys is a hard thing to design around, and you're right that sharding doesn't solve that problem. Worse, it exposes other keys that just happen to share the shard (or the core) to poor performance.

There are a couple of well-understood solutions to this problem. The obvious one is to dynamically re-balance shards based on heat, either moving some keys or split/merge. This is the same tactic that many distributed databases use, and while it's complex to do, it's easier to do locally than distributed. Another option is stochastic re-balancing, like the Stochastic Fairness Queuing (https://ieeexplore.ieee.org/document/91316) model used in networking. Here, shards are randomly re-shuffled occasionally. Doesn't fix the noisy neighbor problem, but does mean that the noisyness moves around. That might seem silly, but it's pretty much what's going to happen under the covers of the non-sharded version of the code when the scheduler gets involved, only easier to reason about.

> When that happens it can be a huge waste of resources and can give lower performance (depending on the type of load)

Lower apparent performance for neighbors of the heavy hitter, sure. Under which other circumstances does it reduce performance?

> Imho unless you are absolutely sure about the type of load, leave the sharding to dividing data between servers, or have some mechanism that can shift to sharing the load between threads if the system imbalance is too great

I'm a bit puzzled by this. Distributed systems have exactly the same problem, and solving that problem is much harder there because the cost of contention is higher, data movement is more expensive, and you have to deal with a lot more failure cases.

The statistics may be better for distributed systems because a hot tenant has to be a lot hotter to make hot box than a hot core. But that's a very specific kind of bet, and if you end up with a tenant that does cause a hot box you have an even harder problem to solve.

Dynamo (https://www.allthingsdistributed.com/files/amazon-dynamo-sos...) solves this by moving the key space around, as do many similar kinds of systems. It's not easy, though, and filled with caveats. If you're scared of sharding, distributed sharding should be scarier than on-box sharding.

It is possible to design data sharding architectures where balancing of both data and load across cores is continuous and smooth, with surprisingly minimal overhead and coordination cost. In fact, there are multiple ways of doing it, depending on your workload characteristics. At this point, the design idioms for this style of architecture are refined and robust, so there is no computer science reason the problems you raise need to exist in a real implementation. The reason it seems "difficult" in practice is because so many designs insist on loosely coupling and weakly scheduling storage, execution, network, etc. If your architecture concept is slapping a thin layer on top of RocksDB, it won't be feasible. Every part of the stack needs to understand and be designed to the model. The end result is actually quite elegant in my view, and with unmatched throughput.

People design distributed systems with simple static sharding schemes because they are obvious, easy, and it lets you cut a lot of corners on the rest of your system design. It is not the only way to design a distributed system, continuous adaptive resharding and load shedding is a demonstrably viable option, and it is much easier to implement within a single server than on an actual cluster of networked computers.

If the number of connections is not small, Scylla will crank up any other implementation with traditional locking.

> The solution to expensive context switches is cheap context switches, plain and simple.

Except the performance difference between a kernel-mode context switch and a user-mode one is only going to narrow in the future. The overhead that cannot be eliminated from context switches is their effect on the cache, since you start to run into the laws of physics at that point...

The real solution to expensive context switches is to just do fewer of them... No context switch is always faster than a "fast" context switch.

> I sincerely believe this is Rust's ballpark

I think it's plausible that Rust could get a library-level solution for fibers that does not rely on unstable details of the compiler. Rust will never again have that baked in to the language, as it would make the language completely unsuitable for many low-level tasks.

Fibers, especially the way they are implemented in Go, come with a lot of their own complexity.

Just look at this issue: https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/

This is just one of the segfaults caused by Go's complex stack control. I don't want to rely on a runtime that contains these sorts of bugs, and the best way to avoid that is to avoid having a runtime in the first place.

First, in Rust this isn't really a problem. You can always turn async calls into blocking ones in Rust by calling block_on [0]. In some languages block_on doesn't exist, like in in-browser js, because here, code is supposed to be async. But in Rust there is no requirement, so there's no colored function problem here.

Second, I don't think it's a big problem in the first place. In one of my projects, I'm using an async library and have isolated the async-ness by creating a dedicated thread that communicates with the library. The thread provides a queue of messages that the remaining code of my project can handle.

[0]: https://docs.rs/tokio/0.3.3/tokio/runtime/struct.Runtime.htm...

   async func foo(x int) task[int] { return await(2 * x); }

   // No "async" only because cpu_bound_future_factory() returns a task
         func bar(y int) task[int] { return cpu_bound_future_factory(async func() task[int] { return await(y - 5); }); }

   async func baz(a,b int) task[int] { return await(await foo(a) + await bar(b)); }

   func foo(x int) task[int] { return 2 * x; }

   func bar(y int) task[int] { return nowait cpu_bound_future_factory(func() task[int] { return y - 5; }); }

   func baz(a, b int) task[int] { return foo(a) + bar(b); }

If you compile both for the same goal the system will have suboptimal memory usage or performance, and in general you don't know whether a function will be interrupted in advance if you don't have colored functions (also you can't move the stack if you have references unless you have a GC, which is terrible).

I would go further, and say assuming your program is doing "interesting work" (not just calling a database / reading a file and returning the results), use a threadpool. In almost all applications threads are fine, certainly when you are comparing threaded Rust to people using Python/Ruby.

The problem with custom stacks is you essentially need to have a custom runtime and it makes calling into C functions a lot more difficult (cgo isn’t a cakewalk)

This runs counter to Rust’s C++ inherited belief that you don’t pay for what you don’t use and would have have made Rust less feasible for all kinds of other projects.

async/await might not have the best developer ergonomics, but it does have the best implementation ergonomics from a language point of view

Goroutine (loom) style concurrency only makes things worse both ergonomically and performance-wise, pushing programmers towards slow buggy lock-riddled code as the only way to use such models.

Dont use async-await then. Its just syntactic sugar (minus a few details that make it so ergonomic that you can do things that otherwise would be more work than they are worth). Just use plain futures. Be joyful as there will be no "colored" functions in your editor - only 10x more code (and less readable!).

I think there is a place for arguing about async await syntax and whether fibers are the better solution (try implementing them without a large runtime! Rust got rid of theirs.). Clearly, the benefits provided by continuations are worthwhile in avoiding context switches - to argue otherwise is wild. Context switches are very expensive and, in performance critical code, a huge red flag.

To actually answer your question a bit: Threads allow all the benefits of processes (except the two I just mentioned), with the added benefit of being lower overhead and allow sharing information more easily (but you can still use full-blown IPC to communicate between them if you wish).

Multiple processes works better if you have little or no sharing or communication between the threads.

If you are interested, look into the new Windows I/O scheduler that was implemented for GHC 9.0 (in Haskell, obv). There is also some preliminary work to integrate io_uring into the Haskell runtime if it' available, but the Haskell runtime philosophy is rather different from the Rust approach.