Silk: Open-source cooperative fiber scheduler | Hacker News Reader

Silk: Open-source cooperative fiber scheduler

github.com

123 pointsanimetyan21 comments

It's interesting how some ideas very quickly start popping up everywhere at once. One of my colleagues was working on adding support for fibers to systemd for some time (and the PR was merged a few days ago[1]!). From my understanding, this model is really quite useful for programs like pid1 that cannot use threads.

[1]: https://github.com/systemd/systemd/pull/39771

Funny that systemd appears to copy features from GNU Shepherd now after years of the opposite.

systemd is turning into Windows NT.

I'm not exactly a systemd shill. But what? How?

Well it seems to be slowly including a lot of NT features. Fibers, message bus, the logging system, service management.

I’m starting to think NT got it right.

From what I gather, NT does a lot of things right, just the stuff ontop is what is dragging it down.

Exactly.

Even win32 is mostly ok.

This feature (fibers in systemd) was also modeled after libdex (which I wrote for GNOME).

Looks really interesting. I assume this suggests that ClickHouse is going to gradually switch to using this library for network and I/O, thus addressing the main weakness (in my mind) of C++ thread-per-connection servers, which is, they (surprisingly!) create too many threads and can't really handle more than, say, a thousand active connections at the same time. It mostly matters for async INSERTs in this case of course, not for SELECTs, although generally it applies to both.

Is this comparable to Sea star [0]?

[0] https://github.com/scylladb/seastar

I just took a look a both (thanks for the pointer to seastar! cool project). I'd say "sort of but not exactly". It seems like two approaches to solving the same problem.

Silk seems like mostly a scheduler that uses C++'s native coroutines. Seastar is a future-based framework. Both contain a scheduler and enable async coding patterns.

I have some experience with Seastar.

Seastar and Silk are both event-driven, thread-per-core platforms with synchronization and scheduling facilities built-in.

Seastar is meant to be an integrated platform so it includes I/O and networking primitives for high-performance storage and socket operations to boot.

Silk does not appear to take an opinion on how the actual work performed on the thread should be architected

I've been doing this with something very similar in GNOME for years now. It also does io_uring, threadpool schedulers, workstealing, "pollable semaphores" (like IRIX), profiler integration, and makes fibers implemented as futures themselves.

https://github.com/GNOME/libdex/

The `Cilk` angle is interesting. There’s still room for small runtimes focused just on fork/join recursion.

I’ve been working on one for C: https://github.com/xtellect/cactus

It’s narrower than Silk/SeaStar: continuation stealing for CPU-bound recursive code, not a general async I/O fiber runtime.

Seems not exception safe when task switching during unwind.

Looks like its using MinIO for the S3 benchmark, which is now archived. I wonder what they'll switch to (if anything)

Play on Cilk?

those who don't learn Erlang/Elixir are bound to reinvent the Erlang VM and OTP model.

Honestly, erlang was designed for telephone use cases and for high concurrency, erlang as an idea is fundamentally really awesome.

And elixir on top of erlang really makes the syntax easier to use. Gleam is interesting too!

Depending upon the amount of concurrency needed, erlang can be absolutely good for the situation.

The only gripe that I might have seen online with Elixir might be that it is functional language (and I think has its own way of doing things) and a lot of the times people start with other languages which can scale to only a certain point.

Either way, Erlang VM/OTP model does feel really cool to me. Fun fact: but Erlang was a virtual machine before it was cool (Java)

I definitely wish to learn elixir someday (don't quote me on that though :-D), just because of how unique Erlang/Elixir are compared to many languages in such sense of scalability.

Although not a 1:1 Apples but I also wish to know more when golang is more fit as compared to elixir as currently, I usually prefer golang as a swiss knife with its decent stdlib,cross-compilation,portability features.

Elixir is pretty easy to learn.

Its syntax is very Ruby and everything is message passing / actors, or genservers.

If you have ever used actors at all it doesnt take much to pick it up and even if you havnt its pretty simple.

or maybe erlang / elixir do not have 1/1000th of the libraries and features available to C++ thus it's more efficient to port what can be ported? would you rather have the whole of C++ ported to erlang?

Reply on news.ycombinator.com