Writing Rust the Elixir Way
kutt.it
kutt.it
This is super important and awesome. It's a big part of what makes Erlang simpler to write code for than other highly concurrent environments.
EDIT: The automatic blocking detection of async-std was abandoned after further discussion
You can do a for(1) for loop and it won't hog the cpu because it get premptive. The problem is when you have large amount of problematic mails in the mailbox....
Usually, it's some process that's gotten itself into a bad state and isn't making any progress (which happens, we're human), so ideally you had a watchdog on it it, to kill it if it makes no progress, or flag for human intervention, or stop sending it work, cause it's not doing any of it.
But sometimes, you get into a bad place where you've got much more work coming in than your process can handle, and it just grows and grows. This can get worse if some of the the processing involves send + receive to another process and the pattern doesn't trigger the selective receive optimization where the current mailbox is marked at make_ref() and receive only checks from the marker, instead of the whole thing.
If you miss that optimization, the receives will check the whole mailbox, which takes a long time if the mailbox is large, and tends to make you get further behind, making it take longer, etc, until you lose all hope of catching up, and eventually run out of memory; possibly triggering process death if you've set max_heap_size on the process (although where I was using Erlang we didn't set that), or triggering OS process death of BEAM when it can't get more ram because allocation failed or OOM killer, or triggering OS death, if it has trouble when BEAM sucks in all the memory and can't muster its OOM killer in time and just gets stuck or OOM kills the wrong thing.
But I think maybe the original poster just meant lots of bad mail in the mailbox to mean mail that would take a long time to process, because of how the receiving process will handle it.
Or, possibly bad mail meaning (as you suggest, perhaps), mail that won't be matched by any receive, resulting in longer and longer receives when they need to start from the top.
Yeah; just, if he meant that, it seems like a...weird call out. Since that's not particular to Erlang's messaging model; that's true in any system where you have a synchronization point that is being hit faster than it can execute. Seems weird to call that out as a notable problem, as such.
What's unique to Erlang, and -could- be prevented (by limiting a process to a single receive block, and having a type system prevent sending any messages not in that receive block), if you wanted to change the model, is the fact I can send arbitrary messages to a process, that will never match on them, and because it's a queue, will cause it to delay all following messages from being handled. Hence my focusing on that; yes, that's a potential problem, no, it's not a particularly likely one.
Not that 2k threads is really that high; but it's probably high enough to break something.
But if you run below the explicit limit, but above the implicit limit of whatever continues to work, it's not surprising to me that the OS just freaked out.
You could report it to Apple, but their reporting process is pretty opaque, especially if you're not a registered developer (because why would you be, if you're just using mac os because you like the desktop environment, or whatever). Who knows if they'll fix it, but it's not worth making a big deal over, because you weren't really wanting to run 2k threads on mac os anyway, because it would suck, even if it did work.
From the other message on the thread; it looks like too many threads is causing a watchdog timer to fail, leading to the panic.
Like, can I just start 20k threads in WebAssembly and reboot anyone who visits my site?
Obviously I'd expect the browser to guard against that... but I expected the OS to as well, so my expectations may be way off!
But probably not on Windows 9x, maybe not even on NT4 (although, NT4 was pretty solid), and I wouldn't expect good results on Linux or a BSD from 2000 either.
But macos has lowered my expectations. It's a mash of FreeBSD from the turn of the century, with Mach from earlier, and whatever stuff they've fiddled with since then, plus a nice UI layer. They don't regularly pull in updates from FreeBSD, and the killed their server line (which was mediocre at best anyway), so when it panics if you do something weird, it's not unexpected.
So I bumped it up to 20,000, thinking that would probably work as well: the computer immediately crashed and rebooted. I didn't even get the "kernel panic" screen, it just died (this is on a 2018 mac mini with a 3.2 Ghz Core i7). When it turned back on, this was the first line of the error message:
Panic(CPU 0, time 1921358616513346): NMIPI for unresponsive processor: TLB flush timeout, TLB state:0x0
Weird. I really thought that this wouldn't be an issue at all. And: if it was an issue, that the OS would be able to handle it and kill the process or something, not kernel panic.
If you have any more OSes laying around to run the test on, I’d be interested to hear how well windows and Linux handle the same thing.
Because on the face of it this seems like a serious bug in the OS. I’m only used to seeing this sort of thing with bad drivers
2000 threads does nothing - everything's still responsive, and the process is shown as using 0% of the CPU.
16,000 threads uses ~30% of a core, with a ~136MB RSS. The system still handles it fine, though, and everything stays responsive.
At 20,000 the program panics when spawning threads, with the message "failed to set up alternative stack guard page" due to EWOULDBLOCK. I'm not sure exactly what limit it's hitting, though.
That’s the sort of thing I expect - the OS starves the program, not the other way around.
> In most cases it's enough to use a thread pool, but this approach fails once the number of concurrent tasks outgrows the number of threads in the pool
I think something has been lost from all the dialogue about concurrency. The only way the number of concurrent task can outgrow the number of threads is when you're handling IO. What I see rarely discussed here is forms of non-blocking IO and why those haven't become the norm. Why are we all trying to work around models that involve blocking on IO? I feel I almost never hear about using non-blocking IO as an alternative?
For all other tasks which involve computation, thread pools are needed to improve throughput and performance, only threads can be executed in parallel. Yes you can still multiplex many more small compute task over many threads, if you wanted a really responsive but overall slower program, that could be an option, but I think most people going for a concurrent approach aren't doing so for that, but really just as a way to leverage their IO cards to the fullest.
So my question is, what's wrong with having a thread pool combined with non-blocking IO ?
Maybe I don't understand what you mean by non-blocking I/O, because it seems to me that it is the norm in many environments, for better or worse. Async/await syntax is now available in C#, JavaScript, Python, Rust, and possibly others. And before that we had callbacks in JavaScript and in Python frameworks like Twisted and Tornado. So it seems to me that non-blocking I/O has been widely used and discussed for quite a while now.
Say you used the thread pool model. And you receive 100 requests, and you have a thread pool of 50. Your first 50 requests go to your 50 threads, and the other 50 get queued up. After any of the running 50 requests make non-blocking IO, they release the thread back to the pool, and they yield themselves into an IO waiting queue. Then the scheduler takes from either the request queue or the IO waiting queue prioritizing one over the other as see fit. If any of the queues are full, then the system blocks and waits, dropping requests that'd come in during that time.
Or some similar models.
Is this just async/await that I described?
One is the way GoLang does it. In GoLang whenever you execute a blocking IO function, the language under the hood executes the non blocking equivalent, and reuses the thread for some other task by literally switching stacks. This is a conceptually easy approach, but it has some downsides.
One big downside is a greater cost when doing FFI calls. Because you have many stacks, they are small by default, and you need to grow them significantly before entering FFI code, since C code assumes a large stack is available. You may also need to dynamically change the size of the thread pool if too many threads are currently running FFI code that is blocking. (A logically simpler but more costly option is to have a separate FFI thread pool with large stacks, and all calls to FFI functions are treated as blocking I/O calls on the main pool, where the FFI function gets scheduled to run in the FFI pool, and when it completes the task that called the FFI gets queued back up to continue running, just like asynchronous IO had completed.)
The other approach that most async-await languages use is called "stackless". In this approach, all functions that can block have a different return type, which is some form of promise-like object (these functions are known as "async"). In order to get the result (or ensure all side effects have been completed), you need to "await" the promise, which is a keyword that causes the function to logically pause execution until the result is available.
However rather than stack switch, using await makes the compiler transform the function. The parameters and "stack variables" for the function are instead stored in an object on the heap, along with information about where in the function we paused. The function is transformed such that under the hood, you pass in this heap object. (Alternatively, the function becomes a method on the heap object.) When the function is called, it looks in the heap object to find which pause location we are at, and more or less does a GOTO to that point in the code.
When an await is encountered, if the value of the awaited promise is not yet available, the function will write its current position into the heap object. It will inform the scheduler that it needs to be called again when the promise is complete. It will return a promise associated with the logical invocation to the caller. If this is the first time it returns the caller will be user code. If it had previously returned, then its direct caller will be the scheduled task runner. When the function finally reaches the end (e.g. a return statement), it will update its promise, and return it (again or for the first time).
The net effect of all this is that each thread has a single stack. The transformations result in the call stack pretty much always being the same as if callback based approach was used, if callbacks got scheduled to run on the threadpool (instead of direct execution). But the code flow as written by the programmer looks more like like normal blocking I/O code, making it much more readable than having a whole bunch of callback functions.
The downside is needing to add the async and await keywords all over the codebase, and having more heap churn.
Of course, there are more downsides and upsides to each approach, and there can be variations. It is not impossible to utilize the async/await keywords with a stack switching approach for example.
It can feel clearer and easier to reason about what happens on a single connection if the code is like:
accept, read, process, write and somewhere else handles the concurrency
Especially if the one system does a lot of different things, but they're all related enough to be in one OS process, mashing together a lot of isolated non-blocking IO can be pretty tricky, and hard to keep track of.
https://itnext.io/modern-storage-is-plenty-fast-it-is-the-ap...
https://thenewstack.io/how-io_uring-and-ebpf-will-revolution...
But instead I would like to ask about the future of Lunatic. What is you vision for it? Like, is it a hobby project that you are doing for fun or is it an endeavor that you intend for to be powering big systems in production?
Furthermore, how will you fund the development of Lunatic? And where will other contributors come from? Will they be people scratching their own itches only or will you hire people to work on it with a common roadmap?
And definitely looking at doing something like this next year.
Another question: What about sending code between process? Like sending a fun between Erlang process.
IMHO this architecture has the potential to go beyond BEAM, good work!
Regarding the question about sending code, this also can be implemented. Wasm allows you to share function indexes (pointers) and dynamically call them.
However, targeting WASM seems like a pretty huge compromise to doing this natively in Rust. Conversely, it looks like what you're trying to do wouldn't currently be possible without the WASM/WASI stack. Curious what your thoughts are about not implementing it as a Rust runtime target? Did you investigate this and why did you rule it out if so?
When you say "implementing it as a Rust runtime target", I assume you mean targeting x86/64. In this case it would be impossible to provide fault tolerance. The sandboxing capability of Wasm is necessary to separate one processes memory from another.
Using WASM is a tradeoff, but WASM is very fast, surely faster than BEAM process.
Why not? Isn't that just an artifact of Go's implementation, or is there fundamentally something in CSP that prevents this?
Lunatic is not like that:
> Each Lunatic process gets their own heap, stack and syscalls.
Similarly, Rust code that uses threads and/or async isn't going to work in Lunatic (or at least won't get the advantages of its concurrency model) without being rewritten using Lunatic's concurrency primitives. The concurrency model is more like JavaScript workers or Dart isolates, though hopefully lighter weight.
I'm guessing that the Rust language might work well because move semantics assumes that values that aren't pinned can be moved, and that's a better fit for transmitting messages between processes. But there will probably be a lot of crates that you can't use with Lunatic. If it became popular, it would be forking the community at a pretty low level. You'd have some crates that only work with Lunatic and others that use async or threads.
https://play.golang.org/p/DAoobz43yud
OR
https://play.golang.org/p/m0VntGBORBw
Also, I can't think of what scheduling capabilities Go's concurrency has?
In this sense, it's as safe as mutexes: everything works as long as everyone (including any 3rd party library) does exactly what they're supposed to. When compared to what modern and robust concurrent programming looks like (e..g Erlang), it seems like a compromise to me
This app explicitly wants to use threads, for some reason.
Working with async Rust is very simple if you are not writing futures yourself. Tokio and async-std provide api's that mirror std except for the fact that you have to add `.await` to the end of the operation and add the `async` keyword to your function. With proc-macros, `async main` is just a matter of adding a small annotation to the top of the function. Async functions in traits are blocked due to compiler limitations, but `async-trait` makes it easy. What part of async Rust is more complicated than synchronous Rust?
> and it just doesn't provide you the same features as Elixir Processes do.
Can you explain this? How is tokio's `task` different then elixir processes? From the tokio docs:
> Another name for this general pattern is green threads. If you are familiar with Go's goroutines, Kotlin's coroutines, or Erlang's processes, you can think of Tokio's tasks as something similar.
Green threads can share memory while Erlang processes cannot, they are strictly forbidden to do it.
Also Erlang scheduler is highly optimized to work as a soft real-time platform, so they never run for infinite amount of time, they never block and never (at least that's the goal) bring down the entire application, the worst thing that can happen is that everything slows down but it's still functional and responsive.
I don't know about Tokio.
So message passing is the only way to communicate between proccesses? I guess that makes sense with elixir being a fp language. This was not clear in the article:
> Lunatic takes the same approach as Go, Erlang and the earlier implementation of Rust based on green threads.
There are escape hatches. Obviously, you can do something really heavy like open a network port or open a file descriptor, but it also provides you with use ets, which is basically "really fast redis for intra-vm stuff" and you can transact over shared memory if you drop down to FFI.
each process also has a receive() block where you essentially listen to everything that ends up in your mailbox and pattern match on it to take action.
> Can you explain this? How is tokio's task different then elixir processes?
Tokio's tasks and go's goroutines and kotlin's coroutines are cooperatively scheduled, i.e a infinite loop can block other tasks from running.
Erlang and lunatic have pre-emptive schedulers(similar to a OS scheduler) that schedule processes fairly by giving time slices to threads.
> In other languages they are also known as green threads and goroutines, but I will call them processes to stay close to Elixir's naming convention.
Go makes it easy to mess up because the abstractions are superficially simple, but it's a pretty thin layer that you punch through.
Go read it all but here's a relevant quote:
""" So how does Erlang ensure that processes don't hog a core forever, given that you could theoretically just write a loop that spins forever? Well, in Erlang, you can't write a loop. Instead of loops, you have tail-calls with explicit accumulators, ala Lisp. Not because they make Erlang a better language to write in. Not at all. Instead, because they allow for the operational/architectural decision of reduction-scheduling. """
I haven’t finished it yet but the chapters on scheduling are great.
> Tokio and async-std provide api's that mirror std except for the fact that you have to add `.await` to the end of the operation and add the `async` keyword to your function.
There are absolute pain in the ass problems with `async` currently, in large part due to async closures not being a thing, which means it's very hard to create anything other than an `FnOnce` unless you desugar the future entirely.
You are now limited to non-async functions and if the operation of the crate depends on the return values of those functions, you will need some extreme measures to make it work.
The Rust part, of course :) Seriously though, the compiler error messages alone make it a major pain - although I can't figure out if it's an issue of maturity (language and ecosystem), a fundamental tradeoff with Rust's borrow checker, or me just getting way ahead of myself.
I can rarely go a few days of async programming in Rust before running into some esoteric type or borrow checker issue that takes hours to solve because I borrowed some non Send/Sync value across an await point and the compiler decides to dump a hot mess in my lap with multi-page long type signatures (R2D2/Diesel, looking at you).
- The async/await feature is an MVP
- async/await desugars to code you can write yourself that leverages a bunch of relatively advanced features, namely trait bounds and associated types
- It also leverages a nightly only feature (generators)
- Async programming is inherently harder :)
- We've had less time to see how people misuse the features to see where they land in order to clean up those cases and make the errors as user friendly as they can be
Put all of those together and the experience of writing async code in Rust is similar to the experience of writing regular Rust code maybe two or three years ago. When you encounter things like this, please file a ticket at https://github.com/rust-lang/rust/issues so that we can fix them.
Another would be raw compute power, WASM comes close to native in many cases.
My question: are you familiar with the Lumen project? https://github.com/lumen/lumen#about. Both projects appear to have some overlap. Secondly, what, if any, will be the primary differentiators here? (I've not looked at either projects in too much detail)
One of its creators, Paul Schoenfelder (Bitwalker), you're probably familiar with as he's authored a few popular libraries in Elixir and the other core developer, Luke Imhoff, is ensuring that WASM is taking players like Elixir/Erlang into account being a part of one of the organizations or committees if I recall correctly
Also, I haven’t looked at your code, so maybe this is not understanding things correctly, but wrt targeting wasm, could your work ever make it into the actual rust runtime?
Rust has the same level of runtime as C, with no plans to expand it. So, regardless of how awesome this project is, this is unlikely.
(Notably, there is no "Rust runtime" for async, you have to bring your own if you want to do it.)
> Well then that is not comparable to NIFs at all. In fact it is an extremely misleading example...A lot of the time you can not compile the C code to wasm and what do you do then? How do you interface with OS libraries?
Have you considered making this a port driver for BEAM? Then you could call some function from Elixir to launch a wasm actor (that happens to be written in rust)?
Your BEAM would still be imperiled if the Lunatic? layer violates conventions, of course; but it may (or may not) be simpler than reinventing the rest of OTP?
What is the proper way to call `Process::sleep` in your example? I don't see it in the lunatic crate or documentation, and I can't compare results without running the same test in lunatic.
Edit: I guess async rust is mentioned, but it doesn't really explain in detail what lunatic is solving that async rust doesn't provide, besides a vague "more simple" and "extra features," which the example doesn't really show.
- Message-based communication
- Runtime fault-tolerance - a lean mesh of actor system.
- A Supervision system that makes it easy to kill or restart subprocesses
Does anyone have experience with Bastion or other Rust actor systems?
But as I mentioned in the blog, if you can use Elixir/Erlang do it! It's a great language and runtime.
If you haven't worked with both, you just don't know. The developer experience in go is nowhere near the developer experience in Elixir. It's just so much easier to write scalable code in Elixir than Go. Sure it's not as fast, but it's not god-awful slow (speedwise, you're doing better than python django, for example), and for 90% of people shopping for what elixir and go offer, the network is the bottleneck.
[1]: https://vagabond.github.io/rants/2014/05/30/a-week-with-go
The host can be the browser when running on the browser, or it can be one of the WASM runtimes (wasmtime, wasmer, Lucet)
"Lunatic treats WebAssembly instances as actors. Using WebAssembly in this context allows us to have per actor sandboxing and removes some of the most common drawbacks in other actor implementations."
That is a problem of your MacBook. I can run thousands of threads when testing some servers sustainably and with no problems on my Windows and Linux laptop, never mind desktop and real servers. So it is pretty much practical. Whether it makes sense pretty much depends on what are you doing in particular. Your conclusion means zilch without context.
That seems like an extremely confusing mistake given that there is already a very closely related concept called a "process"?
I would consider calling them "workers" or "isolates" since unless I'm mistaken, you have basically recreated WebWorkers (AKA isolates in V8/Dart)?
Presumably this also means you can't share memory? Very neat idea anyway!