It has never been an issue in my use case of Rust, but the lack of an interface for when the system is out of memory is problematic.
Not only in kernel, but in all system-level software.
Then, it happens, less and less frequently, but still, I want to know and I want to be able to handle it.
- most such interfaces do have a constant overhead that all programmers using the language have to pay.
- some major operating systems (like Linux, ehem) make these interfaces useless for all their user space apps by enabling overcommit by default, which makes it hard/impossible to write portable code that can handle OOM
Explicit OOM handling would mean that most users end up paying a relatively high upfront cost for something that in practice for them (e.g. if they are Linux programmers) delivers no value.
For example, on Linux with overcommit (the default), even if the system is out of memory, malloc won't return null. It returns a pointer that's not null, such that your if(ptr == nullptr) will act as if everything is "ok", but then, when you try to read/write that memory, that will trigger a hardware exception, that the kernel will catch, and then the kernel will tell the OOM-killer to "make space" by killing "some process", and maybe your app is just killed, or some other app that your app is working with is killed leading to a race condition, or... or....
When your app is killed by the OOM killer, the signal that your app gets is "irrecoverable", which means that your app will die, you can at best try to do some cleanup before it does, but it will die nevertheless.
So I find it extremely ironic for Linus to argue that practical programming languages are hard to use for environments that must handle OOM errors, when they are championing one of the major platforms that makes handling OOM useless by default.
I keep saying this, but the obvious fix is for the Linux kernel to use overcommit internally just like they expect user space to do. When the kernel then runs out of memory, it should then start killing drivers at random to make for some space. If they think that's such a great default behavior, they should commit to it. \s
You can change the Linux memory overcommit policy using sysctls if you need it.
action on run out of memory should be finish up currently running work and hold off new stuff until you have resources, not die and start to reprocess the thundering herd. When you run out of memory last thing you want to do is empty your hot caches.
Unless your app is faking it’s own caching, in which case it might be contributing to the problem.
• Linux is not the only OS in the world.
• Even on Linux you have containers/cgroups that can impose hard limits.
• Platforms without virtual memory are also an important target for Rust.
• On 32-bit platforms you can run out of address space before you run out of RAM.
• Regardless of what the OS does, the application may still want to impose its own internal limit (e.g. https://lib.rs/cap) to avoid being OOM-killed or swap death.
People keep telling me how Linux never runs out of memory, while I'm currently firefighting a torrent of coredumps caused by Rust's self-own on OOM that actually happens.
I generally love Rust, but its OOM handling is awful, and the "but the OOM killer" nonsense is to blame for stalling the absolutely critical fixes it urgently needs.
The second argument is that complex programs that need to handle OOM properly will be full of complex untested error paths.
How should one handle a OOM error If one can't allocate some temporary node during an algorithm and how to recover without leaving a corrupted state? That's not a question I want to answer for almost every function call.
That's why the rust standard library data structures don't handle OOM. To make it easier to use for the most common case. And for the cases where the OOM handling is required, one can simply use data structures for it, which is what Linux will do (problem solved)
Also compare that to the case of mutex poisoning which can return an error on each unlock, and which is now considered a mistake because in practice, nobody want to handle a poisoned mutex error.
• Statistically, likelihood of failure is proportional to the allocation size. You'll most often fail to allocate a very large Vec, and then only need to allocate a few bytes for a string that says "error!". I'm using https://lib.rs/fallible_collections and just a few strategically placed try_reserve has brought down my coredumps by 99%.
• Panic on OOM would unwind, and unwinding tends to free memory rather than allocate anything. The standard library should reserve/preallocate enough memory to be able to start unwinding.
• Rust is good at avoiding temporary allocations and relies on the stack a lot. You don't have to use fancy error-handling libraries that collect backtrace on every error. Error handling with enums doesn't touch the heap at all.
• Even if everything fails, and you get OOM during OOM handling, that's fair, and still better than abort every time.
Note that Rust currently doesn't handle OOM by panicking. It unconditionally aborts the whole process. Libstd could switch to panics without API changes, and that would make OOM handling possible via catch_unwind.
OR abort the process.
Panics are for fatal irrecoverable errors, not for recoverable errors.
Provide `TryClone`, there, problem solved.
For that kind of low-level work I’d probably do the former and implement my own data structures, which is what you do with no_std currently.
As others have said there’s nothing in Rust the language which assumes allocations always succeed. The standard library made the choice to panic on allocation failure, which was the right choice in order to make an ergonomic API that’s suitable for higher-level work. If you’re not using the standard library (which currently includes people on eg embedded targets) then you can handle allocation any way you like. And if you’re writing drivers for the kernel you’d assume you’re ok handling that.
GP was pointing out that the Clone trait doesn’t allow for returning a Result, so you can’t use that trait to clone anything that allocates. But that’s not really a big deal since you can just make your own TryClone that does.
I was just saying it is a right choice in the alloc crate, which is meant for user space programs, to simply panic on case of OOM, because otherwise the API would be more complex for everyone.
Only on HN discussions (the disaster part) ;=)
(Through it's is a broken discussion, OOM on Linux is broken, but it becomes increasingly "un-"broken and like you said other platform exists, too).
Anyway there is no disaster as:
Rust does support handling memory allocations gracefully, the allocator API defaults to this!!
Just various types defined in [lib]std (or precise [lib]alloc) default in there default methods to calling the (replaceable!) allocation error hook.
This is due to ergonomics, there is no sane/ergonomic way for "common non embedded/kernel programming or non special purpose use-cases" to make literally every method which potentially allocates return a result. And outside of such special purpose cases you can always recover from panics (i.e. outside of special purpose cases panic=abort is a anti-pattern).
Still, even in std, methods like `try_reserve` exists and are used, e.g. by serde deserializers to make sure to gracefully fail if the user tries to allocate a 10GiB array (because compressed data formats ;-) ).
BUT for the kernel all this doesn't matter as it is very unlikely it will use [lib]std or [lib]alloc. So they can default to just always implement their types in a way which doesn't panic on memory allocation.
And wrt. to panic in other (basically guaranteed to be a bug cases) panic=BUG() (ther kernel BUG() macro) can be done.
I'm more worried about the float/128bit integer thing he mentioned, as I don't know anything about it. I assume it's not just a miss understanding of overflow checks and other debug assertions the kernel likely will disable for non-debug builds?
Idk about 128bits integer, but the floating point question has been discussed on Rust's subreddit[1]:
> Normally, the kernel leaves the floating-point state of the CPU in whatever state the userspace process left it in, so that you can do a quick system call without having to save and restore all that state. It's possible to use floating point in the kernel with a great deal of care, but you have to notify the kernel that you're doing so, so that it can save the userspace floating-point state.
> So, it'd be helpful if by default Rust didn't allow use of floating-point, and then you could opt in to allowing it for specific code.
And from Linus[2]
> In other words: it's still very much a special case, and if the question was "can I just use FP in the kernel" then the answer is still a resounding NO, since other architectures may not support it AT ALL.
[1]: https://www.reddit.com/r/rust/comments/mqxr1a/rfc_rust_suppo...
[2]: (from the same reddit thread) https://ipfs.io/ipfs/QmdA5WkDNALetBn4iFeSepHjdLGJdxPBwZyY47i...
https://rust-lang.github.io/rust-clippy/master/#float_arithm...
# What it does
Checks for float arithmetic.
# Why is this bad
For some embedded systems or kernel development, it can be useful to rule out floating-point numbers.
But on the other hand in my experience with C and C++, languages which do allow explicit OOM handling, most applications either just crash when that happen (the `xmalloc` route) or attempt to recover but often really can't do much.
It goes even beyond programming languages. Very few environments deal nicely with near-OOM conditions. My Linux desktop becomes effectively unusable if I start aggressively swapping. And technically at this point as long as I have swap I'm not literally out of memory.
My general point is not that I don't think Rust would benefit from allowing the developer to explicitly handle OOM conditions, it's more that the vast majority of applications are not written to gracefully degrade in these conditions anyway, and if the only handling you do is effectively `alert("out of memory!"); exit(1);` it's really not worth bothering with it.
On the other hand for the minority of applications that want to do something more meaningful on OOM you probably want to take the time to design some ergonomic system that won't litter the code with boilerplate, i.e. you probably don't want every single Vec and String and basically every single facet of the standard API that can allocate behind the scenes to return a `Result<_>`.
I'm sure there's something to do, but I think the Rust devs are right not to rush it given that there are so many other features left to implement.
I'd argue that even this is worth it, for most software. At least then you get a clean exit and an obvious problem. The alternative is to not detect it and let the code walk off into undefined behaviors and potentially subtle bugs that can harm data and/or break security.
The faster you fail the more readily apparent the root cause will be from your logs/core dump/etc. For this to work though reboots (of the application/environment) need to be inexpensive, and you also need supervision. That supervision can either be at the OS level, like Linux's systemd, Solaris' SMF, Apple's launchd, etc., or at the application level like Erlang/OTP.
I don't disagree, just saying that its as nuts as killing random user-space programs on OOM.
"The web browser misbehaved, lets kill the pulseaudio daemon, and if that doesn't fix it, lets kill WiFi, and if that doesn't fix it, lets kill vim, and... "
FFS if some app tried to allocate too much memory just let the app know, or kill that app, but don't start randomly killing user-space processes.
Even worse, Linux by design stalls to a complete crawl when memory almost runs out, even with 0 swap and a fast disk, as it will first start swapping out code pages before giving up. Which means that every time a process is context switched, its code ends up first needing to be read from disk, the worst possible kind of cache thrashing imaginable.
Fair enough. But this would mean that Rust is (factually) a systems language in the same sense that Go was initially declared to be a systems language.
If I look at the Rust community (I give it a try from time to time), I would totally agree with your point, that most users would be pestered by this overhead. I see mostly CLI tools, "web apps" and the kind.
Furthermore, I just can't shake off the feeling that the embedded (bare metal, 16 or 32 bit architectures) or (in-house) kernel crowd will always be 2nd class citizens.
I keep saying this, but the obvious fix is for the Linux kernel to use overcommit internally
just like they expect user space to do.
Rust has all the tools needed to gracefully handle memory allocation failure.
Yes, the tools are a bit limit wrt. usages in the standard library where you mainly can use them to handle "known to potentially fail big" allocations but not every single small allocation.
But enter no_std and it's basically all your choice, which is also the default thing to use for embedded/bar-metal and I count the kernel as embedded/bar-metal. (It's default because the std library types are tuned for the most common use-cases including web-server and user-space system programming, in which you always can have panic on OOM, and error kernel as a much more ergonomic pattern then explicitly returning a result on every thing which potentially allocates).
Through without question the discussion around it is a mess.
No. What I am saying is that Rust intended to be a systems programming language that could handle OOM properly everywhere, but that got a lot of opposition because some operating systems, mainly Linux, make it impossible for programs to handle OOM _at all_, so adding proper OOM support to Rust would mean that every Rust Linux programmer would be paying for a feature (in ergonomics, etc.) that they cannot use _at all_.
That's the irony.
Linux design has made it not worth it for new programming languages to be designed to handle OOM properly, because that's impossible to do in OSes like Linux.
People have been complaining about this for user space programs for the last 20 years, and the Linux kernel stand was "that's a feature, not a bug".
But now that they are confronted with it, you see Linus writing stuff like "that's not acceptable".
That's a huge double standard IMO.
...plus, they're already planning to write their own `alloc` replacement if for no other reason that they need to support API features of the kernel allocator that are absent from the userspace allocator, like GFP flags:
https://github.com/Rust-for-Linux/linux/issues/2#issuecommen...
You argue that the needs of kernel space and user space differ when handling OOM errors.
If that were true, one wouldn't have had to patch the standard library with the `try_...` methods to try to support OOM handling in user space as an afterthought.
That request came from firefox, where a user in userspace can click on a link pointing to a 1 Tb large, e.g., image file, that would kill the browser if it can't handle OOM properly.
So yes, it is a double standard, because both user space and kernel space have millions of valid reasons for wanting to handle OOM errors, but Linux and in particular Linus Torvalds motto here is "do as I say, not as I do".
It is also extremely ironic that this Linux kernel policy that makes it impossible for user space to handle OOM, now causes Linus to cry about it when the kernel wants to start using tools develop for userspace in the kernel.
They are directly responsible for this outcome.
As such, they started with the versions of allocating APIs that are most commonly desired... the ones equivalent to how array[index] will either get what you want or panic if you're out of bounds. The try_ allocation APIs which are equivalent to .get on a container then came later.
A web browser, while a good program to stress the capabilities of a language's design to make sure it can meet demand, is not a typical program.
Not always. The default overcommit policy will reject certain allocations that are too big. Also, you can change the overcommit policy if you need it.
Also, overcommit policy not rejecting allocations by default tends to be more useful than harmful. One example is forking processes: the memory space gets duplicated with copy-on-write, so if you didn't allow overcommit you could end up with processes that couldn't be forked due to that behavior. Not to mention that most programs don't use all the memory they have assigned, so without overcommit you'd have a lot of problems with RAM underusage.
> So I find it extremely ironic for Linus to argue that practical programming languages are hard to use for environments that must handle OOM errors, when they are championing one of the major platforms that makes handling OOM useless by default.
Kernel and user space programming are very, very different. A lot of developers use languages with garbage collection without issues, but that's not acceptable in kernel programming. The fact that the platform provides X feature does not mean it has to be developed including X feature. In this case it's very clear. Even if they used overcommit in kernel code, Linux does not always overcommit memory, so you'd still need to manage the OOM case in kernel properly.
It's not lacking, the standard library just defaults to not use it by default because handling OOM is a conceptual mess (yes, the OOM handling features C/C++ has are included in this).
But even in the standard library you have methods (some not stable) like `try_reserve` which returns an error is allocation fails.
Anyway all allocation parts are part of the standard library (or the alloc library), i.e. they are not a core part of the language itself.
I don't think anyone ever planed to use rust's standard library as-it-is in the kernel (it's just not designed for this, e.g. see panics) or pull in external dependencies from cargo/crates.io without vendoring them.
So it's totally possible to run rust without alloc caused panics.
Now besides that there is the question about panics outside of allocations. Like e.g. if code realizes it ran into a violated invariant, e.g. it knows we have a bug which isn't explicitly handled.
And guess what the kernel already has a handler for it: `BUG()` so basically any panic will call `BUG()` (and we don't panic on memory allocations).
Now maybe some tweaks to the panic system are necessary to make sure this all works well (panic=BUG(), is like panic=abort and on abort call BUG()!).
Now wrt. the integer and float parts, there are two thinks first the panic will likely call without debug assertions like overflow checks (which would call BUG()!! So many are gone, but here are some special cases around floats and 128bit integers on platforms which don't support it where calling BUG() is inappropriate but I have to look into this, tbh.
So short:
- panic on mem-alloc failure is a (lib)std/alloc thing, kernel code would anyway have used something easel
- panic in the kernel can be made into being basically BUG()
- issues around float and 128-bit integers should be fixable, at worst with a compiler flag. But embedded code is by it affected, too. So there is a good chance there is already a fix.
I've worked on some systems which claimed they were dealing with it, but when I purposefully pushed it (by making a malloc which would occasionally fail), I quickly uncovered dozens of bugs, and the solution was to stop pretending would could sensibly handle low-memory situations. SQLite famously does handle this situation correctly, but it is a huge amount of work.
I know of one application with a worldwide customer base that supports this.
Out of memory means, your system is simply not designed for the task at hand. The kernel returning -ENOMEM only masks the fact that eventually Linux will have to OOM Panic if it can't OOM Kill. Hell imagine the swapping and the I/O spike because your VFS cache has been or is currently being purged. I honestly think the best case is to just fail when a fundamental resource is simply not there.
Speaking as somebody who's never built such a system, wouldn't maintaining some opportunistic cache (e.g. for some space-time-tradeoff) be a valid use case of asking for more memory and gracefully degrading in case it's not available?
Error handling would simply consist of not expanding the cache size (if it happens during an allocation related to the cache) or freeing up some cache memory (if it happens for an essential allocation).
But on practice programs ask the users to decide those things, and just fail if the user choice is invalid. It's very rare that some program makes those decisions by itself, and it's usually frowned upon, because each single program can not assume that it owns the entire system.
[1]: https://ziglang.org/learn/why_zig_rust_d_cpp/#no-hidden-allo...
see https://github.com/rust-lang/rust/issues/48043 and https://github.com/rust-lang/rust/pull/80310
This is what Zig has as default behavior.
I'm sure Zig is great, but I don't see from what you linked how that changes what I said.
All code alloc'ing mem needs to deal with this potential failing: the dev can choose to "panic/die/exit/etc" or possibly a different strategy can be chosen (display smth in the UI, not allocating memory, etc).
Rust has this "nicely return a Result, instead of exceptions/panics/etc", in most cases, but not in the case of mem alloc. That a big difference between Zig an Rust.
Now Rust has alternative methods for mem alloc that return a Result: but these are not the default and thus hard to discover. This is "unsafe behavior by default" in Rust. This is special, as Rust usually promotes the safe path.
Indeed. I'm using Firefox in a VM. It's a pain, because invariably at some point Firefox uses enough memory that the VM starts to swap. And then the whole thing grinds to a halt, and I usually just end up with the VM equivalent of power-off.
Instead I'd be perfectly fine with the kernel telling Firefox "computer says no" when it tries to malloc, before the system runs out of memory, and then Firefox can do whatever it wants with that.
Yes I know there's some cgroups magic or whatever I can do, but man, why does it have to be so painful?
We have essentially 3 categories of allocations: (1) those that are large and/or the size is user-controlled, which are (mostly) handled; (2) most of those that happen within the JS engine, which are handled but we're constantly debating whether it's worth the cost; and (3) all the rest, which includes all other allocations outside the JS engine as well as the ones within the JS engine that are expected to be rare and are too hard to handle in any sensible way. For (3), we crash on OOM.
So if you do use cgroups or ulimit or whatever, Firefox may do something reasonable with OOMs. Or it may not, depending on what code sees the OOM. It's still an open question how often the JS engine handling an OOM is worthwhile (as in, it won't just continue to OOM until it finds something that will choose to crash.) OOM telemetry is a little iffy, so I don't trust statistics based on it.
The cost of (2) is not just code size and code complexity. It's also a larger vulnerability surface that is rarely exercised. Within the JS engine, we have ways to synthesize OOM events to at least get some level of testing. (We'll run a chunk of code repeatedly, OOMing on the 1st, 2nd, 3rd, ... allocation, and make sure we either handle it properly or do a controlled crash.)
For example Slack uses several gigabytes of memory if I forget to reload the tab every hour or so. It has a DOM live-leak or something. Perfectly fine to let that thing crash and burn.
Letting my PC grind to a halt is far worse for me.
Now my point is, I don't want the kernel to try to fluff Firefox and try to limp it along. Sure for a few applications it's good that the kernel tries its utmost.
But in most cases, I don't want one application to dictate my PC's performance. Gone are the days where I use my PC for one thing at a time.
And if the kernel was more strict with applications, then hopefully sane OOM handling would force its way into more applications.
You seem not very familiar with the memory "management" behavior of Linux and the wonderful ecosystem it has engendered. For example no matter how much physical memory you have, Chrome for example will just crash all the time if you turn off Linux's insane "lie about memory allocation succeeding" default.