I’d say, let the one who tried to allocate memory crash, and if you’re a critical process like xlock, use statically allocated memory and don’t alloc again.
Nothing like statically allocating memory can work when overcommit is enabled because the kernel is free to compress memory, page it out and etc. and then murder you the next time you try to perform any operation that it doesn't have the space for, no matter how safe and static your initialization was.
Note that overcommit is very useful in many cases including the ones where swap saves the stability of the system under conditions that would otherwise completely lock up or panic, so it's also not viable to just prevent it from being used.
> Granted, then you have a locked screen that the user may or may not be able to unlock, which is awkward if better.
The most secure system is one that cannot be accessed, technically. In some cases it's better not to let anybody in than to let an attacker in (technically). Of course, this is frustrating for the user.
No, security includes Confidentiality, Integrity, and Availability; a lockscreen DoS is a problem
I’ve seen plenty of server log with OOM killing mariadb processes, and then being restarted automatically by systemd, often with no one noticing if not days later.
The thing that bogs down systems and often makes them unrecoverable is when a memory hungry process starts swapping. Good luck trying to SSH in. Swap is such a silly idea on servers - good to deal with pages no one accesses, catastrophic when you’re out of RAM and memory latencies suddenly become 4 or 5 orders of magnitude slower.
This doesn't save you if someone other allocates and OOM killer chooses you as victim
So the OOM condition can hit any random process, not necessarily one that just tried to allocate. If you don't have some sort of selection, then you would still have an OOM killer, only it will be killing completely at random.