Hacker News Reader

Comment by derefr | Hacker News Reader

I guess I'm one of the few people(?) who like the OOM killer. If all your deployed software is written to be crash-only[1], and every process is supervised by some other process which will restart it on failure, then OOM is basically the trigger for a rather harsh Garbage Collection pass, where software that was leaking memory has its clock wound back by being forcefully restarted.

Of course, this works better when you have many small processes rather than few monolithic ones. But now you're designing an Erlang system :)

---

[1] http://lwn.net/Articles/191059/

View on HN

> every process is supervised by some other process which will restart it on failure

I'm curious if this works in practice for you. The current OOM algorithm in Linux sums up the memory usage of a process and all its children. So there are good chances that the restarter process is killed first, and then the main software is killed too (when OOM killer realizes the last kill didn't free enough memory).

This is exactly the problem we're facing at work here: on a computational cluster, users sometimes start wild code that consumes all the memory, but the OOM decides to kill the batch-queue daemon first, because it's the root of all misbehaving processes. We have to explcitly set `oom_adj` on the important daemons to prevent the machines from becoming unresponsive because of a bad OOM decision.

My "restarter process" is upstart. It's convenient, since the OOM-killer tries to not kill init (for bad things happen when you kill init), so it's a somewhat-safe place to put supervisory logic. One of the better calls Canonical has made, I think. :)

Still, in your use-case, I'd definitely recommend only letting users run their "wild code" inside a memory cgroup+process namespace (e.g. an LXC container.)

Crash-only systems only work when a faulty component crashes itself before it crashes you. Processes modellable as mutually-untrustworthy agents should always have a failure boundary drawn between them. (User A shouldn't be able to bring down the cluster-agent; but they shouldn't be able to snipe user B's job by OOMing their job on the same cluster node, either.) And on a Unix box, the only true failure boundaries are jails/zones/containers; nothing else really stops a user from using up any number of not-oft-considered resources (file descriptors, PIDs, etc.)

Do you have any good resources on where to get started going about setting up failure boundaries/jails/zones/containers like this properly?

I think it's surprisingly easy to get yourself in the situation where this is a concern for you[0] but you don't know how to solve it.

[0] Just run "adduser" and have SSH running, or just create an upstart job, or write a custom daemon that accepts and executes jobs from not-quite-trustworthy-undergrads, or...

If you are running Ubuntu, docker.io makes life pretty easy for you to create and maintain LXC containers.

> The current OOM algorithm in Linux sums up the memory usage of a process and all its children.

It used to sum until Linux 2.6.36.

>I guess I'm one of the few people(?) who like the OOM killer.

Diff'rent strokes. I also like the OOM killer, its a dastardly wonderful thing to tie lots of safety-critical things to .. and in the SIL-4 OS business (my domain), it is indeed a crash imperative to understand how to use the OOM properly. Or: not.

So in this light .. I know Erlang is "the thing" right now, but I feel I must just mention that:

>Of course, this works better when you have many small processes rather than few monolithic ones. But now you're designing an Erlang system :)

.. one could also be designing a Lua-based distribution, or JVM, or whatever you like, essentially, and integrating with oom_killer. There's nothing Erlang'y about it. Because if you're playing with the oom_killer, you're really making a distribution choice, in the topology.

If your app cares about oom, well kiddo .. you better not be doing anything less than excercising complete control over your distro, its launch policies, its use of the TextSegment as installed, and so on. Absolutely you're making Distribution decisions about the functionality of the combined system. oom_killer isn't useful by itself.

My point being, worrying about oom_killer isn't just something Erlang users need think about, nor are they the only ones who really 'get' why an oom_killer can be used nicely .. if you're building a distro, either for use as an embedded machine, a tight secure server image, or indeed even as a desktop user, well .. careful memory integration is, as you say, a harsh pass.

Incidentally, I use the oom_killer exactly as you mention, in a few embedded distro applications, specifically indeed, a kill of whatever 'lua' is hogging resources. Its an extraordinarily functional mechanism for recovery ..

> I know Erlang is "the thing" right now

I don't think Erlang is "the thing" right now. The "hot" languages for concurrent programming are Go and Node.js, for better or for worse.

Those environments don't provide the supervision hierarchy which is being discussed in this thread. In Erlang, every actor you create has a supervisor, which will restart the actor (the behaviour is tunable) when it crashes.

"Hot" is fashion, not engineering.

That is until you get a process that triggers OOM due to a bug on startup (pissing all over RAM) that ends up getting stuck in a restart loop which cripples your machine so you cant get in and fix it. Either that or it fails and neutralises supervisord or something resulting in site down anyway.

ulimit is a better solution i.e. set reasonable constraints based on available resources.

I've never seen a process supervisor daemon that didn't have an "if process FOO exceeds X restarts in Y milliseconds, stop trying to start FOO" clause.

You've never seen an "enterprise" software init script then. 5-nines? Stick it in an endless loop and no one will notice (Atlassian I'm looking at you).

I have written enterprise software. Guilty as charged.

Yeah, but looking at Atlassian is easy, for pete's sake they gave us shitty software like JIRA, Confluence and others... and I say that with a HUGE dislike of those products due to how obtuse they can be.

Fair point! Agree entirely by the way.

daemontools / supervise doesn't have it and it's a fairly popular solution. (not that I recommend ever using it...)

> daemontools / supervise doesn't have it and it's a fairly popular solution.

Ok, it doesn't exactly support the quoted feature ("stop restarting if it exceeds X attempts in Y seconds"), but it does sleep for a second between restart attempts to mitigate the same problem: http://cr.yp.to/daemontools/supervise.html

I'm curious why you wouldn't recommend it. I've never used it myself, but I've been reading about it the past few weeks. The design seems really well done compared to init.d scripts in the sense that every init script must reimplement daemonization, pid files, etc. (which is helped by examples, start-stop-daemon, etc. but is still a huge and delicate chore in my experience).

Is your objection that daemontools is largely unused / unmaintained and lacks features (such as flapping-avoidance)? Or something else?

Just curious, thanks.

It's mostly about the lack of features. No support for cgroups, user switching, adjustable backoff, syslog, etc. Sure - it works, but it's the same thing as with qmail - it's the very minimum that you can call a useful SMTP deaemon. The moment you want any feature on top, you have to implement it yourself.

Either one of upstart, systemd, supervisord will give you equivalent solution that has more practical features. Daemontools was good enough a couple of years ago. (I used it then happily)

In short - there's nothing wrong with it really, but many alternatives are much better.

I disbelieve that ulimit is a real fix. If you disagree, save this shell script, set a reasonable set of ulimits then run it:

  #! /bin/sh
  sh $0 &
  while true
  do
      perl -e 'push @big, 1 while 1' &
      sh $0
  done

A particularly bad case of the OOM killer making a mistake is in-memory databases. If some other process is using lots of memory, but still less that the DB, it'll get killed for no good reason anyway.

One would hope that any machine hosting an in-memory database has gobloads of RAM and not much else to do. :) [Still, this is what replication is for! "In-memory database" is only a scary concept if you don't have any hot slaves.]

One might have 2/3 of memory used, with well known database size growth. Then a runaway process could use 1/3 of memory, cause trouble, but it's still the DB that gets killed.

This isn't disastrous of course (replication + snapshots + redis aof), but still annoying.

Another thing you can do, in this case, is to enable swap system-wide, then put just the database process into a cgroup with memory.swappiness = 0.

Thus, the database itself will never degrade due to spilling to disk, but all other processes might. But if it's a DB box, that won't precisely matter.

Wouldn't another option be to setup hugepages for the db and then just have the db process use hugepages?

They'll never swap out and bonus less memory used. (note transparent huge pages have only brought me pain so far)

So OOM killer as Chaos Monkey? Sounds like an intresting idea that should be implemented as a config setting.

The OOM killer has almost never worked for me. I think once or twice only, the rest of the time, high memory pressure combined with high storage pressure effectively locks up the system. Easiest way to trigger it is to start eclipse, virtual box and start a system update (just package download, not install) and I can't move my mouse any more.

Reply on news.ycombinator.com