Memory inside Linux containers (2014) | Hacker News Reader

Memory inside Linux containers (2014)

fabiokung.com

65 pointsaburan289 comments

CPU also causes a similar problem - for instance Java (and lots of runtime Java libraries) use the number of available CPUs to decide how many threads to use for various tasks, which can cause terrible performance if you're giving a container 1 CPU of a 48 CPU server and it thinks all 48 are available - we encountered this at work recently and eventually bodged around it, but definitely a gotcha that this stuff isn't handled properly by runtimes yet.

Anyone who is running important production java programs without carefully considering and testing the JVM’s command line arguments is headed for trouble.

java9/hotspot has -XX:+UseCGroupMemoryLimitForHeap to deal with the memory part and I think the method to get the number of CPUs also takes cgroups/taskset into account now.

Actually the `-XX:+UseCGroupMemoryLimitForHeap` option is also in Java 8 since Java 8u131, so it's available for nearly everyone right now. It doesn't solve a lot of problems though: by default the JVM will use only 1/4th of the memory (cgroup) limit as its heap size, which wastes most of the container's memory. It's thus attractive to use `-XX:+UseCGroupMemoryLimitForHeap` together with `-XX:MaxRAMFraction=1` to get maximum heap size given your container limits. However, because total JVM memory is heap+permgen+threads*stacksize, you'll leave no memory for threads this way. If you're running an average web server, it will spawn dozens of threads and get your application above the cgroup memory limit (=OOM killed) pretty fast.

The only way out is to configure memory limits by hand, taking into account the maximum number of threads your application will use. That usually isn't trivial.

They're changing MaxRAMFraction to take non-integer values, so in some upcoming java release (yay 6month release cycle) that problem should go away too.

> if you're giving a container 1 CPU of a 48 CPU server and it thinks all 48 are available

Are java processes usually deployed in (1 server to 1 java process ) configuration ?

Seems a bit strange for runtime to assume certain deployment configurations.

CPU works differently though. A server with little load really can give you all the CPUs unless you're pinning workloads (which most people aren't).

>Or why don’t free and top work in a Linux container?

You can use the lxcfs fuse filesystem to get that and a little more: https://github.com/lxc/lxcfs/blob/master/README.md

I definitely don't think systemd should be the go-to system for memory info. It already has too many functions, and does way too much.

Reply on news.ycombinator.com