Hacker News Reader

Comment by xcq1 | Hacker News Reader

Anyone here actually use Java containers in production?

Sadly the article mentions very little in terms of practical advice. We've tried running some small Java 8 Spring Boot containers in Kubernetes which are configured to use max ~50M heap and ~150M total off-heap yet decide to use in excess of double that memory so we end up with either a lot of OOMKills or overly large memory limits.

View on HN

Yes, we do. It's actually a rather small setup (13 dedicated server, about 100 containers).

The absolutely most basic advice is probably: "-Xmx" does not represent the actual upper limit for memory usage. We actually most often only set 50% of the assigend memory for the jvm.

Somebody mentioned https://github.com/cloudfoundry/java-buildpack-memory-calcul... which seems pretty interesting.

You may be experiencing the same bug as we did: memory freed by the GC of a containerised JVM was not able to be returned to the host machine.

IIRC it was due to a bug in glibc >= 2.1. Something about how mallocs are pooled. IIRC you need to tune it to be <= num of physical threads. Usually people advise 4 or 2.

  # openjdk 8 bug: HotSpot leaking memory in long-running requests
  # workaround:
  # - Disable HotSpot completely with -Xint 
  # - slows the memory growth, but does not halt it:
  MALLOC_ARENA_MAX=4

So, ensure that your java process is launched with that environment var (so, export it in the same shell, or precede your java command with it).

If you happen to be using Tomcat, I recommend putting:

    export MALLOC_ARENA_MAX=4

into:

    /usr/local/tomcat/bin/setenv.sh

As for how much memory you allocate to your containers: as of JRE 8u131 you can make this far more container-friendly:

    -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap

This is equivalent to saying:

    -XX:MaxRAM=$(cat /sys/fs/cgroup/memory/memory.limit_in_bytes)

https://github.com/moby/moby/issues/15020 https://github.com/docker-library/openjdk/issues/57 https://bugs.openjdk.java.net/browse/JDK-8170888

We use Java containers in production on a very large scale system. We are actively migrating stuff away from Java to Go, because the JVM is a nightmare in containers. Sure we could make do and configure the JVM to hell and hack things to get it to work...but why bother? We have to allocate hundreds of MB of memory for a simple HTTP server. The tons of configuration and hacks are maintenance nightmares. It's been a terrible experience all around. Java is bloated, it's a resource hog, and the ecosystem doesn't care about cold start times. It's just a terrible fit for a containerized environment.

Spring Boot is, by design, eager to load everything it sense you might need. Most of the memory usage is not Spring itself, it's the libraries it's pulling in on your behalf.

In Spring Boot 2 I'm told you can can use functional beans to cut down on RAM usage. Not sure how it works.

But really, it comes down to deciding if you need a dependency or not.

As is normal for questions about Spring Boot performance, Dave Syer has done a lot of investigating: https://github.com/dsyer/spring-boot-memory-blog

Where I work most of our dockerized services are Spring Boot in Kubernetes, they do need more memory than what you've posted and generally run with about 300M ~ 600M usage depending on what they need to do.

You can also use smaller frameworks (Vertx? Javalin? possibly Spring Boot 2). I hope that with Java 9 we won't see this amount of memory usage anymore, however our organisation isn't there yet though.

Yeah, we deploy containerized Jenkins environments for almost 100 teams on VM's running Docker. These are massive heap containers (20+GB in some cases). Probably not the best use of Docker, but we actually are doing pretty good. Working towards migrating to an OpenShift environment and then evaluating some new tech from CloudBees in this area.

Honestly I don't feel there is any need to run Java in containers. The war/jar file is its own container with its own dependencies. The JVM still makes the same syscalls as it would inside a Docker/Kubernetes container.

In fact I would rather look at serverless architecture before considering docker/Kubernetes.

when you run a polyglot stack with java/python/go/node on top of a cluster of machines, you will love to have them containerized and uniform. It makes scripting and CI so much easier.

or, when you have a legacy app that relies on java 6, but you want everything else to run on java 8, the ability to drop everything into a container with its runtime is a life saver.

source: I'm the devops person that's responsible for making this work

We already run a polyglot stack at our company, and we use Docker(nvidia-docker) for our Python environment. With Java there is no need, and it is a lot less work updating and upgrading the JVM and our Java applications. I would use Docker for Java 6 though.

cgroup limits can still be beneficial, consider for example contraining the (by default unlimited) metaspace.

We run in production with conservative memory limits.

If you want to be more clever, you could try this: https://github.com/cloudfoundry/java-buildpack-memory-calcul...

If you ditch Spring Boot and wire Spring yourself the startup-time and footprint should be smaller. But 250M may be a bit optimistic.

What are the other common JVM Memory offenders ?!

(Electron i guess being the obvious JS-to-native example)

(And isn't Sprint Boot the light-weight solution to the Sprint memory problem? /s)

Reply on news.ycombinator.com