Hacker News Reader

Comment by jacobgold | Hacker News Reader

We're trying to solve a problem: we don't want huge latency spikes caused by swapping memory to disk. The only good solution to that is to have sufficient RAM. It's that simple.

Introducing mlock is operationally annoying and trying to solve the wrong problem. It requires elevated container permissions, it's not something your SRE team is going to expect, few programs do it, and it has all kinds of technical implications and interactions with kernel OOM system, forking, the Go GC, etc.

And nothing about mlock solves the problem of having insufficient RAM for your workload.

There are very specific scenarios where mlock is exactly the right solution but the typical Go HTTP service is not one of them. The KISS principle applies.

View on HN

Using mlock on all the code and static pages in a binary at startup is not hard or risky.

I didn't claim it was hard or risky.

Have you actually done this yourself for Go programs that you've deployed into production? If so, what were the circumstances? Was it your solution to latency spikes?

It was the default for every production binary. I read the c++ implementation, not really that crazy. It was for latency, and in some cases (with other measures) to keep the binary running if the disk failed.

The alternative (which I’ve also done), is to put the binary in tmpfs, after making sure it’s not bloated.

Your Go binary was being swapped out due to memory pressure, causing latency, and using mlock was your fix for this problem?

Or did someone else do this work and you assumed it was effective? Or did you personally implement this change and then validate that the before/after metrics improved as a result?

As I mentioned, it had long been the standard, the core library gave you a simple flag to use. I would occasionally find cases where it was not set correctly and you could see it causing short spikes (the simplest way to measure is something like a pacemaker task on an event loop, or task scheduling delay since it strips out other variables).

The impact is pretty obvious.

Alright, so you didn't implement it. And you're describing a "pacemaker task on an event loop"? That doesn't sound like a Go program at all, where we don't have a single event loop to measure and we have built-in runtime metrics.

Just seems kinda weird to give advice on forums when you're not speaking from actual relevant experience. I'm getting meat proxy answers and confident replies from people who've never done what they're suggesting others do.

As I said, I was more familiar with the C++ implementation, but we had it for every language in use.

I ultimately change the runtime environment to no longer allow any local durable storage for job (with a handful of exceptions), including swap and binary staging. The fact that almost everything was already using mlock made the transition simpler.

I am struggling to understand which specific problems that pertain to the use of mlock(2) you are alluding to.

Java is an example of the most widely deployed garbage-collected language in production around the world, deployment number and scale-wise. Most of the world's missing critical systems today run on Java. Java can run as a process or in a container. Containerised Java deployments do not require unique operational SRE arrangements, nor do they require any unique/dedicated permissions to run in a container.

Upon start-up, the JVM checks whether the system has sufficient memory to satisfy the max heap size constraint, and it will duly fail to start up if not. Afterwards, the JVM will allocate the required amount of memory, and it will mlock it. Multi-terabyte[0] RAM JVM deployments[1] do exist and exhibit exactly the same behaviour. Java has had such behaviour since its inception.

The prevailing number of Java deployments run on systems with swap enabled, and, from the operational standpoint, it is pure insanity to run mission critical (banking, payments, government etc) apps without a swap. Swapless JVM deployments are virtually unknown or are operational negligence.

The principal exceptions for running a JVM on a system without a swap are hard real-time scenarios and embedded systems. That is it.

What makes Go different or unique that it can't use mlock and that it would require special permissions or unique/distinct SRE/operational arrangements? It is eluding me.

[0] The Azul JVM supports up to 20 Tb of RAM which it will mlock.

[1] See the published Azul/Forrester evidence for reference. It is an interesting read regardless of the mlock debate.

I'm pretty sure almost all of our (very large) java systems were running without swap (it took effort to enable swap for your jobs, and eventually was almost impossible).

Java is generally simpler to run without swap since you know in advance how large the heap can be, so it's just a small amount of variability in the jvm internals to consider.

You don't want swap of heap/stack, or paged out code pages. The latency hit is pretty bad even for non-"real"-time systems. And really, you should know how much memory you are going to use (to within some fair bounds).

Adding only a small amount of swap for very cold pages does not save very much ram. And using it for anything else is very bad for interactive systems. In the same vein paging out code pages you won't ever run just does not save very much.

It requires some discipline in monitoring, load testing, etc, but you should be doing that anyway.

You are correct, of course: if your application needs 2x RAM and you only have 1x, then there is no way mlock can magically save you from swapping. What it gives is protection from a noisy neighbour.

The good thing is that mlock/mlockall does not require elevated permissions. But it is bounded by the memlock rlimit, which you need to configure for the container. If you don't want paging/swapping, set it equal to the total memory you give to the container and call it a day.

I'm curious about the technical implications. Which ones do you have in mind?

To be honest, I don't quite see how mlock makes it complex. It's quite the opposite: it has very clear semantics and makes reasoning about system behavior simple. Disabling swap is the opposite: you're making a bet and hoping it works.

In simple cases, instead of using `mlock` just set cgroup memory.min

That would work reasonably well. With one important caveat: memory controller tracks all memory, including page cache. Setting memory.min will prevent reclaim of page cache charged to that cgroup, as long as total memory usage stays below the limit.

Reply on news.ycombinator.com