Hacker News Reader

Comment by leononame | Hacker News Reader

This has bitten me multiple times. The problem I have is that at work we deploy the application (written in Go) and PostgreSQL on the same machine. The backend app allocates a lot of virtual memory, and initially we had overcommit to 0 (heuristic). This caused crashes on big queries in PostgreSQL and we set it to 2. The whole system became a bit unstable because the backend would still allocate a lot of virtual memory and at some point we ran into errors when allocating.

For now, we have overcommit_ratio set to a value that is stable from experience, but there really seems to be no silver lining. Go is very happy to allocate a lot of virtual memory, but so are most managed languages. The best solution would probably be to host the backend and the database on separate servers.

View on HN

I'm not sure if you are aware but there are relatively recent environment variables you can set to help contain Go memory to a fixed size.

GOMEMLIMIT works very well if you set it to around 90% of available memory as a rough heuristic. You should definitely profile your application to fine tune this number (e.g. if you link with C libraries that hold large memory pools then Go doesn't account for that) but also to identify sources of spikey/leaky allocations. For example, encoding/json is notorious for it's inner sync.Pool hanging on to outsized buffers. There's usually a lot of low hanging fruit.

In my experience Go can be extremely stable in terms of memory footprint at both small (~O(1MiB)) and large (~O(256GiB)) scales, and it takes only a small amount of effort.

As far as GC languages go, it is by far the easiest to work with.

I was aware of GOMEMLIMIT, but it didn't cross my mind in this case, thanks for pointing it out. It could be really useful! I'll have to check our specific use case

You can also use cgroup to set allocation policy for a particular program.

Interesting idea. How does that work in practice? If I've got 64GiB RAM, and PostgreSQL has 32GiB memory usage, and Go has 32GiB of memory.

If the database requests more memory, it gets ENOMEM, but if the backend app requests more memory, it does get some more because it can overcommit?

Sounds dangerous, if the go program then writes to the overcommitted memory, you'd still trigger the OOM killer, right?

cgroups have nothing to do with overcommit and memory allocation. They limit actual memory usage for a specific program or group of programs. If this program tries to use more memory than the cgroup memory limit, the program gets OOM killed.

Ok, got it, thanks. The wording in the comment I was responding to confused me and made me think it was possible to also change the overcommit behavior for a cgroup, not just the limits.

Yes, it would. Basically every serious database tries to allocate everything and more - back in the day we'd just allocate VMs on the machine even with the overhead because knowing it cannot leave its constraints and would work within them was worth the cost.

There are many reasons to use a dedicated host (or VM) for a DB server, but if only the accessible memory needs to be limited a container is the simpler, more efficient tool. Said that, I would expect to be able to configure how much memory a DB process is allowed to allocate. I remember distinctly that PostgreSQL allows such. But of course both can be configured simultaneously, a belts&suspenders approach if you will.

Whether failed transactions are actually so much more desirable than a OOM-killed process isn't quite obvious, but it might be easier to troubleshoot.

Reply on news.ycombinator.com