Comment by drewcrawford | Hacker News Reader

Comment by drewcrawford | Hacker News Reader

Parent Full thread

Hi sytse, while I've got you.

I've been annoyed these last few releases by high memory usage. Every day, my backups fail due to ENOMEM.

I have adjusted unicorn-worker-killer to undo the increased memory usage you did a few releases ago:

    unicorn['worker_memory_limit_min'] = "300*(1024**2)"
    unicorn['worker_memory_limit_max'] = "330*(1024**2)"

(8.9.0's default is a totally absurd 400-650MB.)

and while that stops my server from falling over in the first 10 minutes, it does consistently fall over once a day.

Here is a screenshot of my htop, I'm not sure where to look next. 25% free sounds okay, but it's not enough to run a backup, or even run `gitlab-rake`.

http://weblinksdca.s3.amazonaws.com/Screen%20Shot%202016-07-...

Drew, thanks for sharing your settings. It looks like you are using a 2GB RAM system.

How many unicorn workers are you using? For 2GB systems, I'd recommend at most 2 rather than the default 3. Some discussion about this is here: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1279

We'll be doing more work in these coming months to profile and reduce the memory usage needed by GitLab so that all your tools can run comfortably in the 2GB range.

Hi. I have a 2gb synology nas I run Gitlab via docker...i may mod the nas to say 6gb or 8gb ... But trying to avoid that.

My Gitlab backup is ~5gb monolithic .tar.gz daily. Is there any reason the tar.gz is 1 huge file, would using Linux split stop these massive Worker threads and device utilization? http://unix.stackexchange.com/a/61776/86052

An added benefit of splitting to a configurable max size is certain cloud vendors have a max filesize limit...like 5gb :/ when replicating the backup.

I have it set to 2 already :-) Unfortunately I still ENOMEM at least once a day.

Strange, your screenshot shows 0.5GB free. I recommend configuring swap to prevent the system falling over. It shouldn't actively swap much with 2GB unless making a backup or something like that.

Reply on news.ycombinator.com