The server is completely gone, no ping, no ssh. It takes frantic remote power button pushes to even turn it off.
Afterwards I reconfigure various config lines and repeat.
Would love to read about the actual config issues you're having and how you're solving them one by one -- are you writing about it anywhere?
If any, I think this is maybe too much, every worker just spawns more and more workers. I restricted Postgres from taking all RAM, Sidekiq and Puma to low numbers. But there are more..
I follow the instructions straight from Gitlab.
It's specially important at this size to have git in a different machine than the database as both are I/O constrained.
Is there a reason you do that instead of fixing gitlab or just using something else? Either of those would seem more satisfying to me.
You can reduce the number of unicorn workers in `config/gitlab.rb`.
Thanks for the heads-up.
Then I saw the new installation uses Puma ;)
Both Puma and Unicorn should be killed after they exceed a certain size to avoid this situation from happening. It's possible either this is not working in some situations/configurations, or there is a leak elsewhere although this is the first time I have heard reports of this.
What configuration is being changed from the defaults? Alternatively if you could open an issue with any additional detail we will try to figure out what is happening and fix it: https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues