Hacker News Reader

Comment by pixel_tracing | Hacker News Reader

As much as I appreciate all the components that Gitlab utilizes behind the scenes running an instance of this on a 4GB droplet consumes the entire memory of that unit until things come to a crawl.

Does Gitlab actually need to utilize all those components and when I disable certain things like Prometheus and etc. it seems to only marginally make things better.

What is consumes so much RAM and why?

View on HN

There're special instructions to reduce Gitlab Hardware requirements for small appliances [1].

[1]: https://docs.gitlab.com/omnibus/settings/memory_constrained_...

Thing is I’ve tried this, it only marginally improves things as mentioned in original post

This is something I've noticed a lot of open core projects suffer from. They make architecture decisions that make sense for their huge, multi-tenant production system but correspondingly make self-hosting on a single node feel like administering a rube goldberg machine.

My opinion is that they optimize for huge deployments. Tiny companies are supposed to use cloud. And self-hosting is supposed to be used by big companies with thousands of people. I guess that's where their money is, anyway.

That said, they're doing good job of documenting things required for smaller installations.

So true. I was in charge of setting up self hosted GitLab at tiny computer company. Was very proud of the cicd setup untill the boss asked for review apps to be launched for each commit. Gitlab had no solution for this but to use Kubernetes.

Bullshit, the inability to properly optimize doesn't mean it's optimized well for huge deployments. It will need much more resource on huge deployments and still quite some RAM on small deployments too.

At this point, the code base is too big for them to fix the performance and they want to use their time on new features and bug fixes as that's what customers are asking more while customers may solve the performance problem by spending more money on the server.

It's just that the initial technical decisions sucked and it's no longer possible to fix it at this point.

This is so true. Sentry's architecture[1] is insanely complex. It runs some ~30 containers on the instance and all you've to pray is that their tooling[2] for `docker-compose` _just works_.

[1]: https://develop.sentry.dev/architecture/ [2]: https://github.com/getsentry/self-hosted

Armin from Sentry here: It's definitely something we're battling with. At the scale we're operating, providing all features on an alternative architecture that does not involve as many independent pieces is pretty tricky. I'm somewhat hopeful that we see reduction in the total types of moving pieces over time at least.

At least for our small use case, the web interface is snappy enough that it's just about spending more money to host it but I'd definitely like it to reduce its memory usage as I'll have to double the server size just for Sentry.

Hey Armin. That's very nice to hear that reduction is something on your roadmap. Pre Snuba, the number of external services were fairly limited but I understand as Sentry got more mature and added a bunch of extra services than "just error/crash reports" is when the architecture also sorta started to become complex.

It's not even just that there are extra services, it's that the volume of traffic is just too high for rabbitmq. So we put a bunch of kafka in and with that, the complexity quickly went up as you end up writing more and more consumers. Most of the services still mostly "just do crash and error reports" but they are doing it in more scalable ways now.

Snuba is one of the services that largely really requires kafka to function though I think theoretically limited functionality is also available by directly talking HTTP to it.

Some projects stay very lean. We are running our very small company on:

    - Gitea
    - Woodpecker CI
    - Self hosted private Docker registry
    - Minio

They are easy to deploy and upgrade (the upgrade procedure is for me the most important because I want to do it easily or at the end you do not do it).

My experience with GitLab is that they accelerated very quickly in the beginning with a focus on new features. This made the whole ecosystem incredibly slow, and it's unbearable for me to run as a service on my own hardware / cloud accounts. I guess that's the problem with these open source services that also rely on paid platform income. I have tested out https://gogs.io/ and the difference in speed is just incredible.

My experience with Gogs is now a few years out of date, but after using it for a few months we ended up switching to GitLab. Gogs was amazingly fast and required few resources, but with moderately-sized repositories or even slightly large pull requests it slowed to a crawl. At some point it screwed up some on-disk repository state causing pull request merges to trigger server errors, which required manual cleanup. Meanwhile GitLab (omnibus), with a few users on a VPS with 4GB memory, remains consistently fast and entirely problem-free.

> What is consumes so much RAM and why?

That's just a complex rails application for you, at $DAYJOB we have to provision 11gb for just a single container running a single instance of the app

I’m running a high traffic rails app with 15 years of source history; our instances have no trouble at 4gb for both unicorn (8 processes) and sidekiq (5 processes).

Except for image processing, which uses a ton of ram and readily OOMs.

I've worked at two very large rails shops (think >10 billion dollar companies) both of which have had stupidly high ram requirements for their code without any image processing. Gitlab which also happens to be another billion dollar rails shop, also requires a stupid amount of ram as per the parent comment. I wonder what the common thread is...

You seem to be suggesting Rails memory usage scales with market cap, which would be incredible if true.

"Why does the app keep crashing?"

"Because the share price went up again!"

It definitely correlates. Ruby is not the culprit though. It uses and abuses memory as fairly as other similarly GC backed dynamic languages. The issue may be rails and the monolithic pattern, which favours rapid feature development and postpones optimization for when necessary, or one can no longer realise that snippet you wrote 2y ago fetching data from the db scales well for 5 objects, but not 500 objects, but you forgot about that already.

Ruby has a generational GC which already supports compaction, but not in a production ready continuous way, which means most usages have it turned off, and memory pages tend to overcoming and stay fragmented for long running processes. Python has the same issue and node AFAIK.

It's apparent that a bigger company has more money to throw at the performance problem if that can solve it. Smaller companies want better optimization of their spending with better coding standards.

It has been a few years since, but I wrote a standalone small Rails app for an internal service for a company that also runs a giant Rails app for their main business.

I ended up heavily tuning the ruby GC parameters, which reduced the final resource usage of the app especially over longer continued use nearly tenfold. There is a lot to gain if you know the actual object and memory limits your app needs for it's core usage rather than letting Ruby allocate more and more memory. Caveat is that was a long time ago with Ruby ~2.3 and things have certainly changed and probably improved by now.

That service has been as stable as one could hope. It just runs in a docker container on a dedicated machine that's also used for other development purposes and consumes a couple of hundred MB of ram. The number of times it was down in the past ~6 years can be counted on one hand.

Please stop blaming rails...

It's the developer who just doesn't know how things work under the hood and code as they like and memory usage explodes.

I self-hosted a gitlab instance for a while on a local VM with 4GB of RAM and it (being the VM) would crash randomly because the OOM-killer killed too many important processes.

I no longer self-host or use GitLab for this precise reason.

I think if you're going to self host a git server on a small server you'd be better off using gitea/gogs

I agree. I didn't know about those at the time though, and ended up with a custom solution for no reason other than because I could :P

I remember back in the day we were evaluating Puppet vs CFengine. During our testing we noted how memory intensive Puppet was, which we assumed at the time was due to it being based on Ruby. Afaik Ruby can quite easily consume a bunch of memory, it doesn't surprise me that an app like GitLab, which is based on Ruby on Rails I think, consumes somewhat largesse amounts.

Reply on news.ycombinator.com