Gitlab Architecture Overview | Hacker News Reader

Gitlab Architecture Overview

docs.gitlab.com

218 pointsLukas199443 comments

As much as I appreciate all the components that Gitlab utilizes behind the scenes running an instance of this on a 4GB droplet consumes the entire memory of that unit until things come to a crawl.

Does Gitlab actually need to utilize all those components and when I disable certain things like Prometheus and etc. it seems to only marginally make things better.

What is consumes so much RAM and why?

There're special instructions to reduce Gitlab Hardware requirements for small appliances [1].

[1]: https://docs.gitlab.com/omnibus/settings/memory_constrained_...

Thing is I’ve tried this, it only marginally improves things as mentioned in original post

This is something I've noticed a lot of open core projects suffer from. They make architecture decisions that make sense for their huge, multi-tenant production system but correspondingly make self-hosting on a single node feel like administering a rube goldberg machine.

Some projects stay very lean. We are running our very small company on:

    - Gitea
    - Woodpecker CI
    - Self hosted private Docker registry
    - Minio

They are easy to deploy and upgrade (the upgrade procedure is for me the most important because I want to do it easily or at the end you do not do it).

My opinion is that they optimize for huge deployments. Tiny companies are supposed to use cloud. And self-hosting is supposed to be used by big companies with thousands of people. I guess that's where their money is, anyway.

That said, they're doing good job of documenting things required for smaller installations.

Bullshit, the inability to properly optimize doesn't mean it's optimized well for huge deployments. It will need much more resource on huge deployments and still quite some RAM on small deployments too.

At this point, the code base is too big for them to fix the performance and they want to use their time on new features and bug fixes as that's what customers are asking more while customers may solve the performance problem by spending more money on the server.

It's just that the initial technical decisions sucked and it's no longer possible to fix it at this point.

So true. I was in charge of setting up self hosted GitLab at tiny computer company. Was very proud of the cicd setup untill the boss asked for review apps to be launched for each commit. Gitlab had no solution for this but to use Kubernetes.

This is so true. Sentry's architecture[1] is insanely complex. It runs some ~30 containers on the instance and all you've to pray is that their tooling[2] for `docker-compose` _just works_.

[1]: https://develop.sentry.dev/architecture/ [2]: https://github.com/getsentry/self-hosted

Armin from Sentry here: It's definitely something we're battling with. At the scale we're operating, providing all features on an alternative architecture that does not involve as many independent pieces is pretty tricky. I'm somewhat hopeful that we see reduction in the total types of moving pieces over time at least.

Hey Armin. That's very nice to hear that reduction is something on your roadmap. Pre Snuba, the number of external services were fairly limited but I understand as Sentry got more mature and added a bunch of extra services than "just error/crash reports" is when the architecture also sorta started to become complex.

It's not even just that there are extra services, it's that the volume of traffic is just too high for rabbitmq. So we put a bunch of kafka in and with that, the complexity quickly went up as you end up writing more and more consumers. Most of the services still mostly "just do crash and error reports" but they are doing it in more scalable ways now.

Snuba is one of the services that largely really requires kafka to function though I think theoretically limited functionality is also available by directly talking HTTP to it.

At least for our small use case, the web interface is snappy enough that it's just about spending more money to host it but I'd definitely like it to reduce its memory usage as I'll have to double the server size just for Sentry.

My experience with GitLab is that they accelerated very quickly in the beginning with a focus on new features. This made the whole ecosystem incredibly slow, and it's unbearable for me to run as a service on my own hardware / cloud accounts. I guess that's the problem with these open source services that also rely on paid platform income. I have tested out https://gogs.io/ and the difference in speed is just incredible.

My experience with Gogs is now a few years out of date, but after using it for a few months we ended up switching to GitLab. Gogs was amazingly fast and required few resources, but with moderately-sized repositories or even slightly large pull requests it slowed to a crawl. At some point it screwed up some on-disk repository state causing pull request merges to trigger server errors, which required manual cleanup. Meanwhile GitLab (omnibus), with a few users on a VPS with 4GB memory, remains consistently fast and entirely problem-free.

> What is consumes so much RAM and why?

That's just a complex rails application for you, at $DAYJOB we have to provision 11gb for just a single container running a single instance of the app

I’m running a high traffic rails app with 15 years of source history; our instances have no trouble at 4gb for both unicorn (8 processes) and sidekiq (5 processes).

Except for image processing, which uses a ton of ram and readily OOMs.

I've worked at two very large rails shops (think >10 billion dollar companies) both of which have had stupidly high ram requirements for their code without any image processing. Gitlab which also happens to be another billion dollar rails shop, also requires a stupid amount of ram as per the parent comment. I wonder what the common thread is...

You seem to be suggesting Rails memory usage scales with market cap, which would be incredible if true.

"Why does the app keep crashing?"

"Because the share price went up again!"

It has been a few years since, but I wrote a standalone small Rails app for an internal service for a company that also runs a giant Rails app for their main business.

I ended up heavily tuning the ruby GC parameters, which reduced the final resource usage of the app especially over longer continued use nearly tenfold. There is a lot to gain if you know the actual object and memory limits your app needs for it's core usage rather than letting Ruby allocate more and more memory. Caveat is that was a long time ago with Ruby ~2.3 and things have certainly changed and probably improved by now.

That service has been as stable as one could hope. It just runs in a docker container on a dedicated machine that's also used for other development purposes and consumes a couple of hundred MB of ram. The number of times it was down in the past ~6 years can be counted on one hand.

It definitely correlates. Ruby is not the culprit though. It uses and abuses memory as fairly as other similarly GC backed dynamic languages. The issue may be rails and the monolithic pattern, which favours rapid feature development and postpones optimization for when necessary, or one can no longer realise that snippet you wrote 2y ago fetching data from the db scales well for 5 objects, but not 500 objects, but you forgot about that already.

Ruby has a generational GC which already supports compaction, but not in a production ready continuous way, which means most usages have it turned off, and memory pages tend to overcoming and stay fragmented for long running processes. Python has the same issue and node AFAIK.

It's apparent that a bigger company has more money to throw at the performance problem if that can solve it. Smaller companies want better optimization of their spending with better coding standards.

Please stop blaming rails...

It's the developer who just doesn't know how things work under the hood and code as they like and memory usage explodes.

I self-hosted a gitlab instance for a while on a local VM with 4GB of RAM and it (being the VM) would crash randomly because the OOM-killer killed too many important processes.

I no longer self-host or use GitLab for this precise reason.

I think if you're going to self host a git server on a small server you'd be better off using gitea/gogs

I agree. I didn't know about those at the time though, and ended up with a custom solution for no reason other than because I could :P

I remember back in the day we were evaluating Puppet vs CFengine. During our testing we noted how memory intensive Puppet was, which we assumed at the time was due to it being based on Ruby. Afaik Ruby can quite easily consume a bunch of memory, it doesn't surprise me that an app like GitLab, which is based on Ruby on Rails I think, consumes somewhat largesse amounts.

All these Gitlab pages are really impressive. I found https://docs.gitlab.com/ee/development/fe_guide/graphql.html pretty useful!

GitLab team member here.

Thanks for sharing! :) FYI, the Rails monolith architecture is currently discussed in this proposal issue [0] for a modular monolith approach. This is following the blueprint work in [1][2][3].

[0] https://gitlab.com/gitlab-org/gitlab/-/issues/365293

[1] https://docs.gitlab.com/ee/architecture/blueprints/composabl...

[2] https://gitlab.com/groups/gitlab-org/-/epics/5278

[3] https://gitlab.com/gitlab-org/gitlab/-/issues/31121

Half of me wishes I ever worked at a place that had invested this much in documentation. The other half realizes I'd probably hate it if everything was already this laid out.

I see this blog and my mind immediately goes to:

What was the $ cost in salary to put this out vs

How much should they expect to get back by somehow impressing upon potential customers that they have a sane-ish tech stack (and maybe how much will that help to recruit?)

Its part of the remote-only and open source oriented culture. Haven really good written communication - which documentation is, really helps when you are spread all over the globe.

vs how much $ cost in salary they save because new employees can look things up in the documentation instead of asking colleagues or figuring it out themselves.

I think part of this also contributes to their brand (open source first/foremost) and to potential contributors so there's more to the equation than just impressing potential customers

Well, honestly I prefer that money is spent on documenting potentially deprecated subjects than paying agile coaches and non technical scrum masters

Anyone knows if they use any open source doc project to generate this documentation site?

GitLab team member here.

Nanoc (https://nanoc.app/) is the static site generator we use for our docs.

More info: https://docs.gitlab.com/ee/development/documentation/site_ar...

Just dropping by to say that you guys have really good docs! Some of the other commenters have expressed that they can get a bit overwhelming which I guess is true, but after years of using GitLab I haven't had an instance where the docs couldn't help me figure out how to do something!

(well, there was a case or two in regards to updates in a non-standard config years ago, but community discussions helped there)

I have always been given access too and never setup gitlab myself, but every time I use it, I am pleasently suprised of its featureset.

In the beginning I am always looking for features from other platforms only to find them readily implemented.

It does not matter if it's Mattermost, a container registry, or direct oidc logins for kubernetes and friends, it's just there.

The magnitudes of features, make it an easy choice as a core of your startup.

All in all thank you for creating an amazing piece of software. Like a fridge, for me, it just works.

holy spaghetti! that algolia search palette or similar needs to be in every online documentation.

GitLab team member here.

[0] provides documentation for the search based on Algolia DocSearch.

[0] https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/do...

Please fix the title. GitLab is capitalized with both G and L as upper case.

Sorry but I can't update it.

Reply on news.ycombinator.com