Degraded performance on GitLab.com | Hacker News Reader

Degraded performance on GitLab.com

gitlab.com

90 pointssexy_devil45 comments

I recently started a new position at a company that is using Gitlab. In the last month I've seen a lot of degraded performance and service outages (especially in Gitlab CI).

And then I see stuff like this that makes me scratch my head: https://about.gitlab.com/product/service-desk/

If anyone at Gitlab is reading this ... please, please slow down on chasing new markets + features and just make the stuff you already have work properly, and fill in the missing pieces. Example: https://gitlab.com/gitlab-org/gitlab-ce/issues/63880

> please, please slow down on chasing new markets + features and just make the stuff you already have work properly

I really agree with this. I was working at a company where we were exploring switching from Github enterprise to Gitlab for the K8s integrations, the docker registry, and the CI features.

Following the helm chart installation instructions was a nightmare because we were on-prem w/ a custom cluster. There were a lot of assumptions we had to find workarounds for (e.g. we used an f5 integration to manage our ingresses and performed SSL termination elsewhere so the nginx thing was awful).

The other terrifying bit was the number of services / pods that got brought up with the installation. If I have to monitor my team's services, I really don't want to monitor the health of: NGINX, Postgres, Redis, Minio, Registry, GitLab/sidekiq, GitLab/gitlab-shell, GitLab/gitaly, GitLab/unicorn, and GitLab/migrations.

When it comes to on-prem or self-hosted software I actually prefer running a monolithic application that worst-case I can just bounce or reboot the server.

Slow down, simplify things, and improve the user experience. Gitlab already has enough features to be competitive for a while with the Github + marketplace model.

> When it comes to on-prem or self-hosted software I actually prefer running a monolithic application that worst-case I can just bounce or reboot the server.

So why not just skip the helm chart and install the omnibus package on a dedicated server? If you want you can even disable stuff like the omnibus Postgres/Redis/etc and manage those yourself elsewhere.

That's what I do - run the omnibus package sans fancy helm chart - and it works out pretty well. Granted it does require a fair bit of RAM to run smoothly.

> Slow down, simplify things, and improve the user experience. Gitlab already has enough features to be competitive for a while with the Github + marketplace model.

Completely agree with this and what the parent said - I think this is part of why running Gitlab seems to require an ever increasing amount of compute resources :(

> That's what I do - run the omnibus package sans fancy helm chart - and it works out pretty well.

Can second that. Hosting a instance for our (quite small) dev team on a kinda-small VM and the GitLab Omnibus Package has been one of the nicest (while biggest) piece of software to administrate. Updates are super smooth, the things we use seem to be stable over the last two years. Only three unplanned downtime’s in that time and all because our hoster is plain terrible (hi Telekom!).

For a bit more about WHY we choose breadth over depth - We believe that the company plowing ahead of other contributors is more valuable in the long run. It encourages others to contribute to the polish while we validate a future direction. You can read more on our company strategy page - https://about.gitlab.com/company/strategy/#breadth-over-dept...

As open-source software we want everyone to contribute to the ongoing improvement of GitLab.

"We're Open Source!" isn't a valid defense when you have paying customers. That pitch sounds great for your VCs, but for someone who spends a portion of their budget on your cloud services - i'm appalled.

Gitlab is a SaaS company who also provides an open source set of software. If you don't want to invest in supporting up time - then don't sell paid SaaS services.

Your logic and reason have no place in VC land.

They need more users!

> We believe that the company plowing ahead of other contributors is more valuable in the long run. It encourages others to contribute to the polish while we validate a future direction.

I feel like this is a non sequitur. There's no reason why breadth would increase outside contributions other than sheer surface area, but that relationship isn't very linear, especially if the surface area is being created by Gitlab rather than others.

Meanwhile, "polish" is often the unglamorous work which people typically don't want to do for free.

Try to imagine an exchange at Torvald's Bazaar and Emporium. Gitlabbers offer to work on the fun stuff and get paid for it. In exchange they ask everyone else to work on the boring stuff without being paid.

For some mysterious reason, this exchange fails to happen.

I think I understand the perspective, but the messaging sounds a bit like, "Pay us full price while serving as our beta tester; sacrifice the needs of your company so you can fulfill the needs of ours"

Wait, you are saying that your current customers aren't valuable?

Thanks, I am now convinced that I'll never do business with GitLab.

We are paying money for the service. Yesterday we did not get good value for money.

I know this is probably a controversial post but seeing how big and bloated Gitlab became after years of devops dragon-chasing, i switched to gittea. https://gitea.io/en-us/

I keep separate CI and a separate wiki, so i can upgrade them independently and not have the entire software development process grind to a halt whenever I need to patch.

I don't think that's controversial, I think it's different use cases. If I want a simple git repo I'll go gogs/gitea, but GitLab is an all-in-one tool for a software shop.

Hi GitLab employee, just gonna echo what the other person said and say that it isn't controversial to say that! We know that we don't work for every situation, if all you need is a repo, then by design we can't compete with how lightweight something like Gitea is.

Indeed. Incredibly slow.

Status page: https://status.gitlab.com/

Issue: https://gitlab.com/gitlab-com/gl-infra/production/issues/928

Even though my builds won't start as runners seem to be blocked, it's kinda fascinating to see those in-depth details about the service and how the Gitlab dudes try to find the root cause.

Right? This is EXACTLY what I want to see when there's a service disruption.

A live, in-depth view of who is doing what, any new leads on the issue, multiple teams chiming in with various diagnostic stats, honestly it's really awesome.

I know this can't be expected from most businesses, especially non-open sourced ones, but it's so refreshing to see this instead of the typical "We're working on a potential service disruption" that we normally get.

I think it's cool but I'd be totally self-aware and fearful or making the idiot post in the thread for HN to judge :p

It's certainly something I'd be scared of too. It's honestly really brave of them to approach it this way, but I do enjoy watching them work :D

GitLab is best (imo) as a private instance; we haven't had any issues with it after using it for 3 years now

There have been a number of updates in the last 3 years that would have completely broken or caused 500 errors on your Gitlab instance if you immediately updated it prior to a hotfix being released. If you didn't encounter these I would be shocked or you're keeping off bleeding edge releases (smart). I've managed private gitlabs for 4-5 years. I've submitted ~5-10 tickets regarding various things breaking (gitlab-runner, pages 500ing after an upgrade, etc).

If you give each update a few weeks/month and pay attention to the comments on each release blog you'll save yourself quite a bit of headaches.

I found that the good old “skip the .0 release” goes a long way.

Exactly what I am doing too, always wait for the next patch release after a x.0.0.

Before doing that I had two situations where the major release broke something. Never looked back.

exactly my experience. Assuming that a powerful enough server is available to do that. Our first attempt to run gitlab on some small cheap VPS (2 cores, 4 gigs of ram) was a total disaster.

What server specs are working well for you?

Not OP but I have a number of gitlab instances:

For 5 users we use 18G/4vCPU. For 20 users we use 32G/8vCPU

I keep asking to lower the footprint but the gitlab is the very definition of feature creep. And that comes at the cost of optimisation.

We support 30 developers on 10gb ram and 4 cores. That said, we are probably a year and a half behind version wise and only use it for its SCM capabilities.

We are running it on a 2vCore, 8GB RAM vm. Works good for our ~10 concurrent users on peak. The ~120CI builds per day are running on other hosts though.

quad core with 16 gigs of ram for a team of 4 people. Most likely 8gb of ram would be sufficient for us but the risk of decreasing our productivity is not worth the insignificant difference in price of 8GB vs 16GB VPS monthly fee.

Gitlab has lots of problems. It runs slowly unless you throw tons of hardware at it, and has lots of bugs. Our devops guy burns tons of time keeping it running. There are too many parts to it; too many services. And ruby is just not meant to be used in production stuff, particularly not of that scale.

What is Ruby meant to be used for, if not production software?

Rapid prototyping. I'm not talking entirely out of my behind here; I've done ruby before. It's a joy to write, especially compared to something like Java. Unfortunately, that convenience and syntactic sugar comes at a price - performance.

Ruby is like a back-of-the-napking sketch, almost: it takes nothing to get started, you can just jump in and start writing. Other languages are closer to gradations of a proper engineering diagram, the sort of thing you draw up once you've fleshed out your idea a little and want to scale.

As much as I hate bothering with a proper diagram, it's very hard to scale based on napkin sketches.

Have you also tried Gogs?

https://gogs.io

Gogs isn't exactly the same product though. If all you need is a git repo then sure, but it's all the other features that make GitLab more difficult to maintain

Running Redis on GCP? I’ve seen CPU and latency spike on our production Redis nodes before - solely due to noisy neighbors causing memory bandwidth contention.

Funny. Gitlab has always had terrible performance for me, both the website and the git server.

Thank you Gitlab for the amazing product and transparency. Just watching the conversation as everyone over there is trying to get to the root of the problem is inspiring. What a strategy.

Is the conclusion that redis doesn't handle caching items that reach 99MB size each? I guess I would cache them in blob storage, not in RAM. And make sure to zstd them.

Not sure if this is related but I've received a bunch of emails from Gitlab stating my account is locked for too many failed password attempts. Possible DDoS?

We haven't had any other reports of this, please submit a ticket here if you think there's an issue so we can track it: https://support.gitlab.com/hc/en-us/requests/new?ticket_form...

you have a convoy.

I was trying out Gitlab pages for the first time, but gave up since it was so slow, moving my stuff to Github. Guess it's not always that slow...

I can say for sure, that Gitlab is not running that slow all the time.

However, the services run noticeably slower when the US awakes (I'm located in Germany), which is in the afternoon and evening around here. I have no stats, but it feels like builds and the site run smoother in the morning.

In case you're actually curious, we do publish our grafana stats here: https://dashboards.gitlab.com

Hi! We definitely don't run slowly this often, our dashboard is published here: https://dashboards.gitlab.com

Reply on news.ycombinator.com