Hacker News Reader

Comment by rvz | Hacker News Reader

This.

That's how you do it (and also how many open-source organizations have done it) as I have said and predicted years ago here going against 'centralizing everything to GitHub' [0]

Fast forward precisely 2 years ago since that prediction was made in (April 14th, 2020), GitHub has become (unsurprisingly) unreliable, especially for those that went 'all in'. Self-hosting for your own organization seems to have been more reliable in the long run.

> Seeing constant issues with Gitlab and Github products just reinforces that decision.

This also. Especially the SaSS ones.

[0] https://news.ycombinator.com/item?id=22867803

View on HN

At my start-up we are all in on SaaS because we don't have enough engineers to self-host our things.

The previous F500 company I worked at had enough engineers to self-host their things but thought SaaS was also the future and migrated a lot of their key tools and services onto these cloud platforms.

I wonder if both my start-up and the F500 company will suffer in the long run because of these decisions...

If you have enough engineers to develop a product, you have enough to host your own Gitlab. Most engineers are expected to understand some ops anyway for CI/CD, setting up Gitlab really isn't harder than that (from experience). Especially with low usage, these instances basically never go down.

Most of the overhead is managing users, permissions etc, that won't go away with SaaS.

Gitlab in particular is unbelievably easy to self host. At my last gig I ran it on a t4g Debian instance with unattended upgrades and then just rebooted once a week at like 3am, literally never touched it again and it costs nothing to run.

But how many people would know what to do if something did happen?

Worst case you just spin up a new instance, install and restore the backup. Won't take more than 1-2 hours (less if you have part of it automated).

There was a fair bit of ops expertise on the team, so I don't think it'd be an issue. If there was none, I'd probably think twice.

Don't enable unattended updates for GitLab. The migrations don't always work and sometimes need manual work. (This is rarely the case, but can happen.)

(Also, when upgrading GitLab, make sure to upgrade version-by-version and always wait for background migrations to finish before upgrading to the next version. You can check background migrations at /admin/background_migrations.)

interesting, this isn't my experience. I've been running a gitlab instance for about 300 people for about 7 years now. :\

To be fair though I'm using docker and watchtowerr to keep it updated, there was 1 time where I had to use an older docker image because the latest wasn't working.

For those like me who went familiar with it: https://containrrr.dev/watchtower/

It’s incredibly dumb that it allows you to start the next upgrade before the background migrations finish.

Not unrecoverable though, so saying you need to worry about it may be a bit too alarmist.

Interesting, thanks for the heads up - We didn't have any issues while I was there and probably would just restore if we needed to, but nice to know the footguns.

This happened to me exactly _once_ in the last three years of successful monthly update on our self-hosted GitLab instance. I got an error because the backup script expected a folder where no folder was. Updating GitLab is a breeze!

Reply on news.ycombinator.com