And this is why you self-host on your own instance.
Gitlab is a perfect example. They had database issues and had to restore from backups already.
But I fatfingered a lot of self hosted stuff in my time.
Need to do a launch? Build it and push it.
Need to share a change with someone so they can review?, `git diff` and send a patch via email. Want to use a server? Spin up a server, add users and keys and push up to it.
Gitlab, GitHub and these hosted solutions haven't always existed. They're convenient, but not a OMGWTF moment... unless of course you don't have backups.
Can you link the issue please? :)
For context, Prometheus and observability will be handled with Opstrace in the future [0]. I'd like to learn about your use-case and see which troubles you have been running into. Thanks!
To be clear, I don't care about Prometheus on my instance, I only care when it's causing trouble (e.g. by eating 100% CPU all the time).
Also at gitlab.com scale the problems they face are very different from a typical deployment.
It is like having maintaining your car and using the train.
On average if you can fix your car (or hire a good mechanic i.e. consulting) you would probably have a better experience than public transport breaking down, that you are powerless to do anything about.
I would rather run a business depending on my car than the train ?
Yes, I can also fix it if the server was my mine but more than likely I'll be busy doing my actual job (which does not involve fiddling with self-hosted gitlab instances) so I'll take my chances with the Gitlab engineering team. They do fix things and me being busy, asleep, sick, or travelling have no impact on their response. I intend to keep it this way.
However ridiculing people who want to their control infrastructure better because they don't have the expertise or time as the guys running the railway gets old.
> And this is why you self-host on your own instance.
This was the commentary on the outage, and it's just outright wrong. Your self-hosted instances will also experience outages. That's the point I inteded to raise.
I am not denying that there different people have different needs, not everyone wants or has the time to drive either . Just that self hosting is also valid even with uptime or other concerns.
Having said that achieving better uptime is quite possible self hosting not because we are better but because we have simpler challenges than gitlab.com, poorer uptime is also quite likely if we don't know what we are doing.
Simply put, managed services doesn't mean better uptime doesn't automatically .
Spoken as someone who has never taken a train i suppose? Transit at scale can handle maintenance much better than a single vehicle and/or mechanic, and they do so proactively and on schedules. And when things get really bad ( catastrophic failure of some component you can't just "fix" on the spot), public transit will organise a backup ( a new train or a bunch of buses) to get you to your destination.
I commuted in Mumbai trains for years the experience is terrible and dangerous, in most dense cities there is no other cheap reliable way to get anywhere, even in richer cities like NY the system is pretty bad if you care at all about your journey beyond getting from point A to B.
Scaling is hard for public transit, very very hard, it does not matter how wealthy the city is either. Poor cities don't have money to expand, rich cities have ton of legacy infra, politics[1] and other systemic issues. The NY 2nd avenue line is 100 years in the making and costs $15-20B and it is just 9 miles long. There are some good transit systems but most of them are have ton of problems.
Fr vast majority of people, trains (or managed SaaS ) would be good fit for their needs, however that does not mean it is always better in every metric and fits for everyone, for some people control and experience and other aspects is more important than what managed solutions can offer there is nothing wrong with that.
[1] Large scale SaaS apps unsurprisingly also have similar problems