GitLab is down | Hacker News Reader

GitLab is down

51 pointscloudking34 comments

https://status.gitlab.com

Hopefully an engineer didn't accidentally delete the prod DB again.

fingers crossed

wait wait wait, this actually happened?

See the postmortem https://about.gitlab.com/blog/2017/02/10/postmortem-of-datab...

Fantastic post-mortem that I regularly share out with others. A great cautionary tale on the importance of backup testing procedures (aka why "set and forget" with backups is not sufficient).

even better, watch the livestream of it https://www.youtube.com/watch?v=v0TRHLvYGE0

Self hosting is the best solution for our team. At least we can blame ourselves if something is down...

... but can you make better uptime? :)

Better than GitHub, not better than Gitlab, tbh

Funny enough, I used to think that, but overall having the people that actually build the software make a hosted version ends up being better value IMO.

Our self hosted Gitlab instance needed a lot of hand holding over the years, occasional memory upgrades and involved much more work than initially anticipated. Also keep in mind that you need to spend time to keep the thing up to date and get latest features, which there have been quite a few good ones added over the years.

> you need to spend time to keep the thing up to date and get latest features

That attitude isn't really conducive to custom software development, but Gitlab has many other purposes.

How does it follow that because a cloud provider has downtime that self-hosting is a better solution?

Because my GitLab self-hosted instance is up currently and theirs isn't.

And the next time your is down and their is up, are you suddenly going to arrive at the opposite conclusion?

Without numbers, I don't see how you can make any decent conclusion at all.

My GitLab server has at least beaten the GitHub down average so yes I'm feeling pretty good about it.

Well those are numbers, so it seems like there's more to your reasons than you're letting on.

When our server is down, I know why. Out of Memory, Storage is full. Service is down. We can do something about it. If a hosted service is down there is nothing to do but wait.

> Out of Memory, Storage is full. Service is down.

Is that all you have seen? I have seen many self hosted services fail in our company and the answer is almost rarely any of these simple things but complex things like data loss, data corruption, random restarts, network partition, configuration sync issues etc. That is why companies pay other companies for critical things like gitlab even when self hosting.

Does that actually impact your bottom line at the end of the year?

They say just their website is down and that "Git Operations" are working, but I'm unable to use Git over SSH.

Same, I'm intermittently getting this error trying to work with remote repos:

git@gitlab.com: Permission denied (publickey).

fatal: Could not read from remote repository.

Thanks, confirmed, and updated on https://status.gitlab.com/ to include API and Git Operations.

(GitLab team member here)

Cheers, things seem to be working now

Thanks for confirming. Root cause analysis and more insights have been collected into the incident issue: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1...

Same here - getting: git@gitlab.com: Permission denied (publickey). fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

It's definitely not just their website.

Thanks for your feedback. Updated in https://news.ycombinator.com/item?id=38071672

GitLab team member here.

Thanks for the feedback, I have forwarded it to our teams investigating the incident.

And now it's even worse: https://www.cloudflarestatus.com/incidents/l6x2h1zp69bc https://news.ycombinator.com/item?id=38074906

No wonder people use GitHub even if it scans your repos. If a service is unreliable, why bother?

https://www.githubstatus.com/history

Don't praise GitHub too much. They've averaged an issue every 2 days for the past 3 months.

Issues don't mean downtime though, and GitHub has lots more features. For example an issue noted here was stale commits of up to 7 minutes in 9% of new pull requests... I'm willing to bet such issues aren't acknowledged by most platforms

Because GitHub is never down...

I'm not sure which one has a better uptime, but GitHub seems to on HN at least once a month

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

Without actual numbers, how does this make any sense.

Stock is -63% since IPO

Reply on news.ycombinator.com