I've screwed up before, and I sympathize/empathize with their ops folks, but this should make us think about plan B in case something like this happens again.
Stuff like this only happens once :) haha,
To be fair, I suspect gitlab will a adopt a 'never-again' policy towards making sure their backups work.
GitLab isn't popular because of the stability of its cloud platform. It's popular because you can install your own instance for free practically anywhere with minimal effort.
I run GitLab CE on a box in my server closet for projects that involve livelihoods.
If that is the reason why you use Gitlab, then why not try gitea or gogs? gogs is written in Go and provides a docker image or a drop in binary
I am so happy with Gitlab that I didn't even install Gogs even once to give it a try, though I know about it since it's initial days. For me Gitlab CE just works. Unless Gitea shows a 10x feature I am very unlikely to shift away from Gitlab CE.
The only (minor) issue I've had with it was when I tried to push the OpenCV repository to it, just for the hell of it, on a heavily constrained VM (Debian with 256 MiB of RAM). The poor thing just couldn't handle it without crashing until I upped the memory to a couple gigabytes.
I've found Gitea to handle larger repos fine after running the following:
git config --global core.packedGitWindowSize 16m
git config --global core.packedGitLimit 64m
git config --global pack.windowMemory 64m
git config --global pack.packSizeLimit 64m
git config --global pack.thread 1
git config --global pack.deltaCacheSize 1m
Activity for the last 160 days. There were 175 commits to gogs and 720 commits to gitea.
https://gitsense.com/gogs-gitea/commits-160days.png
Activity for the last 60 days. There were 109 commits to gogs and 262 to gitea.
https://gitsense.com/gogs-gitea/commits-60days.png
https://gitsense.com/gogs-gitea/changes-60days.png
https://gitsense.com/gogs-gitea/changes-files-60days.png
The options and vendors directory are unique to Gitea and they account for a lot of the changes within the last 60 days. I was told the vendors directory is used to store dependencies but I don't know what the options directory is used for. And as the following shows, they account for a lot of the files touched, in the last 60 days.
https://gitsense.com/gogs-gitea/changes-options-vendor-60day...
Based on what I've read on Hacker News, the developer behind Gogs, tends to merge in changes in spurts, so it's hard to tell if this recent flurry of activity is a spurt or not. In this 365 days of activity, you can see the 3 spurts for Gogs so far.
https://gitsense.com/gogs-gitea/commits-365days.png
Regardless of whether or not Gogs will continue to develop at an increased rated, it looks like Gitea will.
A not known library that also addresses the same circumstances as curl might have none listed.
Which would you prefer to use?
The other hasn't had that chance.
Thus, try the first, as it is better tested in the world.
But I don't trust the employee that was shown forgiveness for a horrible mistake. Some might learn to not repeat what they had done, while others learn that they can get away with things through the magical power of phrasing the situation in a positive light. And some might be mistake-makes-for-life. Not all people come out the other side stronger.
Manufacturer A: perfect product. Manufacturer B: omits some pieces, expedites replacements.
People will love and extol the amazing service virtues of B, not A.
You should always have 2+ production nodes in case one goes down.
That's just nonsense. In most cases, the local repo should be more than enough to continue work.
I'm not trying to skewer Gitlab, I host an instance at home. I've also nuked 30% of the ports on our openstack cluster and screwed up everyone's day. I admire the transparency. I just wanted to call out HN's reaction, vs way smaller (technical) issues involving GitHub. But someone did point out downthread that HN's not a hivemind, so there is that.
They are just better at hiding it, smoothing it over and lying.
Gitlab was _one_ (last!) final step from complete data loss of everything. One. At that night, there was quite a long moment there had only one copy (and 6 hours old). Every other backup was missing/notworking/deleted.
This is scary.
I find this to be an eye opener. The real problem was dodged by doing the right thing in the proverbial last minute (6 hours).
Neither should be exclusively relied upon for business-critical services. Ask Github about their production backups and DR plans some time.
I think "everybody learns the hard way" is just one of those things with operations.
In the US maybe, but given that they're a remote company with people all over the world, they don't necessarily have to live by US standard (especially when it comes to spending!).
40k USD is a VERY good developer salary in Argentina, and I'd bet that in other countries a lower figure might make the cut too. I can definitely understand why they'd hesitate to pay thrice that amount.
I believe it's over 5 times the average salary.
And its not just me, they've done this in the past as well: https://news.ycombinator.com/item?id=10924957
Also when you work for a remote company, they are cutting cost in terms of office and all which should reflect back in the salaries, the whole idea of working remotely was to mutually benefit both the employeer and employee and not just gitlab using their whole startup argument to cop out when they want on that "truly remote" so-called transparent company.
I fail to see how this disproves it, it merely proves that you're not a good choice for them (because you live in a, relatively, expensive region).
This argument sounds more selfish than fair to me.
A company should follow some rules to neutralise the discrimination between employees, it shouldn't be all pick and choose and take advantage where they can, that's not really the reason one should have a remote distributed team to get cheap labor.
If what OP is saying is true, then the company might as well be upfront about it.
Regardless of that, the average salaries are posted online and they seem to suggest quite the opposite to what they offer people outside of US. It's just an indication of how much of a bully culture they have in negotiations or discriminatory, they might as well just outsource the site and not have a team of their own at all.
Everyone has their own copy of the gitlab remote.
Does that mean you should only use gitlab for toy projects? I don't think so.
I think they'll quickly learn from this.
Yes, our process is too tightly tied to a single service, but it happens because we don't have the resources to self-host our own solution. We love GitLab, but this has absolutely got us looking elsewhere.
If you love GitLab and don't want to self host you can pay them for GitLab Hosted (paid customers had no troubles today).
If you self-hosted code reviews/CI, would you expect having a similar downtime causing problem in the future?
I expect the answer to be "Yes" for most companies.
I don't know. I'm very hesitant trying out GitLab now whereas I was interested before.
If "too much transparency" is a turn off for you, you're probably just an authoritarian trying to scheme and scam your way into profit, and you probably lack the confidence required to put whatever skill you think you have on display.
Working with people is hard, specifically because you have to get outside of your head and think about how they see the world.
Recently, our Redis cluster failed and both master & slave host machines rebooted.
This caused a latency spike in our app, from roughly a 70ms to 400ms response time for less than 10 minutes.
We posted to status within 60 seconds and posted 3 updates within those 10 minutes.
The next day, a new customer (who hadn't gone live with the app yet) cancelled their subscription because the app was "not reliable".
I guess my point is that there is a balance to strike. Our customers are not tech-savvy in any way and treat any small issue as the end of the world. Maybe there's no need to freak people out for a minutes-long latency spike.
https://en.oxforddictionaries.com/definition/err_on_the_side...
Just in case you ever use that phrase in critical correspondence. Better to err on the side of caution.
There's no reason to hide behind a new account other than to be an asshole.