Hacker News Reader

Comment by overcast | Hacker News Reader

I really have to just start rolling my eyes at this point. I'm just waiting for the official meme, and the cycle will be complete. Get to work making your infrastructure resilient to a simple accidental deletion, and restoring some faith in your product.

View on HN

>Get to work making your infrastructure resilient to a simple accidental deletion

I'm sure they've been working on that since the deletion

But where are the stories on this work? What improvements have been made?

Detailed posts on that is how you begin to restore confidence.

No one is just going to take their word that "stuff are in place now".

This is mostly spread across different issues in our issue tracker (https://gitlab.com/gitlab-com/infrastructure). I suspect we'll write up a blog post once all the moving parts are in place, have been tested/used for a while, etc.

There is a list of issues in https://about.gitlab.com/2017/02/10/postmortem-of-database-o... -- also in https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCx..., see Recovery, 3, l.

I think it's great that they are being completely transparent about this.

That said, it's true that it's been almost two months and it seems that the some important issues there are still open and don't look especially active.

The follow up was pretty extensive and we'll be working on it for months to come. Some issues that have been done:

1. Update PS1 across all hosts to more clearly differentiate between hosts and environments https://gitlab.com/gitlab-com/infrastructure/issues/1094

2. Set PostgreSQL's max_connections to a sane value https://gitlab.com/gitlab-com/infrastructure/issues/1096

3. Move staging to the ARM environment https://gitlab.com/gitlab-com/infrastructure/issues/1100

4. Improve PostgreSQL replication documentation/runbooks https://gitlab.com/gitlab-com/infrastructure/issues/1103

5. Build Streaming Database Backup https://gitlab.com/gitlab-com/infrastructure/issues/1152

6. Assign an owner for data durability https://gitlab.com/gitlab-com/infrastructure/issues/1163

Their reaction has been at least very un-profesional. I wouldn't trust them with my data. Specifically when you have a github in front of them.

My impression is the opposite of yours - I've moved personal projects from GitHub since their incident.

I respect that but why? Not the incident itself, but the architecture thwy pick point to that they have no idea about what they are doing.

Their incident response gives me confidence that this sort of thing is much less likely to happen in the future.

It also brought to my attention how much they've progressed as a platform since I used them last.

>> Their incident response gives me confidence that this sort of thing is much less likely to happen in the future

That doesn't sound logical to me. The chances of such incidents happening isn't related to how they announce that incident.

yes it totally is.

A company culture of openly admitting mistakes, knowing that your team-mates will not play the blame game, means that problems will be reported quickly.

In a "conventional" culture, an engineer who makes a mistake (and who will be fired for making that mistake) is incentivised to cover up the mistake and hope that the blame falls elsewhere.

In an open culture, the engineer who makes a mistake is incentivised to immediately raise the alarm.

So while the chances of a mistake happening are the same (we're all human), the chances of it being caught quickly and dealt with quickly are better in an open culture.

>> So while the chances of a mistake happening are the same (we're all human)

...and that's the point I was making in my comment

think you're missing his meaning. should be:

>while the chances of a mistake happening are the same [at transparent companies like gitlab and at opaque companies like github]

>the chances of it being dealt with quickly are better in an open culture [like gitlab compared to a closed culture like github]

They lost 3h of customer data, it's not open culture. It's just gross incompetence.

To be honest, I have now more faith in their product after the incident :D

I can't say if you are serious (given the smiley), but if you are indeed serious, then it's really odd that something like this incident would increase your faith in the product.

Gitlab.com is a repository hosting product i.e. it's in data storage business. Of course there are other important aspects to the product, like the web interface and such, but they are primarily into data storage. They literally lost data of their users [1] (some of them might have been paid customers too), not because someone accidentally deleted the data (which happens and is understandable), but because they did not have the basic things, that you expect of data storage products, functional or tested.

They have been offering gitlab.com as a SaaS product, since 5 years [2] and yet the very basic thing, like backups, wasn't tested or functional. If that's what increases the faith in a data storage company, then I don't know what to make of it.

[1] https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-...

[2] https://about.gitlab.com/about/

You're a better man than I! Backups aren't something that go in the "oops" bucket for me.

Frankly, backups is something I do; it's folly to rely on a single provider to host both the original and the backup. That's why I don't use services that don't let you export your data somehow - at least, not for important stuff.

It's funny, I started reading this thinking "Ah, how nice they all are" quite sincerely (especially the gifts from Google and Codefresh - though I suppose you could interpret those a bit cynically if you put your mind to it...), and by the time I got to the t-shirts, I was pretty much in alignment with you.

Reply on news.ycombinator.com