Hacker News Reader

Comment by synicalx | Hacker News Reader

Honestly, I'm completely flabbergasted by this. Five backups, and NONE worked properly? Who made this? The S3 bucket was EMPTY? Has no one ever tested any of these backups?

It's not just the impact, which is fairly sizeable in it's own right, but it's the HUGE oversight on their part and the fact they tried to pin part of this on PostreSQL?

Credit where it's due; their report/transparency were good if a little unprofessional, and something I'd like to see more of from other companies.

Putting on my BOFH hat, this is what happens when you let Devs do operational stuff.

View on HN

I agree. Their catastrophe is not the kind of thing that represents mild hiccups in operations that will be quickly resolved by the same employees who allowed this scenario to occur the first time around. This isn't the result of a single oversight. It screams of a systemic problem with the way the business operates - period. Maybe every project is rushed out with the deadline being the only metric that counts, quality be damned. Or maybe they're missing talent in areas that matter - like experienced postgres DBAs managing the database rather than having the developers and/or someone who has only ever used MySQL trying to wrangle it.

What they went through is what you'd expect from a first-time pet project, not a professional business out to make money. I suspect they prioritized trying to scale with enough features to catch up with the competition above all else, to the point where sustainability is an afterthought. When desperately trying to gain market share is more important than the quality of the product, this is what you can expect.

I totally agree with needing an experienced DBA, we've had a vacancy open for this for a while https://about.gitlab.com/jobs/specialist/database/

To be a bit more specific, it is extremely unlikely (nearly impossible) that you could ever find a single individual who is both a "postgres database specialist" and a "ruby developer". Do you even know what the word "specialist" means? It means specializing in one area - like postgres - without trying to be a Jack of all trades, ie: with ruby.

"DBA" is a full-time position, not an addon to a developer's duties. They are separate skillsets; you don't get a "2-for-1" special by hiring an expert developer + DBA in one person for one lowly salary.

Do you realize this event would have never happened if you had hired a pure DBA? Or do you really believe you can pin the blame on your ruby developers for not being able to wrangle a production postgres database?

The oblivious or intentionally cheap "the DBA must be an amazing ruby developer" expectation shows your hiring staff - or the management guiding them - has absolutely no clue what they are doing. I can just imagine the internal discussion right now; pointing the finger at the developers with no postgres experience, or downplaying the significance of this event and pretending like it was simply bad luck, and lying to yourselves about how "it will never happen again".

This job posting is completely outside the realm of reason. If that job posting has been up for months or years, I can see its description being exactly why you didn't have the right talent on board to avoid this incident.

>> As a database specialist not only will you work on the database, but also the application's usage of the database. As a result a significant amount of experience with both Ruby and PostgreSQL is a strict requirement.

This is a mistake. The number of DBAs who are good with postgres is very small compared to something like mysql. The truly talented pool for such a position is too small to expect them to also be a developer. The very mention of terms like "ruby" and "programming" should be removed from that job post. It's not a realistic expectation.

They want a developer who can do database stuff, not a DBA (based on that job posting.)

They are who you're replying to

> I'm the CEO of GitLab https://about.gitlab.com/ More information about me is on http://sytse.com

Fair enough, but I don't see him denying it.

Just as an aside, you can hire a remote DBA to help out, until you get a full time person. We have a contracted 3rd party that does secondary monitoring around the clock, and a very, very good pro that we work with on projects. In our case, we have a 'retainer' for 15 hours a month. (and a fixed cost per hour after)

The entire GitLab team is remote, as far as I understand their company model.

https://about.gitlab.com/2015/04/08/the-remote-manifesto/

But maybe you meant part-time?

Wow, that explains quite literally everything. Never, ever, trust a company whose cost-cutting measures involves opting to not pay for office space to run a legitimate business. The attempt to tout the lack of a full office as being "modern, hip, and cool" is a blatant lie from management - it's the same concept as open-office floor plans, reimagined and magnified by a thousand. They get to save hundreds of thousands (possibly even a couple of million) dollars a year; meanwhile, you get the exactly the kind of product you'd expect from a bunch of people pretending to be productive from home and working in such a disjointed fashion.

It is always a HUGE red flag when a company opts to have all or a majority of their workforce working remotely. It's a cost-cutting measure, nothing more. Cutting costs equates to cutting corners, and the business - and its customers - suffer the deserved consequences.

This really explains the flippant "it's 11pm and I want to go to bed" reaction in their report. The guy doesn't have an office to go to when shit hits the fan. He's sitting at home, with a bloody ssh terminal open, trying to remotely debug critical engineering problems over a slow vpn connection. Alarmingly huge red warning flags.

Cost-cutting measure don't just end at the office space level, they hire cheap developers too. There's quite a big divide between salaries they pay to their team in US and remote team and they get to do it just because they are "remote".

I think remote is better than open office floor plans in many cases. It just have different problems.

The word "backup" doesn't appear anywhere in the job description.

Might wanna revise that today.

They'd probably be better off hiring a "backup engineer" to go through all of their systems (end to end) ensuring full backups exist and work for everything.

Instead of backing up every system in isolation, have well though out backup/restore processes for all parts of their operation.

Saying that as someone who's done exactly this before (professionally, for mission critical places). ;)

eg:

· Inventory the systems (boxes, services, etc)

· Determine what each needs (package dependencies, etc)

· Create scripting (etc) for consistent backups

· Work out the restore processes

· Make it work (can take several test/dev iterations)

· Document it

And also (importantly):

· Have the ops staff perform the documented processes, to reveal holes in the docs, and show up parts which need simplifying

Untested backup === No backup

People like to say they have backups or a "backup procedure", but in my experience almost none of them ever tested the backup... Not even once. 95% of the time "having a backup procedure" just means "we have a replica of some data sitting somewhere with no idea how/if we can restore it, or how long it takes".

it's worse. you might be trying to restore from an untested backup and fail, wasting precious time.

five times.

I remember a local company a decade ago where they were obligated to delete part of their data on fridays just to test all the backups and procedures. At the time I thought it was completely nuts. I probably still think it's at least a little bit nuts but I can see the point.

Putting on my BOFH hat, this is what happens when you let Devs do operational stuff.

:thumbsup:

I got thrown on to the team managing netbackup at a massive company a few years ago, and it's actually fairly common that people don't test backups. We sent out notifications constantly warning people that if they aren't testing their backups, they aren't really backing up, but at least once every other month, someone would come back to us wondering why their backups weren't there, when they were trying to recover something.

Reply on news.ycombinator.com