Hacker News Reader

Comment by sytse | Hacker News Reader

I totally agree with needing an experienced DBA, we've had a vacancy open for this for a while https://about.gitlab.com/jobs/specialist/database/

View on HN

To be a bit more specific, it is extremely unlikely (nearly impossible) that you could ever find a single individual who is both a "postgres database specialist" and a "ruby developer". Do you even know what the word "specialist" means? It means specializing in one area - like postgres - without trying to be a Jack of all trades, ie: with ruby.

"DBA" is a full-time position, not an addon to a developer's duties. They are separate skillsets; you don't get a "2-for-1" special by hiring an expert developer + DBA in one person for one lowly salary.

Do you realize this event would have never happened if you had hired a pure DBA? Or do you really believe you can pin the blame on your ruby developers for not being able to wrangle a production postgres database?

The oblivious or intentionally cheap "the DBA must be an amazing ruby developer" expectation shows your hiring staff - or the management guiding them - has absolutely no clue what they are doing. I can just imagine the internal discussion right now; pointing the finger at the developers with no postgres experience, or downplaying the significance of this event and pretending like it was simply bad luck, and lying to yourselves about how "it will never happen again".

This job posting is completely outside the realm of reason. If that job posting has been up for months or years, I can see its description being exactly why you didn't have the right talent on board to avoid this incident.

>> As a database specialist not only will you work on the database, but also the application's usage of the database. As a result a significant amount of experience with both Ruby and PostgreSQL is a strict requirement.

This is a mistake. The number of DBAs who are good with postgres is very small compared to something like mysql. The truly talented pool for such a position is too small to expect them to also be a developer. The very mention of terms like "ruby" and "programming" should be removed from that job post. It's not a realistic expectation.

They want a developer who can do database stuff, not a DBA (based on that job posting.)

They are who you're replying to

> I'm the CEO of GitLab https://about.gitlab.com/ More information about me is on http://sytse.com

Fair enough, but I don't see him denying it.

Just as an aside, you can hire a remote DBA to help out, until you get a full time person. We have a contracted 3rd party that does secondary monitoring around the clock, and a very, very good pro that we work with on projects. In our case, we have a 'retainer' for 15 hours a month. (and a fixed cost per hour after)

The entire GitLab team is remote, as far as I understand their company model.

https://about.gitlab.com/2015/04/08/the-remote-manifesto/

But maybe you meant part-time?

Wow, that explains quite literally everything. Never, ever, trust a company whose cost-cutting measures involves opting to not pay for office space to run a legitimate business. The attempt to tout the lack of a full office as being "modern, hip, and cool" is a blatant lie from management - it's the same concept as open-office floor plans, reimagined and magnified by a thousand. They get to save hundreds of thousands (possibly even a couple of million) dollars a year; meanwhile, you get the exactly the kind of product you'd expect from a bunch of people pretending to be productive from home and working in such a disjointed fashion.

It is always a HUGE red flag when a company opts to have all or a majority of their workforce working remotely. It's a cost-cutting measure, nothing more. Cutting costs equates to cutting corners, and the business - and its customers - suffer the deserved consequences.

This really explains the flippant "it's 11pm and I want to go to bed" reaction in their report. The guy doesn't have an office to go to when shit hits the fan. He's sitting at home, with a bloody ssh terminal open, trying to remotely debug critical engineering problems over a slow vpn connection. Alarmingly huge red warning flags.

Cost-cutting measure don't just end at the office space level, they hire cheap developers too. There's quite a big divide between salaries they pay to their team in US and remote team and they get to do it just because they are "remote".

I think remote is better than open office floor plans in many cases. It just have different problems.

The word "backup" doesn't appear anywhere in the job description.

Might wanna revise that today.

They'd probably be better off hiring a "backup engineer" to go through all of their systems (end to end) ensuring full backups exist and work for everything.

Instead of backing up every system in isolation, have well though out backup/restore processes for all parts of their operation.

Saying that as someone who's done exactly this before (professionally, for mission critical places). ;)

eg:

· Inventory the systems (boxes, services, etc)

· Determine what each needs (package dependencies, etc)

· Create scripting (etc) for consistent backups

· Work out the restore processes

· Make it work (can take several test/dev iterations)

· Document it

And also (importantly):

· Have the ops staff perform the documented processes, to reveal holes in the docs, and show up parts which need simplifying

Reply on news.ycombinator.com