Comment by taco_emoji | Hacker News Reader

Comment by taco_emoji | Hacker News Reader

Parent Full thread

It heavily depends on the quality of management. For a system that needs 24/7 uptime, off-hours support issues are inevitable and it's reasonable for a company to have the people with the best ability to troubleshoot (developers) handle that stuff when it comes up.

HOWEVER: Is management dedicated to making sure those issues are rare? Namely:

1) Do they give you the time and leeway to fix technical debt that causes these things to pop up?

2) Are there reliable code review, continuous integration, and QA processes that ensure that fewer bugs make it to production in the first place?

3) Is it easy to roll-back a deployment at 2am on a Saturday?

4) Is there a well-maintained schedule of IT and development changes, with impact assessments, so that people don't page you during a downtime they should've known about? And so that, after a failure, you can view historical data and determine the causes of a failure and effectively develop a plan for mitigating it in the future?

5) Can YOU page the DBAs at 2am on a Saturday when you need their help? Are they going to be rude when they call you back, or are they going to recognize that the health of the systems is their job, too?

6) Do devs willingly, openly own up to the bugs in their code, in front of their bosses, without fear of serious reprimand? Does the company recognize that mistakes are inevitable and that process and communication are better than blame-finding for preventing failures?

The answers to all of these questions (and more) will, directly or indirectly, indicate the frequency and overall stress of carrying a pager for a given company. (They're good questions regardless of pager duty, too.)

I agree with these points.

I'm a big fan of developers being on call for their application. It puts the pain where it belongs--with those building the systems (modulo lower-level errors--such as power failures or network outages--those should go to the appropriate place).

However, that pain should only rest with the development team if they also have the freedom and will to spend time dealing with it. They will have spend time (either a constant tax, or more likely, with occasional sprints) to reduce operational pain. They are in the best position to reason about the tradeoffs and pragmatically reason about priorities.

In my experience, this produces the highest code quality and the highest team morale. I also like the rule -- if you're paged during the night, sleep in the next day.

I only agree with this if the developers also get to choose the deadlines for the app/features they build. All too often, the higher-ups want a feature but don't want to take the development time necessary to build it properly. And shortcuts are taken all over the place to get the feature done in time. The higher-ups don't care about doing things correctly and don't have any pain when things go wrong. Which makes it more likely to push this sort of behavior.

I think developers should build code that fails in predictable ways with useful error messages that a support team can use to solve problems. If the support team cannot fix the problem with the information provided, then a developer should need to get involved. This way, developers only feel pain if the code they write fails in ways support cannot handle.

I think we should automate recovery for error conditions where possible and change business processes to be automatable where not. If neither can be done for some pressing reason, then the failure condition should be defined as an expected condition that needs dedicated staff to recover. But that cost should be surfaced and tracked and the first and second order approaches should be automation above all.

Of course, teams need the authority to solve the pain if they also have the responsibility for it.

Reply on news.ycombinator.com