Hacker News Reader

Comment by oncallthrowaway | Hacker News Reader

As an in-demand software engineer with oncall experience at a well-regarded company, I will not consider jobs that require me to be on call.

Jobs with oncall don't offer more compensation than jobs that don't.

Scheduling my life around being able to answer a page is inconvenient, and waking up in the middle of the night is something I'd rather avoid.

Operational work is often not considered as important as feature development for promotions, so you feel like you're wasting your time when doing it.

In my experience, system quality is completely independent of whether the developers do oncall or not. But I'd welcome objective data that proves otherwise.

There is no upside for me as an individual to take a job with oncall responsibilities.

View on HN

Companies need to evaluate the loss of man-days when they demand or even suggest on-call duties for engineers. Also the effect of these duties on number of sick days taken, and overall health. Because when I'm up from 3-3:30am fixing something, I'm going to be less useful and emotionally testy that day at the office. When I'm a Good Marine and am up eyeballing a problem from 1am-6am, the same will happen but over a 1-2 week span. And I'll probably get sick.

I really think engineers end up on-call because companies give short shrift to documentation. Maybe if you afforded time to document systems and processes, you could outsource on-call duties, or trust sysadmins to remediate without developer intervention.

Operational work is often not considered as important as feature development for promotions, so you feel like you're wasting your time when doing it.

Which is why developers need to be on call, otherwise they don't build operational support into apps (monitoring hooks, useful logging, etc)

At my company devops is primary on-call, with a developer(s) as backup, if devops can't fix the problem quickly, they escalate to the developer. If they can't reach the developer or the developer can't fix it, then the entire release is rolled back and then the developers bear the brunt of the release schedule changes.

Without developers in that loop, it's hard for devops to get the tools they need to diagnose and fix problems.

> At my company devops is primary on-call, with a developer(s) as backup

Sort of defeats the purpose of Dev ops, if you have a Dev ops team and a Dev team, doesn't it?

Does it?

Dev Ops is focused more on operations (tooling around releases, monitoring, etc) as well as doing initial triage for system problems to figure out where the problem is, and if they can't fix it themselves, determine which engineering group can help resolve an issue. Most of the Devops staff does development, but more around operational needs, product features are implemented by Dev.

Dev Ops means different things to different companies.

Yep. This is how my org does it. DevOps builds automation/tooling to help the operations focused developers on the dev teams do their jobs better.

I think it's fair to say that you're just ops since you have the exact same functional role. And you can suffer from the same organizational issues that prompted the creation of the devops movement.

That said, there's a need for a way to divide the industry between traditional ops people and those with the ability to handle modern infrastructure and develop custom tooling where needed. And for better or worse the industry has decided that "devops" is going to be the way to determine that. Now the problem is all of the fakers (both individuals and teams) adopting the moniker. So now we're back to square one and we actually need to evaluate people and teams based on their knowledge and skill.

We're more integrated with Dev than a traditional "ops" role where Dev throws the code over the fence at Ops and says "Deploy it!" and Ops throws it back if there's a problem and says "Fix it!"

Our devops team sits in on design and code reviews to help ensure that operational needs are met early on in the process, and we'll change code to fix bugs or support operational needs. We're not full developers and won't rearchitect entire systems, but we will do bug fixes when we can.

So now we're back to square one and we actually need to evaluate people and teams based on their knowledge and skill

Have we ever left that square?

Sounds like your devops and dev teams do devops which is great. I wasn't trying to imply that they don't or that they had the same problems that prompted the movement, only that your roles are divided the same way as traditional dev and ops even if you do in fact do them smarter than average. What I'm getting at is that "devops" as a title is an (internal) marketing term.

You're right that every org defines devops differently but it's totally and unashamedly coopted from a movement which, despite being very generally defined, defines it radically differently. And that's not to say it's always a bad thing either. Sometimes it takes a title to help signal change and make an organization better which in the end accomplishes the goal of the movement.

And I agree, I don't think we ever left that square and we never will. But if prepending "dev" to "ops" wasn't at least somewhat effective in getting something out of people in decision making positions it wouldn't be tacked on to job titles and team names left and right. So at least some people think the marketing terms in a job title are a meaningful shortcut even if smart people like you know better.

If you have a DevOps team you are doing it wrong.

The word Devops means that the Dev team is the Ops team. If you have both teams, you are defeating the purpose of DevOps

Actually devops is supposed to be a culture where the dev team and ops team work together. What people seem to want it to mean is you're an ops team with automation skills. I've yet to see this be a dev team doing ops.

It goes a bit beyond "an ops team with development skills" in my company -- someone from Devops sits in on design and code reviews to suggest changes for operational needs (which almost always means adding metrics so we know how busy it is, and when it breaks or is near breaking, followed closely by adding enough logging so when it does break, we know why and don't need to page a developer to read a stack dump to try to figure it out).

But ultimately, the developer is the one that best understands how their code may break so he needs to instrument the code accordingly. And when the developer knows that giving the right information to Ops may avoid a 3am phone call, they have good incentive to do so.

Typically a DevOps team would mean a team that works on things that make it easier for the whole organization to do DevOps, e.g. tooling for easier building, deploying, testing, configuration and monitoring. It doesn't defeat the purpose of DevOps unless the people at the organization think that it's the only team that needs to be doing DevOps.

So much this. I don't trust an engineer who's never done maintenance work. And I really don't trust an engineer who's never been on call. I find that they write vastly different (usually much simpler) systems. Personally I will now only build a system that I know I can troubleshoot at 4am while drunk.

I 100 percent agree. I'm sorry you feel the need to post under a throwaway, but I'll put my name on this to back your point up.

On call is a deal breaker for my future job searches, and I am considering leaving the company I just joined because they sprung it on me while never mentioning it in the interview process.

On call is justified in my opinion only if someone will die if the problem isn't handled. Or if compensation is dramatically increased and agreed upon in advance.

as someone who runs an on-call team, and is woken up frequently still as a co-founder, if someone sprang that on you, you should quit asap, and tell them why. people need to know that on-call is a decision, not a mandate.

the only reason my team puts up with it is because we're 100% remote. they never leave their families, so that's the trade-off.

Grumble grumble. There are times I'm treated as oncall despite being hourly. Figure that out.

consultants charge double during on-call hours. it's standard practice.

Unplanned == big $$. If you have to travel to the customer you can charge double your hourly rate and an emergency trip fee + expenses.

No just an hourly employee. Support technician at that.

That sucks, but hey, at least you get compensated for it. I think in some states they even have to pay you for a certain amount of time at minimum even if you only work for 10 minutes.

It happens very rarely. The times it does they do it under the guise of me being potentially fired if I answer wrong. Ie they want immediate clarification of what I did in some crisis but really just want info. Not paid and I'm actively searching for work.

find a new job buddy. nobody is going to stick up for you but yourself.

On call is justified in my opinion only if someone will die if the problem isn't handled. Or if compensation is dramatically increased and agreed upon in advance.

What if the company will die if the problem isn't handled? Many companies (mine included) provide a service that needs to be highly available 24x7 (customers run automated tasks 24x7 against our service). If our site regularly went down for hours at night (or even for an entire weekend) because of a software problem that no one could fix because the developers weren't picking up the phone, we'd lose customers and eventually, the company would go out of business.

Even a 10 minute outage is a significant event and requires a full RCA for customers. We try hard to architect for high availability, but bugs do happen.

The answer here seems simple. Include on-call time in compensation, and fire developers.

Note that for some developers, you're never going to be able to compensate 'on-call' hours appropriately due to their evaluation of the opportunity cost of their time.

> Include on-call time in compensation...

I'm glad you brought this up - there's a quick conclusion to this conversation:

"On-call time compensation is part of your salary."

Yeah, the labor laws are broken.

Salary shouldn't be a thing that a company can ever hide behind.

Labor laws really do need to cover the maximums that an employee can be expected to work. They should also make going above those maximums exponentially more expensive in 'bonus time'; and the accumulation rate shouldn't magically reset after time, but only after time back on normal/reduced work load and duration.

Also, while I'm on this subject, 'full time' work should really begin at more like 24 hours / week. Benefits for part time work should be pro-rated. (It should never be more cost effective to split a full time job in to part time jobs. That is defrauding the economy and making others pay for the costs of your labor.)

It's a valid answer provided that on-call was part of the conditions known during hiring and salary negotiations. If we've negotiated a salary of $X for doing Y, but then you suddenly want Y+on-call, then you're going to either pay more or find another employee, because $X is simply too little for that.

The real problem then is engineers being unwilling to vote with their two feet.

In a market as friendly to software developers as we're currently in, the correct response to that is totally "great, consider this my two week's notice, I'll go to this other company across the road that pays the same salary but doesn't demand I have on-time call".

(No, the response to every problem with your work environment should be to get a new job, nor is that even a realistic possibility for many. The economic argument still stands.)

That's just fine, as long as on-call expectations are fixed as part of the job offer, just like salary is. If the on-call expectations change for the worse later, that is exactly the same as a salary cut.

IANAL but I believe (in the US or in NYS at least) this would be grounds for quitting with full unemployment coverage (you are not usually eligible for unemployment benefits if you quit, exceptions include the basic nature/hours/wage of the work being changed without your consent, since that is philosophically similar to having your old job terminated and being offered a new, worse one)

Get any on call expectations in writing when taking an offer if you can, just like you'd expect salary in writing

"What if the company will die if the problem isn't handled?"

Then they should be paying huge bonuses to be on call, or simply hiring for multiple shifts. It's pretty easy.

Maybe they "should" do one of those, but it seems unrealistic to expect a small 10 - 30 person company to do either.

And when the company grows large enough that they can support 24x7 engineering staffing, they usually do it by having an overseas team.

And that overseas team often has little-to-no ownership so they call people while they're sleeping or half ass everything. This problem is magnified if they're contracted and not actually employees.

If you can't pay enough to find someone competent to be on-call, you have to do without. That's life.

"Maybe they "should" do one of those, but it seems unrealistic to expect a small 10 - 30 person company to do either."

Why? What's so special about a tiny company that they get things for free? What's so hard about, "If you want something, then pay for it"?

Because, math? That company may have a team of 5 developers, how do you split those 5 developers across 3 8 hour shifts that cover 24 hours/day 7 days/week + holidays + vacations + sickdays to get 24x7x365 coverage without doing an on-call rotation.

When you're in a 100 or 1000 person company, then it's easier to have dedicated after-hours support staff (or staff working from multiple timezones around the world)

No one is saying that it should be "free", it's built into the salary - every company that I've worked at that's had an on-call is very clear about on-call rotations during the interview.

"Because, math?"

Except you're saying that small businesses should get to ignore that math, and get stuff for free.

" That company may have a team of 5 developers, how do you split those 5 developers across 3 8 hour shifts that cover 24 hours/day 7 days/week + holidays + vacations + sickdays to get 24x7x365 coverage without doing an on-call rotation."

You hire more people so that you can, or you don't make deals you can't afford. Honestly, why is this so hard to understand? Why do you feel so entitled to things you can't afford?

Again, who said anyone is getting anything for free?

My company has 100+ engineers, all of whom do a on-call rotation, they are all aware of it when the sign up, so that on-call duty is included in their salary. The employee can evaluate for themselves whether or not they think the salary is sufficient to cover that, but as far as I know, we've never had a candidate turn down an offer due to the on-call requirement (though I don't know that they'd always tell us that's why)

But really, does any engineer join a small company (in the USA) without assuming that they'll be on-call after hours? Even in early stage companies that haven't launched a product yet, there's still after-hours support to be done to keep dev systems running, fixing broken builds, etc.

Did you ever ask about it during the interview?

I am compensated, beyond normal salary, for every hour of oncall outside of normal working hours, up to a cap of 15% salary.

"Jobs with oncall don't offer more compensation than jobs that don't."

This sounds particular to your experience. There are definitely companies where on-call time is compensated.

I think its more: Comparing multiple companies, my total yearly income won't be any higher if I pick a company that requires oncall than if I pick one that doesn't, even if the one that requires it compensate for it. I can do the same job at company X for 200k without being on call, 175k at another that requires it, or vice versa...there's no correlation.

So if its not obviously worth my time, I just wont' do it.

As someone occasionally told my job is becoming irrelevant by people here on HN, I hope you folks have some regard for sysadmins who do on call regularly and (likely) get paid drastically less to do it. ;)

I've been told here that now developers can and should do everything, but it doesn't seem like a lot of developers want to be called in at 2am.

> As someone occasionally told my job is becoming irrelevant by people here on HN

HN is an enormous echo chamber; Take it with a liberal amount of skepticism.

I think the term developer is a bit loaded. If you're properly developing an application, you should know what your dependies are, where you store your data, what are the failure condiditions are, how to prove your appication is actually functioning.

In my world, developers barely understand the programming language or framework they're using and are more interested in using buzzword foo rather than solving the problem at hand or dealing with actually running the systems that solve said problems.

As someone who does both dev and ops jobs, together and seperatly, I think your job will be safe for a very long time.

"In my world, developers barely understand the programming language or framework they're using and are more interested in using buzzword foo rather than solving the problem at hand or dealing with actually running the systems that solve said problems."

Oh, man. A thousand times this. Many devs I work with have been writing code that runs on AWS for years, and don't know what RDS means.

Do you write production code?

Reply on news.ycombinator.com