Ask HN: What's your company on-call setup? | Hacker News Reader

Ask HN: What's your company on-call setup?

5 pointsanoncall5 comments

I'm a junior programmer at a mid-sized company (150 employees) with an engineering team of 20 developers.

I'm curious what is typically expected in terms of responsibilities, timing and compensation.

- How often are folks on-call? How frequently do issues occur at your org?

- How much do people get paid? Is it extra when issues happen?

- Is it normal to have all developers on rotation? Or do you have a separate operations team?

- Can you work remotely? Are you expected to be available at short notice? Do you have a second level backup person?

Any other insights I should know about are welcome. Thank you! :)

View on HN

I work on a large SaaS product that is deployed in AWS globally and is expected to have 100% uptime. Each service team in the SaaS has its own on call rotation and is responsible for being highly available. There are also architects and more DevOps folks (SRE team) always on call so you can always get help at 2am if you really need it.

I'm the primary on call every 3 weeks, but secondary the week after. Technically, since I'm the team lead, I'm always on call if the issue escalates to me at level 3. In practice my team is very good and it's extremely rare to get a page when you are not primary.

The team doesn't get paid extra, but it is mentioned in the interview completely up front and my company's compensation is pretty solid.

If paged after hours you are expected to work remote. If you can't get to a computer in 30 minutes you should escalate the issue (or just ignore the page). This is company policy, but I expect most companies to have some type of SLA for responding to pages.

As far as what type of pages we get...when there was flooding in AWS Australia a few months back I got paged because we lost a certain number of AWS nodes. Sometimes we deploy code with a memory leak and we find out a day or two later. Often, AWS just does something stupid (like the AWS S3 outage a few months back). Sometimes the DB cluster loses a node. Sometimes we ship code that interacts with another service in unintended ways. We NEVER interact with customers, all our pages are in response to specific technical issues. If customers need to be contacted there are people with customer service skills that convey the message appropriately. We do not get paged out for issues that are not major (e.g. the product or a major feature stops working).

What's expected? As an owner of a software company that requires on-call staff (I'm basically 100% on call all the time): If they did not mention it during the interview, nothing is expected of you on-call. If they bring that up, renegotiate your contract.

- How often are folks on-call? How frequently do issues occur at your org?

I worked in IT that had a rotation. We'd have 6-8 techs and just rotate weekly. However, issues were infrequent: maybe 1 every 2 weeks. Minor things.

- How much do people get paid? Is it extra when issues happen?

Depends. Maybe your total compensation is worth it to you. Maybe not.

- Is it normal to have all developers on rotation? Or do you have a separate operations team?

It's absolutely normal to have only some devs on rotation. It depends on your business.

- Can you work remotely? Are you expected to be available at short notice? Do you have a second level backup person?

Legally, at least in most US states (if not all), if you need to be available you must be paid for it if you're not salary exempt. So if you're not being paid, feel free to go camping.

Always have a second level guy. Then a number that gets called to ping the CTO/CEO if that second level guy doesn't check in. There are systems for this.

All of this depends on the type of service the company offers, your contract, your country, and other stuff.

For example in my company we have 5 support engineers who self-organized their rotation. The important thing is that someone is available from 1 pm to 2 pm to 6 pm till 8 pm, because one of our clients pays for that time range.

The engineer needs to answer phone calls and emails in a timely manner, depending on the issue (from 30 minutes for issues that impact business operation to 4 hours for minor stuff, like resetting a password).

They get paid a little for the time they're available and roughly the same as overtime if they actully need do something.

This is in Italy and, I want to really stress this, it's how it works at my company and with our customers.

I truly don't get what kind of company is expecting software engineers to be on call.

What do they even want you to do about issues clients have with software? If the software is down for some reason, call devops. Or call AWS. I don't care.

If there's a bug, its not going to get fixed at 4am by one person pushing a hotfix through production. Although if thats how bugs get fixed at your company, maybe you should revisit whether thats a good approach to devops and when you find that its bs, you can finally abolish "on-call rotation".

"btw, you're on-call.." "if I wanted to be on call, I'd be in medicine."

> I truly don't get what kind of company is expecting software engineers to be on call.

What if you have clients who pay you big $$$ and they work all the time? That's not that unusual.

> What do they even want you to do about issues clients have with software? If the software is down for some reason, call devops. Or call AWS. I don't care.

Very narrow view. You can have a software team of 4 people and bring in 10-100 mil contracts. There's not a DevOps staff in this case. And what would AWS do?

> If there's a bug, its not going to get fixed at 4am by one person pushing a hotfix through production.

I've solved 100s of problems ~4am by doing just this.

> Although if thats how bugs get fixed at your company, maybe you should revisit whether thats a good approach to devops and when you find that its bs

- Buy/start to manage legacy app for XYZ company

- Legacy app breaks

- Need to fix it

None of that has anything to do with the software company being a bad company. Things break.

> "btw, you're on-call.." "if I wanted to be on call, I'd be in medicine."

I'd be super mad if I didn't know I needed to be on call at a company and they just told me I needed to be.

I'm not arguing that it should be expected in software, just that there are plenty of software companies that require on-call staff, like mine.

Reply on news.ycombinator.com