Building an in-house on-call training program for your SRE team
fiberplane.com
fiberplane.com
I think that employee training in general is essential to the overall efficiency and quality of your business. Companies always skimp on it, and you can see how it hurts productivity and the bottom line. But it's not necessarily going to impact your culture or day to day practices.
The thing that's going to maintain a successful incident response practice is follow-through. Training just tells them what they're supposed to do. But if they're not doing it, training doesn't matter.
What's the most common problem with on-call? It's not waking people up in the middle of the night, or a lack of runbooks, or lack of postmortems. It's the follow-through. When you have an incident, and you "deal with it", and even have a postmortem, what's the last thing on the page? "Action Items" or "Next Steps". And are those action items actually getting done? Is a regular review of your practices getting done? Is the same alert popping up again and again? That is the biggest problem with on-call, and training doesn't solve that.
To run a good incident response process you need people who are getting paid to do two things: 1) due diligence, 2) continuous improvement.
1) Give someone the primary responsibility of ensuring everyone is following the process. Do not let action items stagnate. Do not let weeks go by without writing the postmortem. Do not have alerts without resolutions, or runbooks for that matter. Somebody needs to get paid to do this, because it's boring as hell, not fun, but needs to be done.
2) Make sure the process is getting easier, and better. You don't want to have to check up on everything all the time, of course, and it doesn't scale well. So you want to improve your process so that it's more likely that the right things get done the first time.
You can train your folks what to do. And you can "get by" with an incident response process that nobody actually follows and is never improved. But as incidents mount, it will drain your people and org more and more. By all means, if you're currently facing a deluge of incidents, do training! But the due diligence and continuous improvement is what's actually gonna push incident numbers down.
Still, it's nice to have SREs who understand the rhythms of an active incident instead of somebody whose last conversation with someone outside their immediate team was in 2016 and seemingly doesn't understand that they're on call or for what services.
This builds skills and confidence for the learner, and has the additional benefit of exposing pain points in our tooling.
Forcing people to sit by their phones/computers 24/7 unpaid waiting for a system to maybe have a problem should be a practice explicitly barred by the FLSA.
Slightly off topic: stdout's On-Call.
If the SRE is unable to solve the issue and needs to page the escalation engineer they should get OT for that if it outside business hours. The whole "oh you got woken up at 3am just log in late" is BS because I still have the same amount of work to do.
I am constantly thinking about how abusive this practice is
That's almost 6 weeks of pay in a single week. Yet we still had to pull teeth to get people to be on-call because of how much it sucks.
On-call for many companies (most I've worked at in the past 23 years) is a borderline abusive practice; and often times they already run the team on fumes.
Overtime isn't necessary if expectations are set up front and the salary is boosted accordingly.
But yeah, it makes people feel better if you pay them less overall and then give them overtime when they're on-call. It's also better for the company because then the raises are smaller since they're based on the base salary and not the overtime.
Honestly I'd rather just get paid a higher salary with expected on-call hours up-front.
If you pay me N extra hours, where N is less than the hours I worked on call then you stole from me. Plain and simple. This should be pursued by rabid labor lawyers and interest at the prevailing 30 year rate should accrue on those dollars hourly until they are paid back.
The problem is, of course, the law dictating “exempt workers” was written in a time where the only people that would be exempt were executives with actual million dollars golden parachutes. Now, white collar labor is almost universally exempt allowing the state sanctioned theft of millions of dollars a year by corporations who lobby to keep the laws the same, and promoted by “grind culture” immoral bottom-barrel blood-suckers like PagerDuty.
It’s bullshit. You’re suffering from Stockholm syndrome. You’ve somehow managed to say stealing from you is okay because they put some nominal amount of unrelated dollars in front of you.
Someone who is on-call should be exempt from overtime if they're not paid hourly and it was clearly communicated up front. I think that's part that's missing. Most jobs don't make that clear up front.