TimeWeSp

I've been putting together another oncall improvement, for a completely different area: making the scheduling of oncall work better (oncallscheduler.com). I'm trying to get feedback from people who work oncall about: 1. Is the scheduling of oncall work a significant pain point, and worth making more fair, predictable, and controllable by everybody. 2. Does the solution oncallscheduler.com is built on make sense? If you can spare a few mins, I would love it if you sent any thoughts to kristian@timewesp.com.

Isn't it amazing that PagerDuty hasn't jumped on the AI band wagon to do things like this themselves? It seems they're leaving the door open to let in all the new oncall platforms into the market: incident.io, Firehydrant Signals, Rootly, ... Perhaps someone from PagerDuty will see this and buy your tool. It seems like exactly the kind of simple and immediate help on needs when responding to an incident, to have the biggest impact in resolving the incident quickly without redoing a bunch of work someone already did for an almost identical incident last week.

I think what you describe is the ambition of every oncall team. But even when this works really well, so the calls are few and far between, just being scheduled to work oncall means you're more limited in what you can do. You've got to be reachable by phone. You have to be within a few minutes of accessing a computer with good networking. There'll be no road trip drives or flights to see the in-laws while you're the one on oncall duty. Do you think people prefer to avoid just being on oncall duty over big holidays, even if there aren't many calls? Perhaps that preference only shows up in a small portion of oncall teams.

Didn't you still run into times when most people wanted the time off? For example, times when schools are off, so everybody who is a parent wants that time off, regardless of whether it lines up with their cultural or religious important dates?

Oncall for small engineering teams is not easy. With less than 6 people on a rotation, I think one just has to accept some lower guaranteed uptime, by having hours when there is no oncall coverage, at the time when the system sees the least customer usage.

To make oncall scheduling work better for everybody, once you have at least 4 oncall engineers for a rotation, check out https://oncallscheduler.com. it doesn't solve Oncall compensation, but it makes planning for vacations, fairness about who works Oncall on holidays, and such, work well.

I've experienced all these problems. There are solutions trying to address them. E.g. https://incident.io/ (which I'm not affiliated with in any way). It's not easy though. I think they all come from the root cause of teams not investing enough into making oncall processes and solutions good, and in particular not keeping things up-to-date. As you say, runbooks are often outdated. The same happens with lists mapping component ownership to teams.

There's another problem (#8 to add to the list) I also felt pain from: how you're scheduled to work oncall. We had ad-hoc manual scheduling of who would work oncall when. A tool for solving that is https://oncallscheduler.com (which I am affiliated with). It automates the oncall scheduling, while making it fair, predictable, and gives all engineers self-service control over when and how they're scheduled. I'd love some feedback on it.

That's a lot of great, rational, decisions for how to set up those rotations. On the scheduling front, how do you decide which engineers will be working which shifts? Have you considered scheduling solutions like https://oncallscheduler.com? It makes the scheduling automated, fair, stable, and gives the rotation members control over when they work. It writes schedules into PagerDuty, and gets the times when people work onto their calendars.

+1 to that. You're right it's more of a bidding system, where game theory is needed to figure out if it'll be a balanced "game".

There are several aspects to it that aim to make it feel fair. First there's the credit system, and people who take less desirable shifts getting more credits, so they end up working fewer shifts in total. But there are also limitations on what the scheduling admin can do. E.g. they can't assign credits to themselves, or enter "block periods" (= a calendar time when someone is blocked from being assigned any shifts, e.g. because they're on parental leave or military leave, or something like that). There's also a "credit accounting log", which shows all the transactions in the system, and how those transactions changed the credit balances of all the rotation members. I hope (and think) that the kind of geeky game-prone people who work tech oncall jobs might appreciate the game-like nature of this system.

Oh, you're going way farther in optimizing the schedule, than I know how to collect data to do. If I could collect detailed info about how each oncall rotation members feels about every single shift, I could use a model like what you describe. I remember these models from school... But that was a long time ago. I'd have a lot of reading to do.

What I built now, is pretty simple, but I think it solves the problem pretty elegantly: * Each user has a set of credits. You get credits when you get a shift assigned. You pay credits to other people who are assigned shifts. * You can self-assign shifts (which makes you earn credits). * You can bid credits to avoid getting scheduled for certain dates/holidays(e.g. Memorial Day)/importantEvents(e.g. Super Bowl Sunday). * Shifts which nobody has self-assigned are auto-allocated up to a given time horizon in the future. The auto-allocation of the next shift, assigns it to the person with the lowest value of "currentCredits + bidsToAvoidThisShift * rotationMemberCount / 2". - The exact algorithm isn't that important. But what it does is: it gives a shift to the person with the least credits (and they'll then get a bunch of credits), but it avoids giving shifts to people who have bid to avoid that specific shift.

The real value is that there is an algorithm doing the assigning. That makes this feel FAIR to all involved. If you didn't bid to avoid working on Christmas, don't complain about working then. But you'll also get more credits to work on Christmas, because you get everything that others bid to avoid working on Christmas (which means you will work fewer shifts in total over the year).

All of us who have worked oncall for years, know how stressful and difficult it can be. Among all the pain though, I've always felt that the scheduling of who-works-when is something which should be solvable in a great way. I've tried to build something which solves oncall scheduling. This post is a shameless plug for that thing. But besides trying to get someone to click on the link, I would also love to get some feedback and hear some input from all of you who work oncall. Is the scheduling painful enough to be worth my time to solve? What I've tried to do with this tool is: Make the scheduling FAIR. There's math and an algorithm deciding the schedule. All scheduling actions are available for all to see in a log. Make the people who get scheduled have great CONTROL over when they work or (more importantly) _don't_ work. Maintain a PREDICTABLE and STABLE schedule at least several months into the future, which doesn't get all scrambled if someone leaves the team and their shifts become holes in the schedule. Make the scheduling FULLY AUTOMATED so nobody needs to think about it more than at most once/quarter.

What do you think? Is this a problem worth trying to solve for real? If you look into the tool, do you think it looks like a good solution, or does it miss the mark somehow?