Ask HN: How do you balance support and sprint tickets? | Hacker News Reader

Ask HN: How do you balance support and sprint tickets?

Our DevOps/SRE team has been trying to do Scrum for over a year but the sprints are always derailed by support work.

They've tried to add support tickets to the sprint as they come in, but that changes the sprint scope and renders the "velocity" metrics useless.

They've tried to exclude support tickets from the sprint and keep the scope very small. Some weeks it works, other weeks they overdeliver by a lot (because there were fewer support tickets and they got a lot of tickets from the backlog).

I'm thinking Scrum is useless for that team but management insists all engineering teams must use scrum.

Do you have any advice for managing tasks in a team that has support/adhoc and project tickets?

View on HN

I've been working on and/or managing both dev teams and devops/ops teams, and it is my very strong opinion that Scrum is terrible for devops/ops teams. No managers (besides me) want to hear it, but it's true. It provides the wrong incentives to the team and leads to pain and sadness all around, and eventually production outages and employee turnover. The Scrum game (heavily) incentivizes delaying support work at least until the next sprint, which frustrates the hell out of both teams. If you're doing the whole roadmap thing with goals and stuff, it's even worse because then the incentive is to either #WONTFIX or half-ass it or somethign else to make it go away so you don't attract the eye of Sauron from the spreadsheet overseers by slipping on your "already pretty aggressive" goals.

What to use then? Kanban, with little to no deadlines (estimates are ok but be careful as it's very easy for an "estimate" to turn into a "this work is late because we thought it would be done by X.")

Time to market tends to dominate most conversations around planning, but as I look back on a 10 year stint that I recently finished with a company, I am firmly convinced that this is the wrong strategy.

Time and time again, I saw teams making compromises and taking shortcuts to shave a week off of a project timeline—only to have a crummy and bug laden release create negative first impressions with our customers.

Many features were poorly validated for market fit and were sunset just a year or two after release, as the maintenance cost outweighed value.

My (admittedly biased) opinion is that the best way to deliver value to a business is to spent a lot of time and energy figuring out what your customers actually want, which is usually just a few key things, and then take the time to build those features the right way (less time pressure because the surface area is smaller).

This lowers the ongoing maintenance cost, as the team has the time necessary to do the work properly, and increases CSAT and retention.

When it is time to build something new, the team isn’t so bogged down with maintenance and debt, so they can focus more directly on the problem in front of them.

Sadly, it rarely seems to go this way.

> My (admittedly biased) opinion is that the best way to deliver value to a business is to spent a lot of time and energy figuring out what your customers actually want, which is usually just a few key things, and then take the time to build those features the right way (less time pressure because the surface area is smaller).

I'm of the exact opposite opinion, but we agree on almost everything, funnily enough!

I think the best way to deliver value is to quickly write small, fast, rock-solid but relatively un-featured prototypes and see in which direction production feedback indicates you should evolve those.

The above is based on two things:

1. You can't know what your customers want because not even your customers know what they want. They only know what they don't want once they see it. You can spend a lot of time an money on research but the MVP is cheaper and a stronger signal.

2. The same idea you put out there: most things are bad market fits and should be sunset fairly quickly. By building small things, you optimise for this case. By also making them reliable, you don't make it too painful to evolve them later on. (Easier to add features to quality software than add quality to featureful software.)

This, of course, also leads to less maintenance and higher pace of innovation down the road!

This is why many of us ops types are annoyed at the current popular fad of rejecting the unix philosophy of keeping programs small and task oriented, but able to pipe and be piped as part of a greater workflow putting those things together.

This is exactly it. I see whole stacks, and theres seems to be always a key app or microservice or two written by a particular team, and it usually: has bad log formatting, doesnt multi-thread or muli-task well (concurrency and parallelism), have some non-standard or extranneous db or rpc/api queries that impact entire flow with locking, etc...

If the focus was to make sure the services were rock solid and that was the focus it saves so much future pain and tech debt digging.

On note of what the customers want: I think the approach I prefer is to not spend much time on it, and make it fast and intuitive - quick brainstorming brings rawer truths to the surface often if done frankly.

Alas, all that said, one can't always manage upwards. The c-suite and the boards are so often high on their own airs they stop listening to engineers or have let the wrong ones become filters, and there is nothing you can do to change that unless you are on the board and can fight at that level, but I don't know many engineers that can. This is the main irony of SV to me.

I'm still looking for something that Scrum isn't terrible for.

It works if you have dedicated team and for a restricted time period, say half a year or maybe a year.

Like sprints work in sports.

I don’t see it working as a constant modus operandi.

And if the scrum master is worried about losing their job, replace the scrum meetings with retrospectives. Devops should have a lot to talk about in retrospectives (what went wrong, patterns, etc) and the scrum master can still do their thing so is more likely to go along with it.

Don't do Scrum when your work priorities changes more often than your sprint length. If they want to do Agile, do Kanban. Or Scrumban. Something where the process fully supports popping a new ticket to the top of the priority stack.

If there is any one thing teams new to Agile need to learn, it is that Scrum is just one choice among many. And nowhere near as universally useful as people seem to believe.

Your point cannot be overstated, I find scrum great for new product work, but pointless for maintenance that comes in ticket by ticket.

HOWEVER; The counterpoint would be if you are doing new product work and your priorities are changing faster than a week or two you have either an exceptional situation or this should be a canary that you have something really wrong with your priority directions.

In the latter case scrum is being a useful tool in bringing this to your attention.

I’ve always found that Scrum is suited for 2 environments only:

1. Pre-customer development

2. Contract development shops where you bill by the sprint

I’ve never seen good outcomes in a live product environment.

Usually the answer is Kanban for smaller teams or Scaled Agile for larger teams IMO…but the situation the OP is dealing with looks like they are prioritizing preventative work to stop these support tickets from having to come in in the first place.

Prevention measures for unplanned work are critical.

>> or Scaled Agile for larger teams IMO

I STRONGLY disagree with this, conceptually, logically and in practice. Scaled agile approaches make you plan your sprints way further in advance and account for emerging (or sustainment) work with the use of reserving a percentage of capacity. This works about as well as riding out a run of bad luck at the casino while you wait for the odds to balance out. Scaling "up" the existing practices to a department or company level will not help an interrupt-driven team. Scaled agile doesn't work for pure development teams; i'ts even less likely to work for an SRE team.

Again, this will depend on the size of the team. Planning on a longer time window while giving dev teams a time to coordinate, get everybody on the same page about what you're doing and why...works. It works extremely well.

It gets the plan in front of everybody so you can discuss the tradeoffs and risks too. There's an entire open session as part of it called R.O.A.M. where risks to the ability to deliver the plan are discussed, not just with the tech people but with the business folks too.

That portion of PI planning is where you call out things that could break the plan. Emerging (or sustainment) work is one of those risks and you have to call it out, then once it's called out you have to discuss what the organization is going to do to prevent it. Capacity isn't going to just spring forth out of nowhere, so what are we pushing off if there's more of it than expected? Do we need to prioritize preventative work to make sure we don't have problems? Do we need some type of infrastructure, monitoring system or automation in our devops process?

Everywhere I've been, the PI Planning process has led to less "we have to do this now" and more "we need to see if this can be included in the next PI", which creates significantly less disruptions to the development team focus. The only remaining interruptions should be production issues (preventable) or significant market shifts (rare).

It doesn't work when people try to adhere to that plan so rigidly that it becomes a waterfall system.

LeSS beats Scaled Agile hands down IMHO

Reading about LeSS, it looks like it's basically the same thing? As long as you're doing PI Planning you're a step above virtually every other org out there. It looks like it could be a good fit for some groups I know who aren't quite big enough for SAFe though, while still drilling in the same core ideas.

IMO the biggest issue with Scaled Agile always comes down to leadership IMO. It's not supposed to be rigid. It's supposed to be adapted to your organization and put the plans in the hands of the developers.

The horror stories about SAFe that I've seen on here usually end up with me asking a few followup questions and those questions make it clear that something was terribly wrong. One guy on here told me they did PI Planning for 2 full weeks, which is abject insanity.

"Scaled Agile" is another way of saying Waterfall but allowing you to expense 20% of your team overhead for scrum masters.

Scrum usually arrives the same way grain arrives in the stomachs of fois gras ducks, so unfortunately I suspect the team has no choice in the matter.

This. Scrum is not adequate for this.

What we used to do (while growing the SRE team from ~8 to 30) was having dedicated teams on specific projects. Each had their own scheduling system as they saw fit. AND a weekly rotating support duty which took precedence: teams had to allocate someone from their own capacity to this duty.

The importance was to adequately rotate everyone, to pair a senior and a junior in the support team, and to be available to them to debrief, or to reallocate high priority/criticity tasks to more expert members, if required.

Also, if the support queue was empty, they could do maintenance/exploratory work, but NOT their own team project work.

Nice strategy. I used to be in a large DevOps team and similar startegy was the eventually adopted.

When I was on a support team that was part of a development team, the support people always used Kanban. We were fortunate that management never had a problem with this. We still had daily standups, just to keep in touch with each other, but we did no sprints.

Or do scrum with one week sprints. If the priorities are changing daily that needs to reorganize the entire team. Maybe its not the time to do feature work.

Scrum with 1 week sprints mean you spend a considerable amount of time in useless meetings (retro, planning 1 + 2, and so on).

Not necessarily, those meetings should be much shorter if you only plan for one week.

We designate a support person every week (like a rotating position) and don't count with their dedication (or just a small percentage) for the sprint.

This also helps avoiding constant interruptions or context switching to the whole team that are now handled by the designated person of the week.

What's really great about this system if you're consistently seeing more support work than the designated support person can handle it's a highly legible signal to management that support work for your team requires >1 full time engineer. This makes it significantly easier to advocate for paying down tech debt or creating self-work workflows for people filing tickets.

As an added bonus, this incentivizes better documentation of your system via runbooks (I'd rather point someone at a doc than walk them through a problem when it's not my week).

The breakdown of this though is that sometimes management thinks "oh all (internal, I assume?) support is solved with the cost of only 1 headcount" and then they just let that rotating week become more and more hellish. Then never prioritize anything to make it better because the maximum "impact" of that work is taking the footprint from "1 eng per week" to "1 eng per week" (zero perceived "real" impact. And if you provide alternate metrics to correctly show the load, they'll just say, well clearly it still fits into 1 eng week, good job, keep it up).

And eventually employees burn out and start mailing in that week and now suddenly nobody cares except the other teams having trouble getting support, but apparently that is a diffuse enough complaint that nothing ever came of it.

Not that I speak from experience...

The problem here is bad management. If it wasn't for this failure mode they would find another way to mess people up. No process works well under bad management, except perhaps some variant of anarchy/direct democracy.

I'm not sure I understand - you'd have a constantly growing backlog of support tickets and the time for things to be resolved would grow dramatically. More and more people internally would be complaining nothing was getting sorted.

Under bad management the growing backlog is translated into ever more overtime and hurried low-quality band-aids.

All systems fail in the face of sufficiently bad management, at which point your only recourse it to convince someone higher up the chain that there's a problem. Legibility can make convincing people easier, but nothing is fool proof.

We do the same but count the person's availability at 100%, instead we just create and story point the support ticket as usually taking more or less half the sprint (our support time is half the time of the sprint).

Do you track the designated support person's work? If so, what does that look like?

Not OP, but my team uses tickets to track support work. We use JIRA. We have a service project (SWS) and a software project (SWE). If someone is working on support, you'll see their names on SWS tickets.

That requires (mostly) homogeneous knowledge of everyone in the team though

It encourages it as well.

How many people in your team?

We do the same with a 3-person team. The on-call person spends one week of the two-week sprint focused on support, and usually picks up a few smaller tasks/backlog items to fill in the time if it's a slow support week.

That works out to leaving one team member a full sprint for a deeper heads-down task.

To alleviate some of the extra load on the on-call rotation, we dedicated a decent portion of our sprints to automation for a few quarters, which allowed us to reduce some of the recurring manual tasks and ease the load on the on-call role. Now the on-call person will usually see an average 15hrs of support on the regular (of a 40hr week), down from damn near 35hrs.

That is to say there was about 20hrs of automatable tasks that had been shouldered by one team member for years. When that person left the tasks were placed onto the on-call rotation as a duties as a way to ensure every team member became familiar with these tasks instead of being siloed into a single person (there was a scramble to learn those tasks as this person was offboarded, and a rough couple of months as we skilled up on these).

We do the same in a 6 member team

6 now. 10 a year ago.

I love the way my team does it.

- All support tickets go into a Kanban board separate from the scrum board

- Whomever is primary oncall works exclusively in this queue and is excluded from the sprint for that week. Depending on volume, you can add a second goalie.

- If the support queue is empty, the oncall works on lower priority debt or other OE.

- Oncall changes weekly for us on Monday

As a manager, I love it because it's predictable for planning. The team loves it because they're not constantly jerked around.

This is how we manage things as well. Support is a totally different stream of work from product development.

I've often had a difficult time getting non-engineering management to understand the difference between proactive work that can be planned and reactive work that is unplannable by its nature. All of it looks like deficiencies in the product to a certain cohort of stakeholders. Eventually, once they see things smooth out by separating those workstreams, they catch on.

YMMV.

This is, in my experience, the core differentiator between a tech company and a non-tech company.

This is how our team does it as well. We are 5 engineers at a startup, so inbound support tickets that need to be turned around fairly urgently are common. Having one person dedicated to on call and dealing with support saves everyone else from constant context switching and leaves them to focus on planned work.

We're doing something similar and it seems to work pretty well.

That is exactly what I did on my previous team.

This is the way.

+1, same for us

Scrum is useless for that team, you are right. You can't do scrum if more than 1/2 your work is ad hoc support tickets.

When I ran SRE at Netflix, our general trick was one person is on support for the week. They were the primary on-call, they handled any emergency support. Everyone else was primarily building, or at least doing longer-term follow up work from support tickets.

In major emergencies of course everyone would drop what they were doing and help, or if a support ticket came through for a tool someone else built, it was fine to ask them for help. But at least the mindset was "jedberg is on call this week, so don't expect him to be building anything this week".

We also had good management support so that when we went to review our quarterly goals, and 75% of them were red, they trusted us to know that it was because we had a lot of support load that month. Everyone who depended on our tools knew that they would be ready when they are ready.

Combining both of these into one team is a terrible idea rotten to its core.

This is because the pathway to excellence of one leads to mediocrity in the other. Sprint tickets require an internal orientation, focused work, and blocked-out time. Support tickets require customer orientation, fast response time, and high availability.

If you require your developers to be responsive to customers' needs, they will never have time to do sprint work. If you require them to have excellence in sprint work, they will not be responsive to customers.

There is no way to "time box" these matters, i.e. devote a certain number of hours for this for a certain number of hours for that, because excellence cannot be time boxed. Excellence requires you to do whatever is necessary to achieve it. And these two types of excellence cancel each other out and nullify each other.

> Some weeks it works, other weeks they overdeliver by a lot (because there were fewer support tickets and they got a lot of tickets from the backlog).

OK, this is a problem because why? Some weeks it works, and other weeks they accomplish more than expected... and this is a problem why??

Like, what's the actual problem here, if good progress is being made on features and bugfixes?

My guess it's a problem if management is trying to use "scrum" to treat developers like sweatshop workers who they wring the most possible production out of. How can we know what the most possible production is, if they have a different amount of time to work on it every week?!?

We could come up with different arithmetic ways to solve this. Assign "points" to support tickets too (even after the fact, how much time they actually took?) to include them in your "velocity". Or keep track of how much time is being spent on support to, to "normalize" the velocity of the other stuff accordingly (different way of doing the same thing).

But personally I do not really want to help companies become better at treating developers like sweatshop workers to wring maximal productivity out of. I'd rather realign to "agile" it was originally intended, to empower developers to bring value, not to treat them as commodified interchangeable widgets to exploit and burn out.

but, I mean, exploitive companies gonna company I guess.

> I'm thinking Scrum is useless for that team but management insists all engineering teams must use scrum.

Which they also insist means relying on those "velocity" metrics? Can you do "scrum" without "velocity" at all?

> OK, this is a problem because why? Some weeks it works, and other weeks they accomplish more than expected... and this is a problem why??

OP didn't mention it but it doesn't take much effort to imagine many weeks will have lower velocity because of influx of support tickets. Management focused on metrics won't be much happy.

> I'm thinking Scrum is useless for that team but management insists all engineering teams must use scrum.

All 'support' created tickets go into the backlog then, earliest they can possibly be addressed is next sprint. (Assuming you want the rule to change/it be someone else's problem/proposed solution.)

As a bit of an aside, I'm a bit curious about your role/the org structure that you seem to have oversight over SRE as well as other teams' working processes, but also have 'management' imposing roughly what they look like from above?

Scrum is training wheels. The time boxes are supposed to make you reflect and improve. Ditch Scrum. Go to Kanban, scope your stories smaller and prioritize bug fixes over all new work until you're not writing bugs. Can't ship code without writing bugs? Improve your skills as a team. Start pair or mob programming. Do TDD to improve your design, testability & maintainability of your code. Start moving towards trunk based development and use feature flags to do REAL CI/CD (hint: using a build server doesn't mean you're doing CI). Code deployment != feature deployment! Strive to "roll forward" your code as much as possible and disable/enable features if you run into problems - learn to not "rollback".

Now of course, since this is HN somebody's going to sneer divisively at everything I just said and tell me it's not possible (despite the fact that I've done this, repeatedly in different organizations in a developer and coaching capacity for almost two decades now). Here's my preemptive caveat/STFU for detractors: the above method only works if you, as developers, have full control & ownership over your application code data, and do your own deployments or are partnered strongly with an OPS team that gives you full monitoring & Read Only access. If your team is working in an environment where you don't actually get ownership over your code and data, this won't work. If management, architects, or egotistical prima donna staff/senior developers "won't let you" pair/mob, do TDD, do trunk based development, or do proper CI/CD, this won't work.

P.S. If you're in an environment where "this won't work" - QUIT! Life's too short to put up with being expected to build software with one hand tied behind your back. These things are often easier to do in medium to small sized companies. These things are often easier to do on greenfield (or at least recent) projects.

Every sprint we have 1 person in rotation for bug duty. It works quite well. Overtime role takes some refactoring and tech depth as part of bug duty.

I have a blog post on the subject a while back: https://blog.rstankov.com/bug-duty-process/

I think this is what you are saying but to reword it for how we do it.

We have one "on duty" person each week/half-week/two-weeks. Their priorities are:

1. Support tickets.

2. Whatever they want (usually code cleanup).

They aren't expected to get any project work done. We also track the amount of time that they are spending on support tickets. If it is getting high we move more cleanup tasks into the regular project work cycle.

If you have oncall you can often co-locate these as well (oncall is priority 0).

The main risk is that if your support load gets high than there is no room for cleanup/automation which can become a feedback loop causing more support load. I haven't seen a good solution for this other than keeping track of the load and scheduling proper "project work" to address it when it gets too high.

The only sane approach. If someone is on support they also work to lessen the support load permanently. Recurring issues with no root cause analysis and fixes are deadly to a team.

Yes. Hiding errors leads to a lot of issues down the way.

This is part of the process. If an issue requires deeper work, it moves a proper task in a sprint.

We try to fix the easiest issues first so the deeper problems bubble up.

How the person in the support role knows that an issue is deeper? It comes from experience. Thus everyone does 1~2 week of support.

DevOps lead here; technically we have 2 week sprints, but we use kanban and basically add support tickets for each item that the team needs to work on during the sprint. This means that sometimes our "burndown" is more of a level line, but when that happens my manager understands, and we either determine the type of work that would reduce it or discuss adding more capacity. For tracking types of support tickets that come in, we try to determine which ones we can automate or heal completely so that those types of tickets dry up. Those items get roadmapped and worked down... eventually. It works OK, not perfect but I haven't found a better method so far, and everyone's happy with my team's work.

Isn't the classic answer to take urgent tickets and allow that to show up in decreased velocity?

I have never worked anywhere, including using scrum, where I could avoid incoming issues for 2 weeks at a time.

At the end of the day, better systems, better code, less technical debt means less support. Issues with previous work might indicate a false velocity of the past, releasing low quality.

There are long term maintenance exceptions that happen to all code bases, that have nothing to do with quality, like supporting new platforms or unpredictable platform changes, those can go in as backlog tasks.

The real question is what you are using "velocity" for? Is it to help get a feel for work you can complete, or is it a metric to be judged by management to evaluate you?

> Isn't the classic answer to take urgent tickets and allow that to show up in decreased velocity?

It should, but I've been surprised by some companies interpretation of "product velocity" to just mean "how many tickets does each developer complete per 'sprint'".

That's not product velocity, that's just some micromanagement nonsense so they can measure/push devs. No wonder many devs hate "agile" like this.

The "cover your ass" strategy involves tracking 3 things:

* Estimation time to complete each project task (hopefully estimated by the person who is going to do the job. Then add 20%)

* Status of each project task (from 0% to 100%)

* Time devoted to "support interruptions" each week. Doesn't need to be super detailed - something like "week 4: 5 man-days on support tickets x1, y4 and z6"

That way at the very least you are equipped to answer the inevitable questions "why isn't feature C done?". The answer would be something like "we still need around 2 more weeks of full-time work in order to complete that task, but given the current rate of interruptions we will realistically have it ready in one month".

You have discovered a couple of things:

Scrum doesn't survive "reality testing" in the context of actual s/w development projects.

Management (at least in your case) doesn't care about reality.

The usual solution is to participate in "software process theater" where you have the meetings and maintain the project plans for scrum, but actually use an alternative reality-driven management process to run your team.

Not perfect, but it's how software development has been done in at least 1/2 the places I've worked for the past 30 years. Before Scrum there were other crazy things, and there will be more crazy things to come. It's just the nature of humans + the endeavor of developing software.

Go through the motions of Scrum and just reduce the amount of stuff in the sprint to compensate for the lack of work. If they are looking for points consistency, fiddle with the estimates to get the right numbers.

Scrum is an accounting tool. Apply all the shenanigans that one might find in financial reports to it.

At my first job, we just edited estimates after the fact to hit the velocity target management wanted.

> management insists all engineering teams must use scrum

You could always point out that the Agile Manifesto specifically says "Individuals and interactions over processes and tools," and therefore you aren't doing real capital-A-Agile if teams aren't empowered to use the processes that work best for them. (It probably wouldn't do you any good, but it might feel good to say)

Why does management insist all engineering teams use scrum, though? I hope they're not trying to compare different teams' story points, because that would imply that management has a gross misunderstanding of how story points are supposed to work.

Most doctor's offices set aside x% of time for scheduled visits and y% of time for acute visits each day.

We do it similarly. We generally reserve 20% of our time for production support and 80% of time for new development. If we have fewer production issues that week, we get more new development done. If we have more production issues that week, get get less new development done.

We have one generic production issue story where we put small things and we'll create a fully fleshed out story for larger production issues.

If we have a trend of having too many production issues then we raise that as an issue itself and get to the bottom of what is causing it.

I'd strongly suggest you look at Kanban over sprint for your work, if you deal with a lot of emerging priorities. The value of sprint is the shorter-but-fixed work windows. If you need to consistently manage priorities that change work within this timeframe, sprint is a worse approach. You probably have a good idea of how much of your work (over time) is interrupt-driven, so if you need to do some project-oriented work you can split capacity across these streams and then run them differently.

By value. Don't overthink it, don't arbitrarily call them 'bugs', 'support', 'features' etc.

Just do the most valuable thing first. Ship. Repeat.

Just do kanban. It's made for this.

> management insists all engineering teams must use scrum.

They aren't doing scrum. At the very least, just "do scrum" externally and internally do kanban. Management is clearly dumb enough to believe it.

You triage which will get done and your velocity goes down if you have a lot of support tickets. This will make your velocity a meaningful metric over time, reflecting your rate of production toward new features.

Orrrr.....you assign points to the support work as well, your team hits their 50 points every week or else they get hit over the head by management as to why they weren't on track.

It sounds like OP is doing this. The point would be that even doing that, you're not making progress on team goals. (Your goals are instead being set by the interrupting tickets.) There might be 200 pts/wk or whatever, but it isn't going to be going into the project your PM is wanting it to.

A couple of times in my career, I've had the extreme pleasure of working as a dedicated support/backstop engineer within a small-ish (~20 person) development group buried deep inside a giant multinational corp. (I'm one of those seemingly rare weirdos who strongly prefers troubleshooting, bug-fixing, etc.)

My priorities were inverted: customer escalations first, and if you have time left over, then go work on bug backlogs that no one else wants to address. Everyone else sprinted while I bumped along at my own pace -- subject to escalation priorities, of course.

Those were dream jobs for me, and the rest of the teams seemed to appreciate having someone -- anyone, just not them! -- dedicated to the support role.

Is it just my imagination that this kind of (I would say "enlightened") management/organization is rare in the industry at large? Or do lots of dev teams do this sort of thing? And where can I find them? :-)

From an individual perspective, we've really valued Google's SRE whitepapers in interrupts and incident handling. The key message is that setting up your team to enable bundling 'jumping on things' and also 'focus delivery work' makes both people happier and more productive than asking two people to do both. https://sre.google/sre-book/dealing-with-interrupts/

From a PM perspective, that also lets management pre-allocate capacity for the sprint. '2 people on prod support, and 3 people on development/project work' is a decision that can be made and pointed back to. And being allocated to 'jumping on things' means you don't have to justify your productivity/velocity/ticket count beyond responsiveness and issue cycle time.

> management insists all engineering teams must use scrum.

DevOps/SRE are actually System Admins (always have been, always will be), and they are NOT engineers. They are operations. You cannot run Operations using scrum/agile.

First, it's right to expect a mix of new dev with support/maintenance tasks, and you're not alone. Second, if you're overwhelmed by support work, then I'd ask why?

We run scrum with 2 week sprint cadence and mix new dev with support/maintenance. The allocation differs based on season and events. Any support tickets we do don't count towards velocity and are mixed in with story point tasks and other maintenance/debt tasks. We track velocity as points delivered to a client, which helps with estimation for delivery of products / features to clients. This velocity is an average, and that's when the numbers work in your favor as "all estimates are wrong."

In a scenario where velocity is constantly decreased by support/maintenance/bugs, that tells a story, either of quality of work done previously, impatient management, or lack of discipline by the team. If you can't go a sprint without having to put out fires, that's indicative of a system that should be mitigated on a larger scale than continually applying bandaids, otherwise you'll constantly be bit.

In my mind, your choices are: change your cadence/strategy, change your values/philosophy, change management style, change your budget/velocity, invest heavily now to replace troubled systems, invest in man power to put out fires, or simply stay the course with a shift in perspective.

Only you can choose.

What's an example of a support ticket that might come in? Do you have a priority matrix for support tickets?

I find it helpful to rank support tasks by:

* Likelihood. Does this issue impact 10%, 25%, 50%, or 100% of all users?

* Pain. Where does this issue rank on a scale of 1 (minor nuisance) to 4 (product usage is impossible)

Adding categories about the "type" of work is also helpful. This lets you stay current on "crash" support issues while also giving you the ability to lesser-priority tasks for dedicated "localization sprints."

Check out excellent article for more examples: https://lostgarden.home.blog/2008/05/20/improving-bug-triage...

For some DevOps teams, I've found that "support ticket" is equivalent to "hold someone's hand while doing ____." This happens because DevOps people tend to be jack-of-all-trades and know their way around a wide variety of systems. If you're swamped by these kinds of tasks, start collecting metrics about the types of tasks coming in and build a self-service knowledge-base. Another common way to deal with this situation is dedicate 1 person to "triage" each sprint cycle. This person's workload is expected to be 100% support/routing.

The whole point of this labeling task, by the way, is to give your team the evidence to say "no" and "not right now" to incoming work.

If there are true emergencies / outages derailing your work, it sounds like the escalation path should be an on-call pager (not support tickets).

What works well will probably vary wildly between companies, due to the way that internal-tier support has to interface with the more customer-facing tiers.

That said, you have to be brutal during triage. Have a bucket of time set aside to handle support. That time is used for operational issues that are important but not more important than any product work you are doing. On a healthy system, most of the issues can usually wait to be fixed, due to either low severity or low number of customer impacted (or both).

For very high urgency issues, you drop everything. The sprint doesn't matter anymore, you are keeping the lights on. If your team is constantly dropping everything and blowing out sprints (or whatever duration measure you use), you need to look into why. Most likely, there is a quality problem somewhere that should be your top priority for sprint work. If you can't prioritize fixing that sort of thing, you need find someone who will listen, explain how much money is being lost spinning plates due to something that could be fixed at the root.

If you still can't get it prioritized, find another company. You'll burn out and quit at some point anyway ;)

You need to use styles of managing work that reflect the nature of the work and the responsibilities of the people doing the work.

"Sprints" work okay for product work. I use them a lot less rigidly than most people do with success. On-call activities should be recorded in a kanban-style project, these things are made for rapid reprioritization. Whoever works that board should be dedicated to only that board; making members of a team work in multiple project management paradigms at the same time usually has bad and confusing outcomes. Lastly, the "support" tickets can be varied. If they're requests that are gated by your team, I'd add them to the on-call kanban board. If they're just question/answer I wouldn't bother tracking them, you'll more or less end up recreating ITIL.

Some additional context would be to dedicate people to these boards for periods of time. Have a rotation that incorporates everyone doing a set amount of on-call on a recurring schedule. Build up a handoff procedure for the kanban board since context will be fresh every week due to the nature of the work.

Ideally go for kanban but if that is not possible don't exclude support tasks from sprint. Allocate some percentage of sprint for support tasks. What percentage would depend on actual rate of support tickets that you are getting week over week set some target for it and get buy in from management that if there is a significant variation in this mix metrics will be affected.

Fixating on metrics doesn't work. For managing humans, human touch is required. I am saying this despite creating a tool for managing metrics. My opinion is that metrics should be used as starting point to dig deeper into your process rather than using them as end goal. In this case if you are consistently getting higher than usual support tasks than it may be indicating some other problem like documentation not being clear in which case you need to improve documentation rather than manage the metrics or it could be a new product launch that would invariably lead to more support tickets until it gets stabilized.

I see a few people advocating for putting one person on support each sprint, rotating through the team.

This works fine for small teams but for larger (and with large, older codebases), the ramp-up time for being an effective support dev becomes a drag because it can be months between support stints. We don't all have eidetic memories and forget the tricks we use to diagnose production issues.

To counteract that we've recently put a developer on a 6 month rotation, and support them with a rotating backup. This allows the primary support developer to not only stay productive fixing issues but also surface intelligence around the problem areas in our app (ie: what's always needing support) and construct tools to make resolution easier or better yet convert to a self-serve.

I infer from your comment that you're a smaller team so perhaps you won't need to put someone on a long stint, but you might want to consider doing so anyway in order to have a resource pave the proverbial cow-paths and make it easier for the team in the future.

I've never seen the velocity metrics be not useless, so I'd sacrifice those. Or people need to realize that yes, a team that also does support will have widely varying progress towards project goals depending on support work load, that's not a problem. Who really cares if a team "overdelivers" because they got less interruptions than expected?

Split the team and have a few members dedicated to support tickets on a rotating basis. Or just abandon Scrum sprint planning cycles and run Kanban. You may have to escalate the issue with management. In most organizations if you go up high enough you'll eventually find someone willing to make an exception based on proper justification.

In our environment support trumps sprint every time, regardless of whether the support task is legitimate or not. By legitimate, I mean that many of the "support tasks" are things clients can do for themselves, but rather than point the client to the (excellent) documentation, the development team ends up carrying the baby.

I'm a dev on an MLOps team, we also do Scrum like you. We keep support tickets out of the sprint and also have a weekly rotation for support work. It's far from perfect, some weeks are just plain rotten and someone has to go off their ticket work to help. But it has worked well enough for us in the year since we introduced it.

That's how I've seen it done in a few places:

* The team does short sprints.

* One lucky soul handles all support requests for the duration of the sprint, having no officially planned work for the sprint.

The staff take it in turns to be the "support-person", and if they have a lot of tickets they work a lot. If they have few then they might sneakily do some work, or help colleagues with their own stories/tickets.

> * The team does short sprints.

How short is short?

Sorry! Two weeks.

More or less like us, yep.

I agree with sibling comments saying Scrum isn’t optimal for interrupt-dominated work like SRE. Kanban or Scrumban are good candidates.

Even for normal dev teams with customer-facing products (and therefore no zero support load) I have found a need to carve out some time for support work from the Scrum team velocity.

As others have suggested, having a dedicated engineer as primary support contact works well. You can figure out what the 95th %ile support burden is and remove that from the velocity. If you have a quiet week of support, the expectation is you spend the support budget on tools and docs, maybe improve automation on some existing scripts, knock out lower-priority incident remediation WIBNIs, etc.

If you have a >95%ile bad week, then the support engineer does less sprint work or the rest of the team pitches in as needed. But most of the time support doesn’t impact your velocity with this approach.

My advice: have a maturity standard for services. Ensure they meet a certain standard before they are handed off to the SRE team. Until the service meets maturity standards to qualify for full SRE support, the team that engineered the service is frontline support. If the service maturity regresses, primary support reverts.

Here is our yardstick: https://gitlab.com/dreamer-labs/tsc/service-maturity-model

This shares the burden of the interrupt work. It's still interrupt work, and it still slows down someone's velocity, but because it punishes the service designers for their poor design decisions, those get better over time and the overall quantity of bugs goes down, not up.

My advice is to first define the importance of support and how many tickets have to be solved within a specific time period in % (with your manager). Then define a part of the team that works on tickets for one sprint (alternating). This worked pretty well for me.

Example:

  Your team has 6 members
  Your management would like to put 15% of the workload for support
  Every sprint 1 alternating team member is doing just support tickets
  You keep track of the support days of every member to equalize the workload (support is no fun)
  Because working alone is not the best situation you could work with 2 members every two sprints, but keep in mind that this means NO support ticket is solved for 1 sprint

This is the best solution I could find being forced to practise SCRUM - even if it does not match the rules exactly.

TLDR: support should absolutely tank velocity, make the org feel your pain. If they want better velocity they need to invest in shoring up issues.

One thing to consider… support tanking your velocity is a feature, not a bug.

Sounds like you need investment in support tooling etc that can lessen that burden. Velocity will improve when support is less burdensome.

This is obviously not one-size-fits-all advice but worked well for a previous org I was at. If the sprint work will have a side effect of fixing support or support is so intense it requires a majority of the team you HAVE to fix that first regardless if sprint/scrum/kanban, etc. if velocity was super high for a few sprints but regressions/bugs were introduced they could impact future velocity, so those high velocity sprints weren’t as productive as you thought. It’s never just one number.

The org I worked at with the absolute worst support/sprint structure also had the only code base I considered “unsalvageable”. They refused to do anything to improve existing processes and spent half of every team’s time on the same support issues on an endless loop. They never had the measurements needed to actually figure out where to improve clients experiences.

Support Tickets are like fires. Typically, there are folk waiting around in case of fire to render aid.

Rotate who the "fireman" is on the team, they are out of Sprint and just fix. Spreads internal knowledge around too.

I've found it to be a square peg and a round hole. In every organization I've worked with. I'm always in a "support" team, which is more of a horizontal capability. Something akin to legal, accounting, etc. We exist to support vertical teams that can align with SAFe/scrum stuff.

We should only have scrum tickets tied to us that require sprint-specific work from a timing perspective. Otherwise, it's retroactive "tracking" points later on or preallocating points ahead of time for "support work" which is absolutely pointless.

Do you do any type of Root Cause Analysis tracking?

If that many tickets are coming in and disrupting everything, I’d be using that to prioritize addressing the causes so the disruptions stop.

If you’ve ever read The Phoenix Project, this is well illustrated when they talk about prioritizing preventative measures for “unplanned work” and they map out in detail why it’s such a huge problem.

Step one is to have Product/Sales define an SLO for support ticket resolution (and for all your key services, if you haven't already). Give more sprint time to whatever's falling behind.

If both are falling behind, you need to slow down the velocity of which you're demanding features be shipped, or loosen the SLOs, or hire more programmers.

We leave on person on "support duty" outside of the sprint. One of my friend working at Amazon in a team of 8 did the same with 2 persons out of the sprint (one being "on call" and dealing with support & operational issues, the other one working on "improvements" voted by the dev team, like better tooling, better monitoring...)

On my side, working in a small team of 5, we just leave one person out of the sprint to work on support requests, bug fixes and if time allow improvements. Large handling of technical debt (ex: transitioning from VueJS 2 to 3, good luck expressing this as a "product increment" in a sprint goal and good luck if you don't tackle it ... ) are part of sprint work.

Overtime, I realized that Scrum is a framework, not a fixed set of rules, and as per Agile, the goal is to maximize business value. So if we are swamped by support requests and our systems are not operating properly, it's the role of the person on "support duty" to warn the rest of the team and ask for help. Sure it may impact some velocity metrics and we may not hit our sprint goal that week, but why would that matter? There is no point writing more code and building more features if what we have is not working properly and not satisfying to users.

The main issue i have seen with this system is that the developer on support duty may not have the right skills to perform all the work that is needed. Scrum relies on the assumption that anyone can do anything, and that's definitely a flaw when working on a larger codebase with a mixed of frontend, backend and devops with more junior developers. It does force us to train everybody and learn to delegate so that everybody can be exposed to all parts of the system but this takes many months / years.

We also learnt to define smaller Scrum goals. It's better to achieve a small goal and then decide what to do next than to always feel like we are running behind. The sequence of "two weeks" sprints is often seen as a "fixed" schedule with deadline to hold at all costs, but that's stupid. The core idea of Agile is to follow an incremental approach following the path of max business value delivery, and periodically reflect about how things are going. If the amount of support requests and operational issues is such that there is no more resources available for new feature development, then it's time to prioritize tackling some technical debt and improve monitoring and automation to prevent the most common issues

We have designated support person which changes weekly. He is the first line of contact in our team and reacts to support tickets. Of course he can't resolve them all by himself but he delegates them further if help is needed. He doesn't even try to do any sprint tickets. This arrangement has so far worked quite well.

By finding a new job! Sprint is completely inappropriate for infrastructure teams. It's impossible to predict support work, and there's a ton of unpredictable engineering effort even for scheduled projects.

If it is implemented, then it'll be heavily gamed, or you have people working ridiculous hours.

Kanban makes more sense.

Usually situations like this involve support added to the workload of a dev team.

... Without adding additional staff of course.

Sometimes the"scrum" is imposed to try to order and manage things as the support burden adds more and more disruption. In a vain search for a magic bullet.

I've worked on a team which rotated the "daily support" tasks, which included on-call duty during those 7 days.

They were in effect excluded from the sprint during that week.

It was also a company who was trying to shoehorn everything into SCRUM because a consultant told them.

I'm in a similar boat. Furthermore, people tend to overlook that many new issues -> lots of routing effort (e.g. reading the ticket to see what team it best fits and then which person) which further cripples mid- to low-level management.

Velocity metrics are mostly useless.

We rotated support engineer among our team members. When they're on support duty, we just assign them half as many sprint points. If they have a light support week, they take on extra work. A nice bonus for our team.

This might sound crazy but if you’re being forced to use scrum, you could try daily sprints. Obviously, you can’t do every ritual every day. But you can prioritise and commit to a plan based on what matters most each day.

Like many others here, rather than answering how to make it work, I would say the premise is not valid - scrum isn't the right tool for devops. Probably something more akin to Kanban should be looked at.

To deal with this we have multiple devsupport devs that sit with support and handle urgent issues, anything larger/multi-day goes into the sprint queue/backlog to be triaged for the next sprint.

If you do 1-day sprints and merge the daily stand-up meeting with the demo and sprint-planning and retro you may get away with calling it Scrum and still have something that works

you may get something that works, but you don't know when

You could do a bi-weekly retro to catch issues not really appropriate for stand-up.

Point was that if commitments cant be made for weeks at a time just shorten the iteration time.

The core part of Scrum is to regularly collaborate as a team on assessing the situation and organise the next action to improve it. Can happen hourly if needed.

In my previous team we used a single kanban board for dev and support. That worked well, and did not require much of effort to communicate possible delays.

We also use single Kanban board for all current tasks. They get pulled fron support and backlog. Similar idea is described here: https://teamhood.com/kanban/kanban-for-software-development/

Just stop making sprint commitments. If your priorities change from day to day, what use is a list of priorities from two weeks ago?

I allocate 30% of the working time to support, 60% to development time, and 10% slack as in reserved for unexpected contingencies.

I've seen scrum failing to work too many times. It can work of course, but generally doesn't.

We don’t. If you do support (or ad hoc work in general), you‘re not doing Scrum. Use Kanban.

We have an oncall rotate in which whoever is oncall is not expected to work on anything else.

Do Kanban if you're a team that is relied on by other teams for support

Is it an option not to introduce support tickets into sprints?

why scrum for such a team?

just use a kanban-board with a backlog ...

just my 0.02€

Reply on news.ycombinator.com