Don’t Confuse a Bug’s Priority with Its Severity
bluelabellabs.com
bluelabellabs.com
> It’s important to note that a bug’s priority shouldn’t be confused with its severity, which exists as an entirely different dimension... ‘Severity’ to represent a (somewhat) objective assessment of a bug’s impact to the functionality of the system. From Critical representing major crashes or hangs, to Minor functional issues or purely Cosmetic blemishes. While a QA engineer will classify a bug’s Severity at creation time, it’s the PM who assigns the Priority at the time of triage based on their knowledge of the client’s business and product requirements
I don't see the benefit in having severity and priority as 2 distinct dimensions. Given a specific priority, what additional benefit do you get from having a severity label as well? Based on the author's description, I can't think of any.
If the goal is to better communicate what "type" of bug it is, using an enum classification (eg, cosmetic vs crash vs data-corruption) would be more appropriate than a numeric scale.
"if you click anywhere in the left half of the screen during the 5th screen all the settings get deleted" might be a very severe/critical bug, but if the client says "thats fine for me don't worry, but can you fix the wrong color on screen one where the logo is purple instead of fuschia that's not acceptable for my needs" then this one has a higher priority, even though it's much less severe.
Your job is not to build the perfect/best product, it's to build the one your client want and is willing to pay money for.
The mixup between priority and severity only comes from the modern era when so many product get made for internal consumption, or where the user is not not the buyer.
The problem here is the made-up definition of severity.. basically, you decided that crash or losing settings equals severe and cosmetic stuff equals not severe. Why?
I could easily give it a different definition. For example, a crash that nobody cares about is not severe. A small "cosmetic" slip up that can cause big damage to brand is very severe.
So I agree with the GP here; why do we need to have a definition of severity that does not align with the things that we care about when it comes to actually fixing things?
Because you, as a developer, are clueless regarding business needs, and thus are unaware of why "cosmetic stuff" might be far more important than the risk of deleting someone's user profile. For example, perhaps the color scheme clashes with the one used by your client's main competitor and therefore might leave him vulnerable to lawsuits.
Then that's the problem we should fix. Instead of creating an extra database field to capture somebody's incorrect opinion, the people who understand priority should be helping developers know enough to have useful opinions.
The priority database field is how you communicate this factor, but there is a point where reasonable people can disagree and the organization needs a way to make decisions clear.
To draw out the example even more: you could have a $600K invoice riding on customer acceptance, when the person at the customer site who signs off won't do so until the customer's logo color is correct. While that crasher when you enter a non-numeric character in a numeric field of a new feature? "We accept the functionality meets the milestone so will sign off but we don't plan to roll it out until next quarter after we have begun training personnel on the new feature."
Sure, every good organization should want everybody, not just developers, to understand the customer's business and such but sometimes you just need to get it shippable.
Why? What is the business value of having severity as essentially a protest field logged by people who don't understand business impact? You are basically in all your points explaining the business value of priority which nobody ever disagreed with and then going "and that's why we need two fields".
I don't understand where you could possibly get the "protest field" idea. Severity is an objective statement regarding the known impact of a bug as verified by a developer. It summarizes the technical impact of a software defect. Stating that bug X is high-severity because it crashes is not a protest, and just because the PM decides to give priority to other more pressing issues it doesn't mean you should throw a tantrum.
Crash bugs aren't bad because crashes are inherently bad, they're bad because they have negative user impact - if the program crashes and loses user context, or data, or takes time to restart... those are bad things. If it crashes a little untidily when it receives a shutdown event from the operating system... maybe not so much.
Same goes for performance issues, scalability problems, security flaws, even badly structured code - they don't have technical impact unconnected to their user (or business, at least) impact.
TFA provides a concrete definition and also the method to classify bugs based on severity.
Severity does not divorce a bug from "the user impact". There is, however, also the problem of low-severity bugs or even tasks having low user impact but high business impact.
But that's a contradiction. Unless the users aren't important (and the business is another entity, e.g, a CxO that has clout and demand a fix for a thing that users don't care about).
I do get that people have all sorts of adaptations to dysfunctional working conditions. So if a severity field is one of them, fine. But I don't want people to mistake that for healthy collaboration.
Are they? That's how majority (all?) systems that are asynchronous work. The data to be communicated has to be persisted. I think asynchronous is a good communication method.
There's nothing to fix. Developers assess severity but the project manager defines the priority. As the PM calls the shots, they pick which issue should be addressed first, and obviously it's expected that some low-severity issues should be given priority over some high-severity issues.
In fact, the only thing that needs fixing is any potential misunderstanding on behalf is some developers on how low-severity issues shall have priority over high-severity issues.
It seems that a recurrent theme in this thread is that some developers have a hard time understanding that PMs might have solid reasons to prioritize low-severity issues over high-severity issues. It's like these developers are stuck with a mindset where the forest doesn't exist and only the trees they personally have direct contact with should.be worth any consideration.
Your question misses the point entirely.
The point is that severity is an element that's used to classify an issue with regards to priority. Severity does not exist in a vacuum, and priority is given depending on context. Severity is an attribute, other attributes, that's used to determine the priority.
That's the point, everyone does not understand the business context. Nor are they expected to. That's the job of the PM it's his main responsibility, and it's the reason PMs are hired to manage teams of developers.
The point of developers is to make software that people use. So if we want to do our jobs well, we have to understand how we are creating value. Product managers may manage that information flow, and they may get the final say in decisions. But if they are specifying software in enough detail that developers can be 100% ignorant, then developers can (and should!) be automated out of that work.
The point is that severity is not ignored; it does inform the priority. Most of the time there may even be a direct correlation between severity and priority.
But other (real-world business) factors also inform the priority; while severity will never change in the absence of new information about the bug, those other factors may change frequently. It doesn't make sense for a PM to reread every single ticket and reassess each one's severity when adjusting priorities, when the developer can just determine that once and record it in the ticket from the start.
e.g. A bug might be critical severity if it wipes the entire production database, but low complexity if the fix is to delete one line of code. And maybe its priority is P1 instead of P0 because the customer said they'll remember how to avoid triggering the behavior but they really need that logo color changed asap for an important demo.
So what's the point of severity?
The point of severity is that it's an objective metric determined in isolation based on specific technical guidelines that any developer on the team can follow (such as https://www.chromium.org/developers/severity-guidelines). Whereas priority is a higher-level determination that isn't purely technical.
It's like the difference between body fat percentage and attractiveness. Any "engineer" (doctor with a DEXA scanner) can tell you your BF%, and attractiveness will typically correlate with BF%, but ultimately your "customers" (romantic partners and targets) decide how attractive you are. Not a perfect analogy (priority is still something you'd decide internally, not your customers directly), but hope that clarifies things.
(...and if you're indeed in a position where you're responsible for hiring decisions or performance reviews: strike "maybe" from the preceding sentence.)
For us, it is absolutely 100% necessary to hire domain experts to prioritize bugs and features. It's not a question of incompetent or dense developers, it's a question of things that are not obvious to someone who doesn't have tons of experience in the field.
It's a problem that I imagine developers working on Chrome, Call of Duty, iTunes, or Outlook don't have. You can hire recent college grads and expect them to understand what the software does, have reasonably good instincts how to prioritize bugs and put together the right user experience even if the description in the feature request is sparse on details.
By the way, I heartily recommend working for such a company. My company works very, very hard to retain people. Someone who's spent ten years getting used to the weird stuff our customers expect is far more valuable than someone with half the pay who needs someone to hold their hand through every single issue. Everyone has their own office, management is extremely permissive about the shit that doesn't matter, there's never deadlines or crunch time, everyone chooses their own work/life balance. (We're hourly, and the expectation is that you work more than forty hours a week and need manager approval if you want to work more than 60 for more than three pay periods in a row) If we want more vacation, we can bank hours and spend it on supplemental vacation. Everything's great.
Eg "the color is wrong because we specified it wrong" and "the color is wrong because the app doesn't respect what we ask it to display" both ends up with the same bug (wrong color), same priority, but not the same severity because the second case is uncontrolled.
Severity is a dev tool, priority is a business tool.
1. We have the wrong .png asset in the database
2. Our entire rendering infrastructure is suspect
I'm not saying this is the wrong approach by the way, it's just interesting how people approach this differently.
Of course it does add complexity. It is the call of each organization which things are important enough to be worth the extra complexity and which are not. Maybe for yours it isn't worth the extra cost - there is nothing wrong with that - but other people have different needs and so they will have different answers.
In short, you are wrong in thinking there is a universal right answer.
If a base release that was customised for multiple customers has an expected number of P1, P2, P3, P4 issues for most customers but a high number of P1 and P2 issues for one particular customer, off of the same number of issues in the base release as measured by severity, then that will stand out in our measurements and we'll dive deeper into that customisation to see what's going on.
(Edited mid-stream... accidentally submitted too early.)
The point of this system is that high-severity bugs might have lower priority than low-severity bugs if you take business requirements into consideration. Yet, this does not mean that severity should be ignored.
For example in the teams I lead I make sure developers participate on PO + stakeholder meetings as observers.
This way when devs fix something or develop a new feature they know first-hand what the business expects.
A nice bonus is that the team often gets personal praises from our clients.
A high priority bug that isn't fixed presumably has severe consequences for business (or it was prioritized wrong).
A high severity bug that isn't prioritized presumably has low consequences (severity) for business, but it probably really sucks for the unfortunate user who is affected.
Here in pivotal-tracker-land the product manager controls priority by ordering stories in the backlog. There is no need for a separate 'priority' field because that is implicit.
FWIW, there's also no explicit 'severity' field; the PM is expected to understand all the factors that make issues important and order the backlog appropriately. If you need more categorization, you can apply arbitrary labels.
If the color is indeed the one set by the dev/designer/... and it's juste not the right one, that's a controlled thing, it's not severe. The app is doing what you want it to, you just gave it the wrong instructions.
The crash on the other hand is uncontrolled. That's severe, because it's a case where the app is not doing what you want it to.
Severity is something for the dev team internally, priority is something for the product manager and business people that the dev team follows.
When you start mixing the two in a single thing, the dev specific needs are always what goes out the window first.
Which do you think the CEO cares about more, the logo or the fact that it crashes sometimes on an infrequently accessed menu?
That's a bad example, something not severe is not severe... can't argue with that. A crash nobody cares about can be quite severe though, there's quite a bit of crash that has been used to make some of the most important security vulnerabilities. When I tried to reproduce the Dirty CoW vulnerability over a VM in a school setting made the teacher VM crash multiple time, he didn't care about the crash either ;), yet that single vulnerability allowed me to skip 90% of the vulnerabilities he wanted us to try to find.
I think your comment point something important that you miss too, different client may have different priority, and you too may. I think severity is closer to YOUR own priority. The severity is the impact this MAY have on your business if it wasn't fixed. Like you said, a small "cosmetic" slip up may be important for our brand, thus severe for us. That "cosmetic" slip up could still not be important for all of our client, like over a one page made to sell, yet still be on top in both priority and severity, because it matters for the next client or the sale team, which does matter to you.
> So I agree with the GP here; why do we need to have a definition of severity that does not align with the things that we care about when it comes to actually fixing things?
You could say that about many information in a ticket. It is still informative though, maybe it's no longer needed, but at one point it did was relevant and in many case, still is.
A ticket with an higher severity is something that you need to put more time on to decide whether it deserve an higher priority. I'll go back to your crash example, why would you works on a crash that nobody care about, why would you see "crash" in the ticket with so many steps to reproduce and believe that this deserve an high priority when it never going to happen? In an ideal world sure you would know 100% of the system and know 100% of the tickets and can understands 100% of the impact of this kind of crash and be able to fit it well in the priority, but let be honest, that ideal world doesn't exist. You have a limited time that you can put on each ticket to decide their priority, you have a limited capacity to understands the impact and everything, you have hundreds of tickets to go through, but something that you can do is see that the high severity set by the developer means that you may need to put more time on that ticket to choose its right priority.
Sure once the priority has been set, does the severity matter? That's a much bigger question that I won't try to answer, probably not, but did it matter at one point? It did. Would you remove any information from a ticket because it no longer matter? I sure hope not.
So why have severity and priority as two distinct dimensions then? If customer impact is the only relevant metric for your process, why would a bug about which the client says "that's fine for me" ever be considered severe or critical?
If the only metric you care about is customer impact, why track two of them?
Maybe the page with the bad bug is getting shut down next month anyway and we’re just accelerating that instead. Doesn’t make the bug any less bad: definitely makes fixing it less of a priority.
(In hindsight I should have used SSRF or SQLi or RCE as examples of the former style bug and XSS as the lower severity of the two to emphasize the difference.)
If you find a RCE vulnerability that never affects your customer, and how much it affects your customer is the only metric by which bugs are prioritized, what relevant information does severity add?
Either it affects your customer, in which case you prioritize a bug's fix by its severity (i.e. simply by how much it affects your customer), or it doesn't affect your customer, in which case you prioritize is by... how much it affects your customer. Either way, you end up prioritizing bugs just by customer impact, which is either measured by the bug's severity (if it affects a customer), or zero (if it doesn't, regardless of severity).
Just to be clear, I am absolutely not arguing that severity shouldn't be tracked, I'm arguing against the idea that purely functional customer requirements should be the only criteria used for prioritizing bugfixes and development tasks.
It's certainly true that "your job is not to build the perfect/best product, it's to build the one your client want and is willing to pay money for", as the parent mentioned. But it's also your responsibility to do the building in a sustainable manner.
Decoupling severity from priority is a great way to run a codebase into the ground. RCEs that are unexploitable today will be expoitable five years from now, after sufficient requirement changes accrue -- and at some point in the future, you're going to have to deal with five years' worth of previously unexploitable RCEs that you have to fix yesterday.
Edit: I've worked for several years on two codebases where customer impact has been the primary metric used for bug prioritization for 15+ years and 7+ years respectively. Both were absolute catastrophes. You could literally crash devices by blowing hot air on them (literally, there was an out-of-bounds access bug in the code that handled overheating alerts and they crashed after a few alerts). An entire team was there just to put out fires, virtually all of which were known bugs that had at one point been deferred because they had "no customer impact" -- until they did.
It's been a very useful experience. Disentangling these things is a very useful skill to have for an expensive, and highly sought-after on the market.
So, to rephrase: you need both precisely because that's often the way you can even start having the prioritization conversation.
> Just to be clear, I am absolutely not arguing that severity shouldn't be tracked, I'm arguing against the idea that purely functional customer requirements should be the only criteria used for prioritizing bugfixes and development tasks.
One of them sometimes suffers a crash that also loses all their work, and it really pisses them off. That's got to be bad, right? I'd like to call it severe based on its impact to that user.
But then the vast majority of your users have never seen that bug, and the bug is damn difficult to track down, so maybe you can't treat it like a stopper.
But that says nothing about why one would want 2 unrelated priority dimensions.
« Given a specific priority, what additional benefit do you get from having an additional severity label on the bug? »
not
« Given a specific severity, what additional benefit do you get from having an additional priority label on the bug? »
Is the information about the criticality of the bug (whether or not the app crashes) relevant to the business decision?
Yes, of course: "When I press the 'more' button in the developer's bio the app crashes" is a more important bug than "When I press the 'more' button in the developer's bio I don't see more text."
Might the criticality of the bug not be relevant to the business decision? Again, yes: "When I press the 'more' button in the developer's bio -- ok, stop, I don't care about this bug."
In some cases the criticality is relevant to the business decisions, in others it may not be. It probably usually will be. The app shouldn't crash, because even if it only happens to 1% of people, they're the ones writing reviews on the app store.
So the criticality is a big red "Hey! This should probably be high-priority!"-flag, but it's not the only piece of information the PM is going to use.
As with all information gathering, you need to weigh the cost of gathering and storing vs the gains of having the information.
People are saying there are no gains from having this information; which, if true means if there's any cost, the cost is too high.
People are saying it is hard to determine, which means the cost to gather it is high, it may include concensus building.
In my personal professional experience, there hasn't been a whole lot of retrospective analysis on the bugtracker data. So information gathering that doesn't immediately help with fixing the bugs, or communicating which bugs need to be fixed sooner is time wasted. Use of non-critical fields in the tracker is spotty and inconsistent, so analysis would require an expensive data cleaning phase.
Sure, you can say I didn't work in healthy organizations if they didn't do this sort of analysis. But that's the root cause of most of the bugs --- there were never any formal specifications and so many things didn't meet the specs; plus occasionally we wrote crashy code.
What people are pointing out is that you're not arguing for having more information. The "severity" is a marginal zero information on top of the priority. There is no context where you would want to use "severity" instead of priority.
So...
GIVEN that priority trumps "severity" 100% of the time,
AND GIVEN that priority is already being recorded,
THEREFORE "severity" has no use, and shouldn't be recorded.
What's happening in the post is that someone wondered why there were two fields, and made up a justification for the second field instead of realizing it was a mistake to have two.
You can see the mistake happening whenever people choose a word that ordinarily refers to the concept of importance as their label for "severity", while simultaneously acknowledging that "severity" doesn't affect the importance of the issue.
> In some cases the criticality is relevant to the business decisions, in others it may not be. It probably usually will be. The app shouldn't crash, because even if it only happens to 1% of people, they're the ones writing reviews on the app store.
> So the criticality is a big red "Hey! This should probably be high-priority!"-flag, but it's not the only piece of information the PM is going to use.
The point is that PM may be inundated with lots of bugs and new features and other stories that all need prioritization before the next release. If they had a perfect understanding of all of them, they wouldn't need any metadata at all (not even "bug" vs "feature" -- just read the long-form description!), but metadata helps people make decisions.
High-severity bugs should generally be prioritized higher, except in cases where PMs have gained a deep understanding of the situation and realize it's not business-critical, and override that.
At the point that they set the priority, the criticality is no longer important, just as the "bug" vs "feature" tag is no longer that important once it's been prioritized and assigned.
I think the argument is that it's not actually more information; it's just smearing out the same information in a way that makes communication and decision making harder. (Note I'm not completely convinced by either side here, just trying to explain.)
Of course? Really?
I mean, the upshot for the user is the same - they wanted to see more info about the developer, they clicked the button, they didn't see more info.
An app crash on most devices isn't actually a particularly dangerous problem. Start the app up again.
Arbitrarily deciding that a particular bug is worse because it causes more harm to the software rather than because it causes more harm to the user is precisely the problem. The importance of fixing those two bugs is probably exactly the same (unless the app crashing causes data loss or corruption or is exploitable or... you get it. Of course, the same could be true of the 'no more info appears' bug - behind the scenes it might be starting an infinitely retrying string of HTTP requests to download the developer bio that gradually leak memory and drain the user's battery. Maybe that's really the more serious bug?)
There isn't some separate measure of 'severity' that independently makes the crash bug magically worse than the functional failure. There's only the actual consequences of the bug. For the user, their data, their time their resources, and your business goals.
If you tell the client that they're wrong and that you should fix the critical bug first how they react is a really good test of whether or not they're going to be a good client. If they insist on fixing the lower priority bug first then you should try to stop working with them, because at some point in the future they're going to demand more unreasonable and stupid things that blow up in their face, and you'll lose that client anyway. Save yourself the stress of working with bad clients and get rid of them at the earliest possible opportunity.
If they accept that you know what you're talking about and give you agency to fix things in the order that you think they should be fixed, do your absolute best to keep that client happy because they'll be a joy to collaborate with for years.
I'm all for firing bad clients. Fire the crazy ones. Fire the ones that don't pay. Fire the ones who make ridiculous demands at the last moment.
But the client wants you to deploy a fix to the logo ASAP, you do it. That's the client's priority. As long as they accept the disruption to the rest of the work, it's no problem.
When your product is used by many clients you can’t rely on what one client says
A developer or tester might be good at specifying how severe a bug is: impact (does everyone get it, or just some users) multiplied by frequency (does it happen in commonly used functionality?) multiplied by a category factor (is it a crash, cosmetic, dataloss...) will give you the severity.
Some completely different person can come along and give this a high priority because of business reasons, and to prioritize fixing the bug compared to other functionality. Two bugs with the same assigned severity might impact two different markets where the business knows one is critical for the survival of the business, and one is already a safe market. Then it's a priority to fix it.
A second reason might be due to ROI (easier severe bugs are better to prioritize than difficult bugs of the same severity).
Ultimately you could have a low-severity bug, but it impacts a really important customer who cares about it, so it's high priority. This is the most common case I've seen.
Severity also can impact whether something gets prioritized at all. If it's low-severity and the business side isn't there either, maybe it just doesn't get done. Whereas something with high severity but low business impact still probably needs to be done regardless or it will lead to bigger problems.
I've never seen impact/urgency/severity as being anything other than different ways to say priority or overcomplicated ways to derive priority. Then again I've never wrangled an absolutely massive bug/ticket queue.
Working directly with PO's both "Technical POs" and "Product POs" I've found that creating a strong distinction between the two as well as what the two dimensions define has a significant impact on the quality of the product over time.
In the original post we see that both Priority and Severity are two different dimensions each representing a scale. This is what I think is causing a lot of the discussion in this thread.
Because both of these values are a scale we are creating a foundation over which we can get confused and we find ourselves needing to differentiate these two in some way. This comment is a good example of the confused conversations I would often see. All of variations on the same flavour:
> If the goal is to better communicate what "type" of bug it is, using an enum classification (eg, cosmetic vs crash vs data-corruption) would be more appropriate than a numeric scale.
To get around this in Agile contracts I've always advocated the following:
"Severity" scale. Severity of the issue on the user from 1-5. (For Jira users imagine this as your 'Priority' field renamed to severity.
1: The user is unable to complete the journey or part of the journey
2: The user is unable to perform an action, but can complete the journey
3: The user is able to perform an action but...
4: The user is able to perform an action but...
5: The user is able to perform an action but...
Of course, you can put in your own definitions. The important thing being that this is defined from the 'users' point of view rather than attempting to define the nature of the issue (crash, freeze, etc...). The nature of the issue can be discussed within the ticket itself.
The second dimension 'Priority' instead being thought of as 'Ranking' and is not a field which is added to the ticket. This defines the order in which the issues must be dealt with. Thus allowing a PO to appreciate the severity of an issue and also negotiate with the team it's relative important through ranking.
Basically, if what you are trying to do is too complex/nuanced for everybody to understand - don't do it.
I could see where a severity dimension could be useful retrospectively if paired with classification. For example, you could look back and say, why did we have a big spike in severity 1 security bugs in Q4? However, I think I'd rather not have the dimension available than be forced to set it on every single bug on the off chance it will help with analysis later. "I bet we can do cool stuff with this someday" is never a good reason to add process overhead. If somebody retrospectively categorizes and summarizes tickets by some dimension and proves that it yields useful information, that could be grounds for asking engineers to track it in the future.
But maybe this bug occurs very rarely and in very specific circumstances.
On the other hand you have also found another bug that makes your app use bright green Comic Sans. It is not severe as it is only cosmetic.
But for some reason, on your last release this happens every time the app is started.
It seems likely that the second issue will have a much higher priority and will have to be fixed immediately, while the null pointer can wait.
Severity is purely technical. Priority is a project/commercial decision (which is informed by severity but also other considerations). They generally do not align.
We have applied the same metric to feature request prioritization. Severity can be thought of as necessity for an individual customer (must have, should have, nice to have, could have, won’t have) and frequency represents an estimation of demand in the market.
We even created our own Jira plugin to calculate the product of these and use that number as a self prioritizing index.
Low severity and high priority can be:
- it breaks the software for just one person, but that person is on the critical path.
- if breaks users workflow. For example a keyboard shortcut that is used every second stops working.
- the bug is an easy, high value fix, so why wait?
A bug might be Major or Minor but there is a high risk to fixing it. You might want to do it early in the development process, ahead of higher severity issues, so you get the most runtime on the fix before the next deployment.
Buying clothes detergent because you ran out and have no clean clothes is high priority, low severity. Booking cancer screening sometime this year because you're reaching that age and there's a history of cancer in your family is low priority, high severity.
Of course, scheduling is often dominated by other factors, but it gives you a default at least.
Lets say you have 10 bugs and they are prioritized in order from 1 to 10. Bugs with the priority 2, 3, and 6 have been given a severity of P0 (a ship-stopper, as the original post described it). Bugs 1, 4, 5, and 7-10 are not considered P0 bugs.
Given those priorities and their relative severity, that means the team is committing to deliver, at least, bugs 1-6 prior to the next release. That makes severity a pretty useful dimension to consider on its own and not just as consideration when determining priority.
"Priority" feels to me like a band-aid on a UI deficiency in many electronic issue tracking systems: No way to manually sort the bug queue. If I can manually sort, it's easy for me to quickly find a spot where it's directly between something I'd rather have fixed sooner, and something I'd rather have fixed later. Which is ultimately what prioritizing really is - anything else is just beating around the bush.
And I don't believe that "severity", conceptualized as a linear scale, is meaningful. Whenever I see "severity X", my very next question is "why?" The severity score itself is, at best, not particularly actionable, and, at worst, something other stakeholders will learn to use as a vehicle for squeaky wheel prioritization. Or an agent for semantic rot, as someone decides to try and make the severity scale meaningful by imposing a strict hierarchy on all the possible kinds of things that might go wrong. (Or, rather, all the ones they thought of at the time.) I'd rather see tags like "data loss" or "known workaround exists". They paint a clearer picture, and they're harder to game.
- Risk of too much discussion on exact ordering. I think to my uses of graphical programming languages vs text and having infinite places to put wires made me waste more time on perfecting it than the coarseness of text.
- Sometimes there is not clear priority between multiple tasks
- Sometimes you want to think locally. I might be looking at a ticket and I'd like to know the general bucket it is in without having to look at it in context of the entire table. I might also want to triage it and put it in a bucket without trying worrying about the exact order.
I think a potential is to combine the two. You define discrete priorities and have dividing lines in the UX. The UX can let you drag between buckets or even order within a bucket.
When I see over-polishing happening in the wild, it tends to be in response to "too many cooks in the kitchen" type situations. Well, that or procrastination.
My main issue with allowing people to define the ordering is that, in my experience, enevitably you have some stuff at the top, some stuff in the middle and a big mostly unordered heap in the middle.
- "severity" as level of importance the tester assigned it when they found the bug and - "priority" as level of importance a developer assigned to it after triage.
That first piece of information is important when you need to determine which bugs to look at first for triage, but how important is it after that point? Can it not just be replaced after the developer's judgement?
In other words, could you not have a single priority field? The tester uses a heuristic to assign an initial priority (e.g., crashes are P0, cosmetic are P4). The dev uses this to prioritize which bugs to triage first, and once they've determined a new priority based on customer experience combined with app behaviour, they replace the old one.
If you really need to go back and check what the tester assigned, then I assume you can just use the "history" or "revision" feature in your bug tracking app.
Additionally, as suggested in a different comment, you can add a label for the bug's type if you feel that's important (crashing, lagging, cosmetic, etc.).
Perhaps the message here is that the app's behaviour in a vacuum is not the sole determinant of its priority. But then that should be the message, rather than claiming there is another metric which needs to be separately tracked when evaluating bugs.
But I think it should be similar to risk calculation, multiply by the "impact" or "damage". A typo might offend a small number of people but have a big publicity impact
If you can't at least do a rough triage of all the new bugs in one sitting, you're either not allocating enough time for triage, or letting yourself get way too bogged down with ancillary conversation during the triage meeting, or it's just time to scrap the product and go home.
If you can stay on top of your triage, then there's not really any need to worry about what order you do them in.
That's why the split between priority and severity exists: The tester is telling you the subjective view of the bug as if a user encountered it; and the triage process is all about judging the priority of fixing the bug in this particular release cycle.
> Effective bug triage is an essential hygiene and success factor for anyone managing a software release.
I could not agree more. To some extend, it applies to all change requests. That said, there is no need to be too specific in your change request specification.
Where I work, we have setup only a few parameters to help us triaging several hundreds of CR:
- Priority: Low, Medium, High
- Type: Add (new feature), Enhancement (small change on an existing feature) and bug
- env: Production, Acceptance, Development, ...
In 99% of the case, this is completely sufficient. As said by others, adding severity does not add useful information on a bug triage (it may provide information on the bug itself). If there is a data loss involved but the client is fine with it, that would never lead to a high priority bug. If the client care about its data, then be sure the priority will be high. It means that bug triage really is priority assignment (no matter the severity).
In our workflow, only high priority bugs (usually hitting the production) are subject to hot-fixes. All other tickets go into carefully crafted milestones (a package to deliver). As all CRs belonging to the same milestone are delivered together, they actually all share the same priority which becomes the priority of the milestone compared to other milestones.
(edited for formatting)
In my experience P3's capture tech debt, cut features, and design flaws. Those should be addressed for what they are, not shoehorned into a bug prioritization system that will never even glance at them.
Something that is Critical severity doesn’t necessarily mean its a P0. That crashing bug in your bio screen? Face it, nobody really cares to read it and few will ever click the button, so its not something that should be a P0 ship-stopper.
In what way is that an issue of 'critical severity'? The quoted text contains an explanation of why it is not.
The real difference is that severity is a measure of the harm done, while priority is a plan for to responding to it. Clearly they are different, but clearly they will likely be correlated, at least when the severity is high.
The only reason would be this: given a large enough enterprise you have one person categorizing a bug as a crash and another making the call on whether that crash is actually worth prioritizing, a decision made on another day at another pay grade in another time zone. In this case it might be clear, but in the next case it might be "the configuration page crashes if the region is France". I as a developer don't know and I don't care that we haven't even launched the product in France so I just want to mark the issue as a very severe crash. Somewhere above my head the can call this low prio because the product launch in France is still a year out. This is enterprise backlog massage for enterprise workflows.
In your scenario, you don't know whether it is severe or not, so you just want to provisionally mark it as severe, and have someone with the appropriate domain knowledge make the final call. Until that happens, decisions made on the basis of that provisional call could be bad decisions.
And what might those decisions be? Priority, for one! To say that the severity should stay at 'severe', and the adjustment should be made only in the priority, is to confuse the two distinct concepts. I see a lot of that going on wherever data are used.
If you want to characterize bugs by the sort of technical outcome they produce, then do just that, instead of trying to bend 'severity' to try and encompass it, while actually hiding the issue that you wanted to characterize on. Again, while technical outcome is an aspect of severity, they are not the same thing (and, FWIW, crashing is rarely the worst thing that could happen (unless the domain is vehicle control.))
However I have little to no knowledge of the business (markets, release schedule). I’m not even aware that we have zero users in this region. A manager I have never met assigns a low priority, two months later.
Domain and business overlaps, but for example if one doesn’t mark this “severe” because the business hasn’t launched for affected customers, then a year later when it is launched in France you have a low severity bug that should be high.
The priority can be constantly re-evaluated against business needs while the severity is fixed.
Yes this is a simple technical characterization. It’s not useless to separate the business-stopping bugs from the cosmetic. And importantly: It’s not my call to say what is important.
Someone might say that the font size is more important than a deadlock. I don’t care and I don’t want to care. But I’m rather sure that those that do make those decisions want to be able to sort the bugs I wrote by a severity.
The crash has no severity to you. It is either severe or not to the client. If you don't know whether it is severe to them, sure you can guess that a crash in france is severe, but that's an incorrect guess, not an independent dimension of severity.
There is no "techical measure of severity", everything is goal oriented. It's not severe unless it's important to the client, or is impeding something that is important to the client.
Do you call a bug that corrupts data more or less severe than a bug that crashes a program? In whichever order you put them, is the severity always necessarily in that order?
>I’m rather sure that those that do make those decisions want to be able to sort the bugs I wrote by a severity.
Why would they do that, if, as you say, "It’s not my call to say what is important"? In what way is this not you signalling that you suspect, provisionally, this has a severe impact, as I suggested in my previous post? I think you are being a bit inconsistent here.
In other words, by adding a severity dimension, you lost your ability to order your bugs in any ascending or descending order. That defeats the whole point of having a priority field in the bug in the first place. Now you need to have a person decide if your [P1 S2] bug should be solved before your [P2 S1] bug by some arbitrary mechanism. And once you've made that decision, you'll communicate it with "Do this first" - which is essentially a priority field that now exists outside your system of [P S] classification.
Get rid of severity - or treat it like an enum that will help inform you what your priority of each bug is and bring sanity to your system.
That said, the P-spectrum may have some crossover, especially at P1-P2, depending on the release cycle. For example, crashes should be focussed on a little more for always-online software, whereas for more glacial, manually-updated software, prioritising incorrect behaviour or functional issues may be more important.
If there are people on the team who insist on objectively classifying bugs with no reference to customer impact, you don’t need to indulge them by giving them a meaningless drop down to play with in Jira, you need to get them up to speed with understanding your product and release priorities.
> crashing bug in your bio screen
The examples makes it clear that they use "severity" to map types of bug to severity level in a completely opaque way. Regardless of the impact of the bug, "crash" -> "high" and "typo" -> "low". A crash is high-severity even if it has low-impact in every category ("functionality of the system" included), but a typo is low-severity even if it is high-impact in an important category? Why?
> a misplaced comma might seem like nothing worthy of emergent attention, but what if it completely re-purposes your brand statement
This just sounds like a high-severity bug: it may be low "functionality of the system" severity, but it is high "brand identity/loyalty" severity.
I think it becomes obvious that there is not much benefit of a special field for "functionality of the system" severity or even the underlying bug "type" that it seems to be representing. All severity is used here as a triage-heuristic anyways. Great. Now you can freely forget about it and find a better triage-heuristic for your QA team so that they can assign the priority more accurately.
Anyway, the developer got this stop ship the day before Thanksgiving, so he worked all weekend and Monday morning turned in a fix. At which time the factory manager told him "thanks, but one of the parts to make this machine is on back order for 4 months so we weren't planing to make that product anyway".
The problem is that "Severity" is being used for something that doesn't really match its intuitive meaning.
Maybe call the field "kind of impact" (or something similar) and have enumerated values or tags like "crash", "hang", "typo", etc. Now the field is intuitive and you don't need to retrain (or write blog posts) to get people to understand your new meaning for the word "severity".
Priority is how the developer (or, more likely, the project management team) sees the bug. Highest priority bugs are the ones in the project manager's spreadsheet and ended up there because of the most vocal complainers (paying customers, an inconvenienced CEO, etc). The rest will never get fixed.
It's impractical to use the severity of the bug to rank the priority, because they're not strongly correlated.
I think they are highly correlated, but they occasionally diverge.
I think there are two separate things about tickets: one is the about technical merit of a ticket: documentation, evidence, reproducer, quick info about context. Basically, a ticket as a tool to gather and sort "technical discussion". This is fine and encouraged.
Then there is the "priority" - I don't think is a feature of a ticket. It's a property of context - do users complain _now_? Is the fix risky? Is the boss shouting? Do we lose money? Does the PM want this work done for personal reasons? How much time will it take for me to do a context switch to start the work?
I don't think it's possible to add this "priority" (as defined by social interactions) to a ticket. I always found I prioritize tickets that I understand the social context of. A colleague nagging me by walking to my desk is 100X the priority for me than "P1 critical" written by some PM on the other side of the planet.
The point: I try not to fight with the social aspect. If you want me to do a ticket. Good. Come over. Do a hangout and explain the context. Setting a mindless value in some field on some random badly described ticket is not going to make me de-prioritize other things.
Tags/label have been working quite well for me. Priority can be replaced with milestones, but you have to create them for each releases to ensure P0 are quickly enough.
Where you see companies get into trouble is when an unlikely high severity bug actually happens (or becomes likely) and it comes out that they've known about it for a while. It's also the sort of thing that's a source for employee burnout.
My old standby for Severity is a 5 point scale based on data loss. It's a little fuzzy and everyone has to agree to definitions, but as a first proposal I'd say 5 is unrecoverable loss of pre-existing data. 4 is losing data entered now, 3 is loss that the user can work around (eg, enter the data via this other workflow and it's not lost), and honestly what constitutes 1 and 2 is nitpicky, because most places can't even keep up with all of their 3's, so arguing that your scenario is a 2 instead of a 1 is a bad expenditure of energy.
Save that energy for the fact that the business is going to rate 90% of all work as priority 1 or priority 5, sometimes re-ranking them after the fact. Especially for perf or security issues - always lowest priority until we don't have it, and then blame the engineer in the grand tradition of, "Why did you let me eat this whole box of chocolates?"
The colloquial usage of 'severity' and 'priority' just has too much overlap. Something like pairing 'likelihood' and 'severity' assessments, as is standard in safety, would still be general and make it immediately clear why something could be severe yet low priority, especially since people commonly mean both likely and severe when ranking a problem as severe. Keeping with the author's definition of severe, renaming severity as 'System Impact' at least immediately narrows down what is severe, but still carries the possible implication of 'frequent'.
I think fighting for a given interpretation here without using different terms is akin to insisting that people simply understand that a 'significant difference' reported in science means 'statistically different' as opposed to an 'important difference'.
It was just extra work/mind cycles for no gain. I don't feel like I'm missing out but happy to hear where this makes sense
High priority, low severity: Your program at one point displays an uninitialized value, which consistently shows "BadDon". You are conducting a demo tomorrow that is make-or-break for your startup. You're pitching to a guy named "Don".
Low priority, high severity: Your ICBM will experience an uncommanded launch when Feb 29 does not exist in a year divisible by four.
The consequences add to the severity. Not being able to save a document versus having your identity stolen versus an aircraft crashing are different severities. Severity fields that have four or five values can be difficult to use due to the orders of magnitude differences in consequences.
Since severity is experienced differently by different users, if only a single severity field is attached to a bug, it is somewhat tricky to use. A guiding principle should be that the bug reporter should imagine the circumstances of the "worst-case user": the user who is most likely impacted by the bug, with the worst consequences, and set the severity accordingly. It can feel wrong to do this, though, when it seems that such a user is purely imaginary; there is no actual user who is affected in that way or to that extent.
Priority is nearly the same thing as severity, but from the perspective of the user being the project itself. A bug has high priority if the project is severely affected. Priority usually drives what gets fixed now, versus later.
The project cares about the users, so if a large number of users experience a high severity, that bug will likely be treated with priority. The project may also care about the largest users who bring in the most business, and other things like pride in its good reputation. It might give more priority to a bug that is more embarrassing. Or it might give priority to a bug that is more widely publicized, such that there is attention on the project's handling of it. A bug that creates a blocking situation in the project's own development will typically be treated as high priority, even if it hasn't been released to any users.
For example: Crashing bugs and data loss are highest severity. UI bugs that prevent the user from doing something are medium severity. UI bugs that don't match the design are the lowest severity.
This should obviously change for Boeing where software bugs have deadly consequences.
When the bugs are filed, they're assigned a priority based on the set criteria. When the bugs are triaged, they're prioritized based on the needs of the organization/project. A typo doesn't actually make the software work less, but it's not something most organizations are okay to ship with.
I don't agree that severity is different if you aren't affected by it. A design flaw in a bridge that could lead to collapse isn't low severity just because I may never cross it.
But who is "the user"?
Bugs are high severity based on the principle that if we put ourselves into the shoes of a user who is affected, we understand that the user is affected badly.
However, a crashing bug can be such that only a very specific, rare way of using the program makes it happen. Only a tiny minority of the users who are doing something very specific might be affected.
Quite possibly, "the user" might even be actually "the", as in the one and only one who is running into it.
Furthermore, the severity of a crash bug can be mitigated by whether the affected users have a workaround: can they alter their use of the program in some reasonably small way so that their workflow can continue.
> A design flaw in a bridge that could lead to collapse isn't low severity just because I may never cross it.
That's only so because you've made it a requirement that there is a single parameter called "severity" that may be associated with the bridge, which is a reasonable thing to do. But then we have to imagine it from the point of view of the worst case user: the user of the bridge who is on top of it when it collapses. The severity estimate comes from "how bad could it possibly be".
A bridge I've never heard of in some country where I will never set foot that is on the brink of collapse is certainly not of any severity to me.
Furthermore, a bridge that is in such a state can be cordoned off so that nobody may go over or under it. If nobody may use it (other than trespassers who go past the danger signs which inform them that they do so at their own risk) then there is no "worst-case user" (that we rationally care about). In that situation, the bridge's stability issue is no longer of the same severity, even though it is in the same state as before.
> A bridge I've never heard of in some country where I will never set foot that is on the brink of collapse is certainly not of any severity to me.
That's why you would set the priority low. You don't care if it's fixed, but that doesn't change the fact that it could kill someone and is quite a severe bug.
I've always used severity as a way of objectively classifying bugs in order to help speed up a bug triage. It works well for that.
1. Severity - roughly defined as in the article
2. Occurrence - How probable will this bug happen in the field? How many users would be impacted? How much support will be needed for this issue? Low - Only some very special workflows or mainly dev work impacted. Medium - Some regular users. High - Many regular users.
3. Reproducibility - How often will the bug occur when we follow the steps. Low - Single occurrence or difficult to reproduce Medium - Erratic behavior, thread problems High - It will almost always happen.
Then a PO can set a prio and give the ticket the status "will not fix" or "to be analyzed". The dev team then takes the bug and does a time boxed (1 day max) first analysis and comes back with an effort estimate of how big the issue it's too fix. Then a PO can reprioritize and set the status to "to be fixed" or not.
If anyone has a workflow where "severity" is important for your decision-making, I'm curious to learn about it.
This has led me to think that a hierarchical priority structure would be useful. So we could say that these 100 Level 3 issues are generally of the same priority, but there's a ranking within that. But even the highest among them is still less than the lowest Level 2 issue.
That's not to say that it shouldn't also be 2-dimensional though.
Likewise, I am fond of PEF/REV https://www.fincher.org/tips/General/SoftwareDevelopment/Bug... where the user side and developer side of a bug can be measured separately to determine the prioritization.
From a developer perspective, I've had a hard time whenever I see the two labels Priority and Severity used together. I find I need to apply some sort of mental kung-fu every time to remind myself what this article is stating. (e.g. Low Priority, High Severity) The reality is, it IS confusing. Kudos to Blue Label Labs for effectively managing it well.
(edited: include an example)
Some would factor severity into priority (This is a crashing-type bug, so we should prioritize it), or priority into severity (This bug only causes minor harm, so it's not a high priority for us). But this was done silently and unconsciously, and different people had different directions for how they would do it. Due to this experience, I think one metric is best.
In the terminology I use:
Severity = total impact of the bug = (impact to each customer that experiences the bug) * (estimated rate at which customers experience the bug)
Priority level = severity / cost, rounded to fit in one of n buckets
Priority is an order. If your tracking system makes you assign a priority like "high" then it is collecting something that doesn't have meaning.
Severity: Extremely High
Priority: Extremely Low
Your P1 is my WONTFIX!
Priority in particular is quite subjective and also changes depending on what else is going on (other tickets perhaps?)
Having said that, the bug tracker we use provides the fields by default and we try to use them intuitively. We also try to discourage the "everything is P0" behaviour that happens when people use an arbitrary numerical scale. (What even is P0? I see it pop up when people discover that all their current tickets are P1.
Anyway, here's ours. They're kind of ordered but mostly try to be descriptive and sympathetic to the idea that the judgement is subjective:
Priorities
* "If time permits"
* "Next"
* "Do not defer"
* "Blocks dependants"
* "Release blocker"
* "Don't go home"
The default is "If time permits".
Note how all of them are pretty "important" sounding except the "lowest" which instead admits that sometimes things just can't be done because of other factors rather than because they're not important.
"Next" means "do this next" or "in the next phase of work / sprint / release" and things can climb up to this priority as we take decisions.
"Do not defer" means that it must be done in the milestone it's allocated to. It can't be deferred to another milestone. i.e. we've made decisions that mean it can't be deferred to the next phase. This is for things that we'll need for the next phase or will become blockers but aren't yet. It's not for things where we have generally decided that Now Is The Time. There must be a specific reason it can't be deferred. We use the milestone features to actually decide what's in or out of a scope of work.
"Blocks dependants" is pretty high up and as such needs to be explicit about what it's used for. If your bug blocks something else or has something that depends on a fix then the schedule is going to get messy if it's not done. It's hard for someone with authority to wedge something in here because they feel like it.
"Release blocker" is also pretty high up. This is where people get to jump up and down and explain why they think their bug is so important (and why it's doesn't fit into "do not defer"). This is for stuff that means we can't actually ship the release rather than things we'd just like to be in it. For example "we can't build the software until this is fixed". Hopefully reasonably rare stuff. You need to explain why this blocks a specific release rather than the release otherwise being enough better than the previous one to deliver value to users.
"Don't go home". Sometimes things really are that bad and need to be fixed. At least the person assigning this knows what they're asking, what the implications are and how bad it looks if everything always ends up in this bucket.
Severity works a bit differently. These things are hints from the reporter about the impact of this bug. They really are very subjective.
* Don't care
* Embarrassing
* Minor peril
* Major shitstorm
* QA blockers
* Showstopper
"Don't care" is the default. We want to capture everything we know about our software and encourage people to not feel too "whiny" about filing the notes. This is often for things that end up as "enhancements" or "tasks" rather than "defects".
"Embarrassing". This is something that a user would notice and be left with an impression that we have less attention to detail than the best in the industry.
"Minor peril". This is a bug that's going to cause friction or trouble and give the user an uneasy sense or stop them from building confidence in the software.
"Major shitstorm". I'm never profane in my software but someone suggested this and lots of people liked it and had a good idea of how it should be used. This is for stuff that causes problems when the feature is used. Stuff that leaves a persistent mess that needs to be cleaned up by hand or corrupts some state or something.
"QA blockers". Without fixing these bugs then we're unable to test our software properly. This means we might attract regressions and we probably shouldn't be releasing stuff to users. For example, if you can't save a record that contains the new features then you can't test the corresponding load functionality.
"Showstoppers". These are bugs that cause the whole software to break. Crash bugs, things that destroy user data, etc.
Most of our open bugs mostly sit at "don't care", "embarrassing", "if time permits", "next" and occasionally "do not defer". If stuff starts creeping up the ladder then we know we have a problem coming. If stuff starts to appear straight at the top of the ladder then we get those things fixed ASAPs: they don't hang around as "open" for very long!
A lot of the "higher" values are things that help us deal with interactions. Problems on their own are often tractable. When things start affecting other things and causing knock-on effects then trouble occurs. We try to head that off by knowing about it when it comes in and fixing those things quickly.