The sad graph of software death

One place a support manager told me that he couldn't keep his indcident queue under control without 10-15 more people. The tickets weren't being assigned and the staff was doing nothing. (The manager dude was trying to get people promoted and gumming up the works for ticket assignment)

Another case a network org had a 20 business day turnaround time for trivial firewall changes. The average request took 40 calendar days, and a small sampling we did showed that many had to be reopened several times -- in one case a DBA's request took ~200 days from open to close.

However, if you know for sure that there are tough problems going on (related to growth, limited dev capacity, code quality issues, etc), then having well-functioning planning tools can be useful. A tracker that looks like this, and the process that leads to this trend is a problem that is worth fixing in that situation.

So that's what my follow up essay will cover. How to put a very simple process in place that gets you back to the point of reliable project prioritization and measurements, without putting in a large time investment.

Because I think even smart, capable teams can end up in this situation due to various forms of external pressures, knowing how to dig out is pretty important, and ideally long before it hits the tipping point I had been at when I started helping with this particular project.

Given this, it's worthwhile thinking about bankrupt only software. Twitter is a great example of bankrupt only thinking. One of the crucial things Twitter did that differentiated itself from RSS is it never put an unread counter anywhere. You expect to go into Twitter, reading only some small percentage of your stream. You assume that if a link is important enough, enough people will post it that you'll eventually see it. But also, if you don't, it's no big deal.

It's interesting thinking about what a bankrupt only issue tracker would look like. Perhaps a version could be that when an engineer logs in, he only sees one issue at a time and has the choice of "I want to work on this issue", "I know someone who should work on this issue" and "I don't want to work on this issue". You can click around a couple of times until you find an issue to work on and then get straight to work. Crucially, there's no way to see the global list of issues and duplicates are encouraged, not discouraged. If an issue is duplicated many times, it means more of a chance that someone will hit upon it stochastically.

I have no idea if this will work, probably not. But I think people need to start thinking outside of the todo list paradigm for issue tracking to make any meaningful progress forward.

The benefit I had in this particular project is that I was an outside consultant with full access to everyone and everything in the organization AND the trust of some of the leadership there as well as a couple of the developers.

That's a pretty big benefit and not typical. Still, I think it's important to talk about what did work (I waited years before publishing this essay, and repeated it a couple times elsewhere), because even if you don't have the leverage, knowing how to make a strong case is half the battle.

Will post the followup within the next day or two!

It's certainly worth questioning whether it applies to software... I suspect the idea behind your thought is that there is no inventory cost to issues since they are just a few bits on a hard drive somewhere.

But I'm not so sure that's true. Every minute that passes after a bug report has costs: the reporter's memory slowly degrades making it harder to reconstruct the triggering scenario. The software itself slowly drifts away from the state the bug was filed in, adding questions about whether it's still applicable. And recent changes to the codebase that might have triggered the bug also slowly fade from developers' memory. Lastly the larger the bug database is, the harder it is to search and the slower de-duping becomes.

In a real sense the value of the report decreases AND the cost of addressing it increases over time.

Would be a clear indication that this isn't the heathy version that you describe but rather the unhealthy version the author suggests.

At some point your bug tracker becomes noise and you just never deal with it. I have seen it happen, and cleaning it up is a whole separate set of pain.

Guess I would delete all issues except for the most critical bugs and any remaining critical mvp features, and then I would say that from that point on you can't add any feature requests without completing something - that is you can only add a feature ticket after completing a bug or other feature ticket.... for sprints I'd say 3 to 1 bug to feature ratio and no more than 1 feature at a time in progress / in sprint at any time.

Then if normalcy returns I'd ease up. I'd also make sure there's one lead dev in charge long term on the project instead of just having whoever has "some time to do it."

Not sure if this is ideal, which is why I'm looking forward to your answer.

You can't assume this unless you freeze development. I wrote the essay... the opposite was true: rushed work was creating a runaway defect density while the product was externally experiencing growth, which in turn brought old tech debt to light as for example... things with a few percent defect rate that were easily to manually deal with before suddenly saw 10x-100x increase in frequency.

So... if you allow for a cooling phase, yes... you end up with a "more issues, but less severe" pattern. If you're in a high-growth situation with a development team that's at near 100% capacity utilization... you get infinite death graph doom. :-/

This is great advice and is ultimately what we focused on once I got past this initial triage/prioritization problem in the org I was helping.

> To make the analogy work the backlog needs to get rated on how much each open issue costs the company. Most organizations use some stratified measure of "severity". Few actually try to put a price on a problem, but I think it's worth the effort to try.

This is hard to do when the backlog is increasing by hundreds of open issues every couple months.

But the trick is ultimately to find a way to very quickly trim the backlog and see what crops back up, and then put each resurfacing issue through an economic decision making framework (even if it's a back-of-the-napkin calculation), as you suggest.

New tasks, requirements, features and nice to have go, on a macro level, into a wiki page, where all the political improductive infighting takes place. I asked the shareholders to keep a ordered list, since giving them priorities is a sure way to have everything escalated to critical. A couple week before every iteratiom three/four of the top macro features go to UX design and whatever emerges is sent to built.

So far it's working well, it isolates the team nicely from the madness above, while also allowing a single place where a roadmap exists, even if in a fluid ever changing state.

Macro task don't get into the traker because they're hardly ever finished and just pollute it; we do as much in a iteration of the feature x as needed to be useful then throw the rest back on top of the queue as x part 2 or x enhanchement.

The goal of the essay though is not to suggest that tracking issue count is a useful metric.

Instead, what I'm suggesting is that if you see lots of stuff going wrong in a project (things that are clearly costing time and money, angering customers, etc.) and then you look at whatever issue tracker is in place and you see a pattern like this, it's a sign of a broken triage and prioritization process.

It's so common for this to be happening that I've seen this in many different projects, and I think you're spot on when you say it's because an issue tracker wants to be so many things at once, and it rarely closely tracks value.

But in crisis situations, people on the front side of the business feel like they're "doing something" to help customers and users by filing tickets and then continuing to ask for status updates for them, to fight for their inclusion in prioritization meetings, etc.

Because this does not actually help anything, it actually ends up hurting things. The tracker ends up as a mediator that obscures or limits communication about the real issues, and without a change, that can be disastrous in a troubled organization.

So... the point of the graph is more of a quick way to confirm that triage/prioritization is being done wrong, so that you can make a decision on how to fix that.

And that's what I'll cover in the followup essay. :-)

Learn the tool or technology that fits well your problem space whatever it might be : golang for highly concurrent server side, functional for something that maps easily to math stuff, event based programming for request/response, etc.

I'm a software project manager. I'd push for focus from the team, and tough decisions from management. If there are 10 "areas" of the app, each with 10 feature requests, I'd pick 2 or 3 of those areas and do an amazing job developing those 20-30 requested features, and close the other 70 issues as "not fixing".

We'd maybe lose some customers over it, but make the remaining customers all the more loyal. If the issues were truly problems, they'd come back up again in issue tickets over time. I bet most were nice-to-haves, and not having them is the price we pay for good software.

But it's important to note that this is an accumulation graph... so it starts at zero but that doesn't mean a backlog of zero (my guess is it was massive, because it was massive at the time I arrived)... this instead counts issues opened and closed during the time period across the entire system. Some issues closed during this period would be ones that were opened long before the start of the period.

This means that the graph will always increase, and that the space between the two lines represents the growing net increase of unresolved issues in the tracker.

I wish I still had the raw data on this, because it's a little hard to tell from the scale that the differences from week-to-week can be pretty big (like 10 issues opened one week, 50+ another)

When I arrived, there were a few specific issues in play, along with all the usual code quality / project management issues that plague any troubled project:

(1) A high number of support requests that required manual setup work from developers, which increased greatly due to overall growth in the customer base.

(2) Some high defect density areas in the codebase that generated emergencies, and fixes that would end up breaking other things in the process of dealing with the acute issue, along w. infrastructure/architectural scaling problems.

(3) As you guessed, reduction and restructuring of team capacity, without a corresponding change in the workload.

So these things... they happen more often when we wish in the software industry, and it produces graphs like this.

(that said, keep in mind this was pretty much a napkin sketch. any issue tracker is going to have a TON of noise, unless it's very well pruned -- the sole purpose was to point out the wide gulf and relate it to the problems already obviously observable onsite, and then use that to motivate real work to change things.)

I plan to fix this when I use this graph elsewhere, but I really don't understand this comment. Is it an automatic knee-jerk reaction because you looked at the graph axes and didn't read the article?

Or did you read it, and have a genuinely hard time making sense of what was going on?

If the latter... sorry about that. However, I'm surprised at just how many people seem to have understood this idea, even without the labels, if it is such a severe problem.