Why backlogs are useless, why they never shrink, and what to do instead | Hacker News Reader

Why backlogs are useless, why they never shrink, and what to do instead

lucasfcosta.com

90 pointsfeross49 comments

I guess no one would click if the title was, "Why I don't like backlogs"? N=1 but I appreciate having an unbounded backlog, knowing that the various things I come across are written down _somewhere_. It's the nature of the business that they always grow and some work never quite makes it to the front of the queue, so what?

> That’s what backlogs are: a bunch of unimportant tasks that we’ll eventually get to, but not today.

Not in my experience. At least I disagree with "unimportant."

> The truth is, we always know what task is most important, and we work on it until it’s done.

Again, not in my experience. There's been plenty of times I've reached into the backlog rather than go idle.

Fundamentally, in my experience there are lots of little feature improvements and bugs you come across that are worth documenting. I believe the issue tracker is the right place. If you don't document these things they'll soon be forgetten -- what a waste!

I also like to dig through the backlog when doing a major refactor or a large feature. I'll look for related things that might help inform the design.

It's surprising how often you find cases where we couldn't do A, because it depended on B which we didn't have back then and it would have been too much work to do B just to get A, so A just gets backlogged. But now that B is finally getting funded, if we do B right, we can let A come along for the ride almost for free. Then a whole bunch of related things like that can get cleared from the backlog all in one fell swoop.

That kind of finding is particularly satisfying.

The other time it's good to dip into the backlog is when the problem has become more serious due to shifts in usage patterns meaning the impact now justifies the work when it may have been borderline before.

Also, sometimes it is hard to either get something off my mind or to get someone else to move on from "really important to them" item. The backlog is a great place to write it down so my brain doesn't keep bringing it up.

That's exactly how we use the backlog: Whenever we come across things which could be improved, we create a ticket for it.

That could be UX improvement, ideas to speed up the CI pipeline, or general code improvements which take more than an hour max.

Each sprint one person (we cycle though) can use 20% of his time to do these tasks (we call them greenkeeping).

Works very well so far.

What a strange article, indeed. I do dislike what most bigger companies make out of agile, but usually sane people keep agile big co bs at a reasonable level.

But backlogs? They are useful, they have the most important tasks, important also to remember corner case issues and prioritize tasks, given the right circumstances they also shrink. And we mostly take stuff from the backlog...

His truth about "we always know what task is most important"... eerm, often, but must admit sometimes not, and sometimes good to align with the team and prioritize?!?

And the "what do to else" is also cute: don't do backlogs unless you do it right (or what I believe is may favourite "right").. sorry full disagree and full lol.

Would more question my workplace given something the author describes, instead of telling the internet how to do backlogs (:

I find that backlog items get reframed into other stuff too. I like having a history of them is an itemized list of things that lead to decisions. Is the only point to have a list of your absolute decisions and context that lead to the things included current product? Like some sort of psychotic performative minimalism?

A lot of times a decision is to not do something at that time. In my experience, the most important things involved in building a software product is what you decide not to do. The backlog is a knowledge base of how those decisions are made. They can be archived as needed. Decision points can be made referencing them. A year or three later when the same problem crops up, sometimes under a different name, you've got information about it. Maybe it's time. So now you have that knowledge sitting there to help new decisions.

Who cares about how long the backlog is? A queue that grows indefinitely is the type of problem bothers the analytic brain of a programmer. But as the author says, the queue does service a purpose:

> Backlogs exist because they’re a great way to avoid difficult conversations and shift blame away from product to engineering.

> Whenever anyone asks a product manager for a shiny new feature, for example, it’s much easier to say “you’ll add it to the backlog,” than to spend an hour explaining why the suggestion is irrelevant.

I think this tone is a little condescending, but the broader point is true: PMs, QAs, and others in the process want to feel like their voices are heard. The backlog gives them a place to express their voice. And of course, not all of the tickets immediately enter the backlog. It's a place to gently lay down bad or low priority ideas without insulting the person who filed the ticket.

> It's a place to gently lay down bad or low priority ideas without insulting the person who filed the ticket.

I mean yes, you can use the Backlog for that, as a sort of idiot pacifier but that's basically the worst way to deal with that. Preferably you would find ways to educate those people on why their Ideas are bad or low priority so they can learn to give you better input.

> I mean yes, you can use the Backlog for that, as a sort of idiot pacifier

Again, that's really condescending. PMs will have ideas. Some will be good, some will be bad. QAs will file bugs. Some will be good. Some will be bad. The backlog reflects the bad ideas. There's nothing wrong with that. If you feel like your QAs and PMs are idiots, i.e. you're the smartest person in the room, then perhaps you're standing in the wrong room?

If you don't give feedback to bad ideas, you're treating them as if they're unable to improve. Maybe "idiot" isn't the exact word, but it's close.

The other poster isn't being condescending, they're calling you condescending.

While I appreciate my posts being defended by other people, my intention was not to call anyone condescending. I merely wanted to suggest that any agile organisation should strive to learn from each other and find better ways to deal with bad/low priority ideas than make people feel warm and fuzzy inside while ignoring their input.

Though the use of the wording "idiot pacifier" was perhaps a bit too strong and distracted from the intended meaning.

But who is the arbiter of good ideas? This is kind of a precision / recall problem. Do you want every idea offered to be good, or do you want to get every good idea out of your team, even if that means tolerating some bad ideas? That's not to say we shouldn't strive to have better ideas overall, but the ideation process itself is valuable, even if it's unlikely to produce solely good ideas.

"Tolerating" and "keeping around forever next to the good ideas" are different things. You can say no without stifling future input.

What is the downside of keeping the "bad" ideas around forever other than you find it annoying? My point is that people only care that the backlog grows indefinitely because it disagrees with their sense of order. This is the type of thing an analytic thinker tends to care about, but the size of the backlog queue doesn't really have any meaningful impact on anything.

I'm not sure you can say no without stifling future input, or if you can it requires a lot of tact. People don't like rejection. Being rejected makes them less likely to try again in the future.

The problem isn't keeping them, the problem is mixing them with the good ideas that are low priority. Then you have to mentally re-categorize them every time you go through the list, and if you actually start to shrink the backlog then they come back to cause problems, like making you still have to reject them or someone wasting their time implementing them.

That's obviously all context dependent and in any ideation/brainstorming process it's valuable to have a free flow of ideas but there's a necessary filtering process afterwards. David Allen writes in Detail about this in Getting Things Done, specifically in the chapters of the natural planning process.

Now with regards to the backlog, if a team can develop common criteria for high value/high priority tasks and learn to reject low value/low priority tasks than they can more effectively manage their backlog and time.

You can give feedback and use a prioritization mechanism together.

"Hey, that's really difficult right now, because ____, so we'll put it in the queue for some future time."

"I'm not sure how exactly we can do that and still <do other important thing>, because ______. It's in the queue so we can think about it later."

"Doing _______ seems to create a significant problem in ____________. Maybe there's a better way. In the queue."

This way the person doesn't need to be told an outright "no" or back down; they can save face and have time to change their mind. Similarly, the idea is preserved so that if these reasons weaken or we come up with a better idea about how to do it, we can move forward.

Finally, you preserve the concerns about the idea. Then, if you're not there when the idea is brought up again, then they can encounter it already-in-the-backlog with the concern/problem.

That's why you need to have backlog grooming sessions regularly. Go through the backlog, prioritise it and see what ideas aren't useful anymore (already implemented, obsolete, stupid etc).

Foolish to generalize backlog’s as a useless technique because the author has had bad working experience from them.

As the author found, if you tune your backlog processes they can be quite helpful. The only part of agile I use today is the backlog. If you keep engineering details to a minimum (or limited until it reaches the top), items brief and to the point, and have it groomed regularly so it is always bounded to a size it works well.

I find that on multi year projects you can “archive” the bottom 20-50% of tasks and no one will care. So it cleans itself up quite nicely.

True. Backlogs are one of several things that people think of as "agile", but have been around since forever. Although prior to agile, we called them "todo lists", but they were dealt with pretty much identically.

We should call them todo lists again, to do away with the problematic capital-a agile association.

I feel that too many people have only seen shitty scrum/agile and the whole terminology is tainted to a cargo-cult level.

Agile or Scrum?

> archive

As long as you don't do what a lot of open source projects do and use stalebot to lock old bugs with no ability to reopen them.

My experience with the backlog is somewhat different than the authors.

Mostly my experience is QA opens bugs and engineering opens tickets for tech debt, refactoring, performance improvements, reliability improvements, etc..

The PM opens new feature tickets, prioritizes those, and pushes the bugs & tech debt tickets to the backlog until their back is absolutely against the wall.

Every so often a VP or someone will come in and call PM to the carpet and engineering gets to do a "quality release" where bugs are fixed and tech debt is cleaned up. But then things will regress again to PM just making sure new features are ticked off at the expense of other tasks that languish in the backlog.

You described my situation exactly as it is. Is this a common case for product centric companies? Is it less prevalent in tech companies?

I'd say it's common for all companies that create software.

Well refactoring should never be a ticket it self. It should be done in order to facilitate, even enable, the construction of a new feature. Perfomance should be a feature request, IMO. If the user can't use a feature, or se a result, due to performance, the should be requested by the PM it self, no? Reliability as well, should fall under the same logic.

The problem is that some refactors will languish eternally as they don’t serve an immediate feature goal or even a bug fix—even if taking on the abstract refactor work broadly enables whole sets of features and fixes to be done more effectively.

In my current work experience, this means having a fairly deep mental model of how a given refactor might impact various seemingly unrelated efforts, and knowing how and why to advocate for them. This advocacy doesn’t translate into a literal ticket (and that’s not how my team manages all work items and their prioritization), but it definitely translates into “I’ve identified a specific unit of work which doesn’t immediately serve these goals, but would broadly improve their velocity and success.”

Next to no one wants to hear that kind of advocacy, because their Babel Fish generally formulates it as “stop or slow immediate progress for abstract goal”. But that’s what the debt part of tech debt is: you have to sometimes prioritize getting out of your own way before you can move forward in any direction.

As far as performance as a feature and corresponding request, sometimes you need far too much technical detail to even know what to ask for. I’m saying this having just landed an optimization which completely obliterated the existing Y axis on one of our monitoring systems. There was some vague sense of where some of the optimization target might be, if users or even the team were able to make the connection. It took dedicated time just profiling a bunch of fixtures, without anything to go on besides “some stuff is slow, for example ___”, to understand the broader performance story which went well beyond known bottlenecks.

Part of being an engineer is, or at least should be, some freedom to dive in to make these kinds of connections independent of specific features and specific detailed feature requests (performance or otherwise). Even if the only “work product” is just a report on what you found, at least it might be cross-referenced for when it does correlate to specifically ordained work items.

Didn’t your engineering team have and agreement with the product manager to devote a portion of resources to tasks added to the backlog by engineering such as technical debt? You need to find a balance between new features and bugs and improvements.

In my experience that is just extremely hard to do. It's usually better just to do refactoring outside of the whole sprint / backlog process.

Yeah it shouldn't be like that but in practice that's the only way it happens.

My experience is that customers want features shipped yesterday and the critical bugs affecting them fixed asap. Companies want money to flow in and penalities not to kick in. Refactoring serves neither of these goals so it's rightfully in the middle of the pile with uncritical bugs and is done when there is time.

Backlogs should be a bounded queue, not unbounded. You need something that's longer than 5 people's collective short term memory. But as soon as it has items that are "too old" * "too minor", there's a great chance they start becoming obsoleted, recognized as worthless, etc.

Tough to do, though, because there can be genuinely important stuff in the backlog that shouldn't be forgotten, but for various reasons won't be tackled anytime in even the medium term future.

I mean, most ticket systems let you filter by date created or updated... seems like you can ignore stuff that's older than x months.

Not always. You get into an X month crunch because your salespeople sold far far ahead, some old things are still relevant. But date's a good proxy.

The older it is, the more likely a newer ticket was created for it as well to be sure

The neat thing about those items is that you’ll never get to them anyway. We sort our backlog by priority constantly, and all of the junk, old issues, or irrelevant bugs simply get shifted to the bottom over time, with more important stuff sorting itself up. Nevertheless, if I come across something I’ve documented before, I don’t need to file the same issue over and over again; if multiple customers request the same stupid thing over time, you’ll have a track record and maybe notice there’s a need for something; and finally, a long backlog gives lots of opportunities to onboard new people with long-standing, but minor issues.

From my experience I mostly agree: Our backlog is too long to be usefeful. It contains small bugs that can be fixed in half a day and big features we will never implement with the current size of the team and everything in between. What we work on comes much more from current customer projects and maybe from the CTOs guidance.

However, claiming that bugs would be a feature after 3 months and can be closed is not my experience at all. We do fix bugs that are much older. They have been annoying for a longer time, but eventually we are happy to get them fixed.

Now one could say our problem is that we are not agile enough. Well, I have seen many processes in more than 20 years of software development. They all had one thing in common: In one way or another it was a mess. Sometimes people are working more or less productively despite the mess. And sometimes they don't. It depends on having the right people and the right spirit. The process won't save you, but a stupid process will spoil all spirit and good people will leave.

> A factory that produces cars faster than it can sell them is not producing cars. It’s producing waste.

Car analogies, fun, usually a bad idea. But if we're going down this path the right analogy is not to produced cars, but to produced orders for cars. Backlog tasks represent wants/needs, not a finished product sitting idle, which by analogy makes them akin to car orders from customers.

Having an unfinished and ever growing backlog means that, in the best case, you have more dreams of features than you can handle (not awful, but bad if you don't prioritize or have a clear objective for the system). In the worst case, it means you have crappy, buggy software and are too busy keeping it alive, or attending to creating more broken features, to fix it (this seems to be more common).

It's more like a list of ideas for cars or parts of cars or colors of cars or how they could sound or . . .

If the list isn't there, someone still had the ideas. I suppose the author thinks ideas are waste, too.

GitHub gave us an eternal backlog by default. Anything we didn't want to ship got closed with some special label affixed to it.

There's no reason it should ever shrink in this kind of setup. The careful open/close visibility separation throughout seems to keep things at ease. We are almost at 20k closed issues, so I feel like any cracks would have formed by now.

The process for searching the "backlog" is about as simple as you might think. All you have to do is constrain for 2 things. Moving something from backlog to triage is literally 1 click.

Depending on how organized your software factory is, this stuff can become pretty magical.

I'm not an SWE and I don't work with them, but I have a backlog of a different type. I'm not sure it strictly qualifies as "backlog" for what the article's referring to. My backlog involves tasks that are blocked waiting on a coworker to take action.

Therefore I can generate a personal backlog at a certain rate, and the requests are tossed into a queue that's common to the rest of my team, and waits for a senior member to investigate and sign off and direct us for the next step.

My backlog grows as a certain percentage of the tasks I take on in a given week, and so it's proportional to the volume of incoming tasks in general. The backlog is cleared at a rate based on how much bandwidth the seniors have to process them, and there's only about 2 who are responsible for this aspect currently.

So my backlog tends to grow and shrink, and it really shocked some other coworkers when they found out how deep it was!

Sometimes the only way to shrink my backlog significantly is for me to stop working altogether until the seniors have a chance to catch up. I am, by far, the most prolific generator of requests compared to my coworkers, and I'm not sure whether that is a function of the percentage of tasks I take on, or if I have a much higher rate of flagging stuff than all of the rest put together.

Anyway, there's no way for me to avoid generating this backlog. It's just a fact of life. So we deal with it. It's overall a good thing that we catch these things and filter them properly, even if it delays resolution by a few days apiece.

The most effective product evolution I’ve been involved in was at a startup. There were no backlogs. Just important features in progress or waiting to start.

Everything beyond that was meaningless. They would no longer be important (for a myriad of good reasons) or look completely different when we got to them.

“Everything further away than two or three works of work should go into a high-level roadmap.” - this is supposed to be a solution, moving text from one place to another. People worry too much about minuscule organizational matters.

Problem: bathtub analogy

Possible solution: theory of constraints

> "when was the last time they had to take something out of the backlog because they ran out of things to do"

Happens to me quite a bit actually. There's a huge crunch of features that need to ship yesterday, followed by a lull where I pick up boring tasks like "documentation" or "unit tests" :)

We have a place where anyone can put a ticket for the product manager to consider. He moves the more important ones into the backlog (with some... coaching from me). The backlog has things in it that wouldn't fit in this sprint but are probably going in the next couple of sprints.

literally everything he wrote is wrong

Reply on news.ycombinator.com