Beyond the Code: Mining Self-Admitted Technical Debt in Issue Tracker Systems | Hacker News Reader

Beyond the Code: Mining Self-Admitted Technical Debt in Issue Tracker Systems

arxiv.org

97 pointseqcho57 comments

I wish there was a way to leave a comment like

    // todo(gravypod, techdebt): some description of the issue

and have that, when pushed up to a VCS, turn into an issue created, assigned to me, and labeled with tech debt. I know this exists in some of the walled gardens like Google's internal infrastructure but I think if this existed in the open world you'd see a lot more consistency between issues/comments and you'd enable more continual cleanup.

My guess is, in some less healthy organizations, tech debt comments get cleaned up more frequently than tech debt issues. Issues probably don't get assigned (a practice I have experienced at many companies) but comments are "right there" and can be rolled into other tickets in an attempt to always leave it cleaner than you've found it.

There are github actions to help out here. I use the todo "bot" that marks up todo comments you would create in PR's or otherwise into issues. You can for instance assign yourself/someone and change how you want it labelled.

https://github.com/marketplace/todo is one of them.

That's neat. I never knew this existed. Unfortunately I mainly use Gitlab wherever I work.

I don’t know much about git hooks, but I think that it would be fairly simple to wrote a small one that parses the changes for the todo comment and then calls to Gitlab’s API. Double points for closing the ticket if the todo comment is removed.

Doing it client side might make it hard to do things like accepting MRs from forks.

You could probably do it with a Gitlab CI step + python (or whatever language of choice) script as well.

There's an issue for this in GitLab's tracker:

https://gitlab.com/gitlab-org/gitlab/-/issues/14259

Sadly, it's four years old and seems pretty stagnant.

It's unfortunate gitlab is built as a monolith. I'd be very capable of writing the code for implementing such a feature myself if gitlab had a first class API. The code, as we all know, would look something like this:

  for commit
    diff = commit.diff
    for line in diff.result
      if 'todo' in line
        line_id = hash(line)


        if existing := issues.find(tags={'todo_id': line_id}):
          existing.update(code_url = new line location)
        else: 
          issues.create(....)

But because these APIs aren't first class a change like this requires deep knowledge of gitlab's internals and acceptance and prioritization of features from gitlab's core team.

Hey there, I'm a GitLab community advocate, just wanted to thank you for brining attention to this issue, our team commented to get the conversation moving today:

https://gitlab.com/gitlab-org/gitlab/-/issues/14259#note_327...

I've been thinking quite a bit about the topic, and I still think having tech debt issues documented as code comments is better than in an issue tracker. You will get more contact with the issues than if they're buried in a backlog

I wrote a bit about what was going on in my head here, if you're interested: https://isidoro.io/writing/tech-debt-broken-windows/

While it's true that we spend more time in the code than in issue trackers, the advantage of the issue trackers is that a discussion can happen with arguments on both sides and a final decision can be taken, along with a plan to do it.

Every time I see a conversation in comments like this:

    // TODO (mr): Make this more generic
    // rp: we only use it once, why change it ?
    // ol: I agree with rp

It feels too weird to leave in place. Should I add a comment with my point of view ? If the function in question is modified such that the comments are invalid, should I delete the comments ? Leave them for "documentation" ?

That's a very valid point. I was thinking from the perspective of the people contracting the debt. Usually you know what you would need to do if you had time and resources.

But yes, for finding existing and uncharted tech debt items, and deciding on their future, the discussion should happen somewhere else.

I've always wondered about other people's experiences with backlogs. In all of the client accounts I've been to, a backlog was at the end of the day, simply leadership acknowledgement that there exists more work to do than there is budget to do it, and the backlog really should be mostly the last stage of an issue triage.

I wonder if any organizations would benefit from the effort to maintain an officially-recognized triage list of "we have to let these remediations die" issues, instead of a backlog. Once issues make it into a backlog and stay there for longer than T time (usually about a year), I've very rarely seen the issue make it out of the backlog into a sprint again, even years later.

At that point, it makes more sense to write up these issues into an architecture document that explicitly acknowledges the technical debt as the cost of doing business around the codebase while relating context to the rest of the architecture, and work up a systematic way to track the technical debt. And track the estimated effort that goes into working around each item in that list in the sense of, "if I didn't have to work around this debt, and it was fixed in the manner noted, I estimate it would take me -T time to complete this task instead".

At least that way leadership gets a sense of the actual cost of carrying specific pieces of debt around over time instead of only being able to evaluate acute debt. It might be acceptable to pay an estimated 100 hours per year for one item indefinitely, while for another similar item the underlying business mechanics makes it worthwhile to fix it now than let it accumulate over a 5 year schedule.

At my current place of work, we‘ve been experimenting with two-week “sprints” (hate the term but it’s standard, so bear with me!).

We spend half of the off-week sprint planning session “pruning” the backlog: pointing and discussing new tickets, and refactoring or deleting the oldest tickets.

If the second half of anyone’s sprint fees thin, we add a ticket (more or less of their choice) from the backlog to their sprint.

I kind of like this system, because it allows you to think in cycles both longer and shorter than the sprint. If I see some fun optimization, but know I won’t be able to fit it into my sprint, I can craft a well-documented ticket in confidence that someone will have fun getting to take care of it in the future. Backlog tickets are also a fantastic way to onboard new engineers, because the tickets come from all corners of our system: from infrastructure upgrades to copy changes.

Of course this system only works as long as you have disciplined engineers. Bad tickets will drown this system. But the “checkpoint” every other week has really been all we’ve needed to keep quality consistently high.

I completely agree with what you said.

But tech debt, at least for me, feels very different from the tasks that should be triaged. It's very uncommon that we allocate time to repay certain debt, so does it make sense to triage and/or backlog? That was mainly my point with previous comment.

My experience is more like doing some other task, coming across some tech debt, and repaying it on the spot if I'm not that constrained by time. For paying debt on the spot having a clearly flagged issue and proposed resolution would likely help me navigating and avoiding known caveats, and get a fix faster.

Even if I don't have time to fix the issue in the spot, knowing the reason why things are like they are might help with my morale and be less frustrated with coworkers.

But you mentioned a systematic way of tracking tech debt and the cost of each item per year. How is that different from a backlog after a while?

From a lead, product manager position yes. When I know what is coming down the pipe I can push the rework backlog items in a prior sprint, so that things don't get difficult for the following sprints feature set. It does work assuming my pipeline is consistent and not at a tail chasing stage.

> But you mentioned a systematic way of tracking tech debt and the cost of each item per year. How is that different from a backlog after a while?

Backlogs I've seen have tended to evolve into a graveyard of wish list items. But there is rarely even an attempt at quantifying the organizational cost of putting off tackling the debt until it ties into some obvious, acute measure that usually is some capex, typically software licensing, or some really way off opex, like hundreds of operations staff manually addressing issues that could be automated.

I'm wondering with so many organizations now tracking software engineering effort at a relatively granular level now with various degrees of agile projects, whether or not I could leverage the mere fact that we're seeing the effort listed out now.

Example: my team has a bunch of servers all mounting an NFS file share. Neither the OS or NAS teams ensure the mounts stay up. For a few years before I arrived, the mounts across the landscape just deteriorate, handled by individual support tickets until the degradation becomes too large to ignore, and then a task is tacked onto the current sprint for some poor schmoe to log into each server and ensure the mount works.

A quick scan of the incident tickets and the historical tasks gave me a defensible estimate of an average of 50 man-hours per year spent on this issue alone, not even counting any meetings to discuss it with customers.

In the real world, I had to use my personal Emacs Org mode agenda to keep attempting (until I succeeded) to add a task to an upcoming sprint to put in requests to automate monitoring, mounting, and alerting the OS and NAS teams when the mount fails. In "systematic technical debt management" world, when I see the incident pattern I would create a backlog task. The backlog item sinks to the graveyard/tech debt status, and every time a new incident or task comes up that would not if we resolve that debt, the time it takes to address the incident or task gets added to the debt.

The problem I'm wrestling with implementing such a mechanism is it relies upon people to remember and care about the debt in the first place, to associate it with a current work effort. That doesn't work in my experience; it's hard enough to get people to share what they are working on in such ticketing systems in the first place.

> I wonder if any organizations would benefit from the effort to maintain an officially-recognized triage list of "we have to let these remediations die" issues, instead of a backlog. Once issues make it into a backlog and stay there for longer than T time (usually about a year), I've very rarely seen the issue make it out of the backlog into a sprint again, even years later.

Another option is to treat backlog as a graveyard for issues. They might rise from the dead, but most of them will not. Still, it is useful to have them. When someone comes with a "novel" idea they can use the old ticket with all the existing discussion pro & cons instead of starting the same conversation over and over again. Surprisingly lots of things only seem like a good idea, but when discussed they turn out to be WONTFIX issues.

well if your code base was like mine you would have //BUG:1234 littered everywhere so no need to not use the tracker.

If handled maturely, TODOs can be written permission to other devs to make wholesale changes to parts of code you wrote without any social awkwardness.

When I was young and had a smaller toolbox, refactoring was right at the top, and people ended up “giving” me modules they had written because within a few months they didn’t recognize their code anymore (especially if they were busy elsewhere). It was great for my professional development, at first, but a developer’s scale is limited to how many responsibilities they can juggle. Older me, with a larger toolbox, also sees that maybe there weren’t so happy to hand over their work like that.

Don’t fix absolutely everything that you know how to fix. Leave people hungry for more. That invites new contributors and keeps the old ones engaged for longer.

I had a good friend in college who was a Lit major and so I got a steady stream, sometimes direct, other times by osmosis, of motivational philosophy for creatives. Apparently Hemingway, in an interview about how to maintain momentum, said you should stop writing when you know what happens next. When you pick it up again you always have something you know how to do.

If someone else beats you to it, then they end up with more ownership of that code, allowing you to focus on other things.

While very occasionally useful I think technical debt issues mostly should not exist. Improving code quality should be considered part of any issue that a person takes up. Any code that relates to the issue should also be viewed as a potential candidate for refactoring as a part of that same issue.

Notice also that in TDD refactoring is part of the TDD cycle: write test, write code, refactor. Doing TDD would make you do what is described in the previous paragraph as part of the cycle. So generally one would refactor something multiple times a day.

The ticket serves as a communication point and a cost estimate tool. Even if the tasks are never explicitly assigned they are very useful. That's why I'd like a comment ticket linkage system. Support the positives of both usages by automating everything.

I could see this showing up in a near-future version of GitHub, along with enhanced default logging views.

In practice this results in too much time spent “cleaning up” and not enough time delivering business value. Hence why tech debt tickets are valuable, they allow the business to decide when to spend time on them.

Yes, debts which need to be paid off immediately hardly serve their purpose.

It’s really hard to convey tech debt cost. Probably many systems have a ton of problems adding 5% to every feature here, 10% there, bad DB design from early in the project that makes a bunch of actions far riskier than they should be (but until it breaks several times no-one cares and if your devs are putting 3x as much effort into those dangerous operations and know what they’re doing, it won’t), and so on. Communicating that in a way the business folks will care about is hard, and the full costs are difficult to account for.

[edit] hell, communicating that in a way the weaker end of technical managers will care about can be hard, for that matter. If they came up “doing it wrong” and are of a certain mindset, good friggin’ luck.

People caring only a limited amount about technical debt is rational, arguably. Opportunity cost is high.

And lots of organizations would be willing to borrow money at a usurious interest rate-- why not borrow technology for the same? ;)

Never ask permission to do what has to be done. If they say no, and you do it anyway, then you are being insubordinate. If you don’t do it and then point to the decision as a reason why the project is failing, then you are right, but are a failure.

And if you take too long to ship you're still a failure.

I personally force stakeholders to define sunset dates on the system / code ill be writing.

If its sunset date is never, cost of development proportional to roi approaches zero.

If the sunset date is near, cost of development gets very, very expensive.

Having that information both informs my priorities and the understanding of how long development will take / cost.

> If its sunset date is never, cost of development proportional to roi approaches zero.

I am not sure that is good economics, because the discount rate for the future revenue approaches 100% over extended time.

Does it?

Not everything is monetarily cost based.

Cant recreate a space probe 30 years ago to replace the one malfunctioning now.

Cant recreate makret share gained against compeitors with stability problems.

Interesting point though, definitely ignored it til now.

Iguess the point is, everything is a trade off, and we shouldnt be acting like its either high quality or high throughput or high cost. Its somehwere in between and you get to choose depending on your contlstraints and needs.

I don't really believe this although it seems rather hard to prove which side of this debate is right. Presumably it also depends on circumstances. I think any sort of messy or bad code immediately slows down developers. The truism that code is more often read than written comes to mind. This 'devivering business value' thing also sounds doubtful to me. Any sort of bad code is a breeding ground of bugs. So you have completed three stories this sprint and hence delivered three items of 'business value'? Sounds nice but it may not be so great if two of them are going to break in the near future because of code quality issues. In that case it maybe is just not true that three items of business value were delivered. In the context of bad coding practices it kind of becomes unknowable how many items of business value were delivered.

I kind of despise the term 'technical debt'. It seems that in the beginning it was introduced to make the business side take code quality issues more seriously by likening them to something they could understand, i.e., debt, but now it seems to function as a way to postpone the fixing of issues, perhaps indefinitely. I think most things that go under the moniker of technical debt have an interest rate that is far higher than any business would enjoy taking out in terms of a loan.

Let me give an example. Some code can be badly formatted. This hardly costs any time to correct but maybe it is also not worth it? Well, I once encountered some code that clearly was in the wrong function, so I asked the colleague who tends to know about that part of the application. Then it turned out it actually was not in the wrong function but instead there was no blank line between the two functions making me not notice that it was actually two function instead of one. In the mean time I had disturbed a colleague which also has a real cost in terms of getting back into whatever he was busy doing. All code quality issues have costs associated to them that are very difficult to know beforehand but I think the safer way is to always be improving code quality and that which is called the internal quality of software should actually just be maximalized and that technical debt most of the time has more costs than benefits and that the costs will come sooner than anyone expects, long before you MVP is actually finished so that you might actually have paid it off profitably beforehand.

Badly formatted code is hardly a true form of tech debt. Especially considering all the automatic tools like prettier out there today.

Tech debt is getting a feature out there, asap by creating work around that normally would require some refactoring.

In a lot of cases, it is forgoing optimization. We have several places in our app where we need to refactor some of our database tables in order to properly join the data and get it in an efficient manner.

But that will take a few days of refactoring and testing so instead, we just query the missing data in a subquery 200 times or whatever.

For now, it works fine, but as the dataset grows, that will obviously become slow and expensive.

But by then, we will have more customers and more budget and hopefully more developers that we can afford to refactor that code.

If we had unlimited time and resources, sure, go ahead and refactor your entire codebase every time you add a feature.

You'll build an amazing company, assuming your customers pay you based on how awesome the source code is, rather than what it can do for them right now.

It’s amortization, which tends to work pretty well from a scheduling standpoint. You don’t have to get everyone to “Make the change easy, then make the easy change.” Even a few people doing that can fight a lot of entropy. It’s enough if everyone makes the change easier, then makes the easier change.

Every feature added to a section of code is new information about the shape of the project. Making better decisions as new information arrives is a perfectly reasonable way to stay between drowning in tech debt and deadlocked in over-engineering. YAGNI (yet) comes up so often I think it should just be YAGNIY. Also gets around the people who want nobody to ever enjoy their work by poo-pooing every single attempt at improvement.

The business can and should set a general tone about where it wants to sit on the sloppy vs. perfectionist spectrum, but it's not competent to make those decisions at a granular level. That requires engineering sensibilities.

We have a small but dedicated percentage of sprint bandwidth dedicated to engineering's internal priorities, off limits to PMs. It works decently well. Certainly better than letting PMs make calls on which modules are most in need of refactoring.

Out of curiosity, what's the percentage?

I do something like this approach too - Friday mornings I try to start with devops / tech debt / hacking instead of product priorities.

It’s been between 15% and 30%. Some of that is just the treadmill of internal platform migrations though. 15% for local team’s own needs.

Hm, at $job we're already sending all git commits to a Bugzilla (no bug in commit comment => push is rejected). Parsing the comment or diff for these instructions shouldn't be too difficult.

Though we have several products + subproducts defined in Bugzilla (that's why we can't easily migrate to more modern solutions). Not sure if there is this can be easily mapped automagically.

Also, keeping track if the comment is moved around, especially if slightly altered with the same commit, might be less straight-forward. Also merges might be a pain.

Could also be done for those chancy subtask/checkboxes gitlab and github offer (and which I really miss in bugzilla).

I created srcoftruth.com for this reason, but it's a hobby project - and I'm currently rebuilding it.

Sounds pretty doable at the most basic level. Spin up a small server or use your PC, make a daemon/service which pulls your master branch in once every 15 seconds. From there you can either parse new changes on your own, or likely use hooks in git. Just parse the files for your syntax, likely using some kind of manual per page uid. From here, I would send an email into our system, as no api or credentials are needed, and we already have that set up. Using an api wouldn't be that much more work.

Sounds like a fun project.

The complication is that information is now in two places.

So the next step is to use your tracker api to create an issue, and get the url to it. Then replace the comment in the code with the url and make an automated commit/push.

Still all seems very doable and fun.

Google's "infrastructure" is just people asking you to make a bug (issue) and put

// TODO(b/<bug number>): Fix this up

during code review. Sometimes I'll just put TODO(<username>) if I don't really care about keeping an eye on it.

It's too bad those tickets aren't created automatically. That probably raises the bar for people wanting to add these sorts of comments.

I've already built a little tooling around pull requests and git commits at autodesk and so when I read this, I thought, oh, yeah, that's a great idea, I should do that.

But here's the thing. Our team already has a policy for TODOs: either you open a ticket and leave the ticket number in the code, or you do not leave a todo at all.

This way, we're never guessing which ticket belong to which todo, and all of the necessary context (including what spawned the todo, why it may or may not have been a good idea, etc) can be present with the code.

I think in this case, the policy is a better system than just randomly opening tickets. It increases the barrier to entry if you are going to leave a TODO (versus leaving it out or just not being lazy and doing it), and prevents the backlog from needing even more grooming.

> It increases the barrier to entry if you are going to leave a TODO (versus leaving it out or just not being lazy and doing it), and prevents the backlog from needing even more grooming

I would rather have many tickets that specify actual work than a lot of work specified in no tickets.

Using monotony as a filter doesn't sound like it would lower defect rates over long periods of time.

Conversely, it actually serves as a reminder for you to prioritize the work rather than coming across and old todo as a happenchance, when you're not in the frame of mind or have the time to fix it.

Yes, it's tedium. But it's necessary tedium. It's documentation.

I've taken to adding a small linter: TODOs must have a date attached, and after that date they'll fail a (fast, skippable) lint job.

My goal isn't to require tackling them, merely to make them annoying enough to prioritize. So far it has worked out pretty well - within a day or three of a deadline passing, about half the time the date gets bumped a month and a task is made for the next sprint, a third it's just fixed, and the remainder are no longer relevant so they're just deleted.

This is a great idea. I think I might do the same. Thanks!

My rule is that a todo should have a way to get to done.

If it’s important enough to write a todo then it’s probably worth doing and prioritising against everything else.

Eh, that's a bit extreme. If I write a todo in code rather than just doing it, it's because I've already made the decision that it's not important/needed enough to do now.

RedHat has a maven plugin for this. Let me see if I can ask someone to share more details.

A tool like the paper encourages won't hurt, but in my experience the biggest problem with tech debt isn't on the identification side; it's on the resourcing side.

I've worked at a few places with a big big backlog of, I guess, SATD. But saying a particular bit of code is "bad" is not enough to get funding/resources to fix the issue.

Usually other departments have a say in resourcing. Depends on your org who they are, but some collection of product, sales, CS, or marketing. Once the code ships and the feature is working well-enough to accomplish its business objectives, developers have lost all leverage to circle back and fix the tech debt. Those other depts don't care how "healthy" the code is; code is only a binary works/doesn't signal to them.

I encourage teams to not make these lists without first thinking about how the list gets consumed. Writing an apology comment or SATD ticket are definitely going to make you feel less guilty about what you're shipping. But it won't help you fix the debt. You have to address that upstream, by injecting engineering concerns into your company's resourcing decisions. Or slow down initial delivery to get it right the first time when engineers still have leverage.

It's so hard to prioritize tech debt and usually those cards only come back to the conversation when it blocks a new feature or causes a big bug.

Now as a manager, when new work comes in I ask my team if any solution for the new feature would allow us to also pick away at any of the tech debt cards. There is usually always a few and we plan it into the deliverable timeline.

I keep noodling my way around this

https://github.com/mikadosoftware/todoinator/blob/master/tod...

The idea is a sort of markdown approach to things todo in the code

[ ] Must foob the blah [x] Rewrite for snoobs

It kind of works - I get a view of a codebase as it was when I last left it. It is rather like trying to serialise my brain state as Instop coding somIncannpick it up again.

We use warnings instead of TODO comments so the thing to do is visible in the logs when you run the code.

This is nagging, visible, and taunts you as long as you haven't done the thing "to do", as opposed to "TODO" comments that become invisible at some point.

You can use flags to run the code to raise exceptions for any warning and you basically have your sprint laid out for you.

I did a research on this topic. Clearing your TODO list is a very efficient refactor. It tends to reduce the ratio of future bugs. I summarise it as: "If it was important enough to mark in a TODO, it is importnat enough to fix!" https://www.cse.huji.ac.il/~feit/papers/Refactor19PROMISE.pd...

I created a tool (bumbailiff) that allows a team to accrue a limited amount of debt, for a limited period of time. Then it fails the build.

Read more about it here: https://cucumber.io/blog/bdd/todo-or-not-todo/

Making a Self-Admitted list of personal deficiencies, and then mining that for self improvement, should be an app for that.

Reply on news.ycombinator.com