Ask HN: How do you plan back end work?

In this context, backend = server side of web dev, and frontend = browser/client side of web dev.

I've worked in both individual contributor and execute level roles for nearly 20 years (still doing both to this day). In most companies where I worked, backend side of web development had trouble reconciling with business-level planning. It seems to be a backend-specific issue because of the following problems that frontend doesn't seem to have:

    1. Refactors are more likely to cause big conflicts with everyone else
    2. Tasks are more likely to be foundational, and become pre-requisites/blockers for multiple other tasks
    3. The famous "make change easy, then make the easy change" in practice makes most tasks become refactoring or foundational work, which leads to points 1 and 2
    4. Due to 3, it's very hard to predict the scope of work up front. If you try to retroactively reflect what you're actually doing (i.e. in a sprint), then chances are your board will start with 20 "aspirational" tickets, and end with 40 new tickets closed, none of which are in the original 20. That's because you only discovered what needs doing after you've almost done it.

The reason front-end doesn't usually have these issues is because it operates in small, isolated worlds, such as pages and components. Backend, by its nature, is responsible for maintaining the entire system's state. Whether it's a set of services, or a single monolith, the fact remains that it's typically a big, integrated machine.

Before you say that in a well-written codebase these situations shouldn't happen often, the truth is that there aren't that many well-written proprietary codebases. As time goes on, with proper guidance, codebases improve. In the mean time, we have to keep going.

Some might say that we must separate any refactoring or foundational work from feature building. Wouldn't it be more damaging to separate them? It makes a lot more sense for features to drive refactors. Otherwise, ironically, refactors will not be rooted in business needs, and become even more detached from business realities. Also, this doesn't work with "make the change easy".

QUESTION: How do you plan back-end engineering work? Typical sprint planning rarely pans out. However, removing sprints from backend team is kind of like giving up on planning entirely. Feels like we need some creative thinking to deal with the backend planning problem.

Do you have any advice that worked for you, that allows for both business planning, and good backend engineering quality at a healthy pace?