Ask HN: Drowning in technical debt and legacy systems | Hacker News Reader

Ask HN: Drowning in technical debt and legacy systems

12 pointsthrowaway11407 comments

I've joined a company and discovered that the "platform" I would be working on is actually 3-4 different platforms with the same purpose (provide a sort of PaaS environment to developers).

Each of these platforms is highly coupled with the other and there is a lot of gotchas everywhere. I'm coming to the conclusion that various managers were incapable of saying "no" every time a developer decided to use the latest trendy toy. They move on and all of that is here today, for me and my team to support. I don't have management responsibilities though.

I'm starting to resent management. Not because of their past mistakes but because that continues happening today. Not a few weeks ago someone was proposing something new and management was all onboard again with the idea of a new development without a plan for all the legacy. It seems they hope everything will be migrated to the newest platform by a miracle.

Anyway, I prefer to focus on the technical side of things only but this situation is preventing that. The lack of consistency in developing this platform means I waste my days lost learning about a new gotcha in the infrastructure (followed by a long story about "this is that way because...") and forgetting it a few days later. Only to be bit again soon enough.

I have rapport with management but this looks like a very long game. Any technical solutions I should use? Better documentation? Just saying things will take much longer because, well, everything is a mess?

Have you been in the same situation? Did you get out? How?

You're suffering from a very common misunderstanding about management: that their priorities should match yours. Obviously, I don't have enough detail about the particular situation but, as described, it doesn't seem that there is any particularly pressing reason for them to give priority to your angst.

The solution follows from the first sentence above. You need their priorities to match yours. Now, it's possible this may happen by accident but that's not particularly reliable. By far the best way is to actively help them come to the Promised Land.

So, how do you do that? You put together a proposal that solves your problems and you demonstrate that it solves theirs. Whatever they may be: cost, time to market, uptime etc. And then you sell it to them.

If you don't know what floats their boat, find out. If you don't know why solving your problems benefits them then why would you expect them to? And you need a proposal that's actually achievable.

Do that and you stand a reasonable chance of making progress. Or at least you'll be clear about why you won't.

At this point, most technical folk decide that it's too much like hard work and don't bother. After all, complaining about stuff can be quite cathartic. But if you genuinely want it fixed then taking responsibility is a good start.

Over the years, I've come to realise that for a not-insignificant portion of managers, their goal is to hide the fact that their whole deparment is huge mess that's (at best) not delivering much value. They only need to do it until they get promoted or get a better job offer from outside. Then, this mess gets inherited by another manager, who, after initial struggles, comes to peace with the fact the the whole deparment is beyond salvation and in turn starts to plan for his exit... They're at least as screwed by legacy as we are.

Your job isn't decision making but you can make sure management is fully aware of what they are dealing with. Outline major problems that you keep running across, you mention highly coupled platforms, that usually means changes in one system can have negative affects on another, and may not happen right away.

The big trade-offs for ignoring tech debt are software quality, system health and speed of development. Those things will get worse with time. If you bring up facts with evidence, on how those are poor because of the tech debt, it could help. Better yet, align tech debt with product strategy. For example, you know the product strategy is to do more X and system 1 is primarily involved with X but changes impact systems 2 and 3. Propose untangling system 1 from 2 and 3 as part of the effort with improving/building more X features. It really pays off when system 1 is loosely coupled from others and can evolve on it's own.

A common theme I've seen from some developers is basically to toss out the old system and build a new one. This usually never works. You probably need to start small with minor refactors to improve things and build credibility that refactoring is a path towards a better system.

When it comes down to it though, if management listens and mostly just ignores your advice, wait for the next failure and after it is resolved, gently (not "I told you so") bring it up again that maybe it could have been prevented.

Who is the system owner? What is the product strategy and technical strategy? How are they aligned (if at all). Is there a high level strategy roadmap for the business? What about a product roadmap? What management framework is in place? In the end this comes down to vision, alignment and execution (and of course financial capability and buy-in to execute on the vision).

The answer is 'no' and 'nobody' to all those questions. It seems hopeless (or at least out of touch for me to change much of that, since I hope to keep on the technical side of things).

Then there isn't much hope and I would look for a new job if changing that isn't on the cards. Your job in the technical side relies on good governance.

I am not sure if I can help with your question much, but I am sure I am in a very similar situation right now and I just learnt how to make myself feel better. No, I didn't "solve" it or "get out," but kind of not so bitter let it go.

Some context first. I have been to my current work place for a year. Our team have been maintaining a legacy system which keep upsetting me once in a while during the past year. It is a build system for many fundamental tools but the best it can give is its best effort. Sometimes a build just fails without any traceable reason, "clean everything before any builds" is the culture here. Other than reliability, the way this system utilizes machines is a mess. About 15 people across several software teams manually partition all the resources, and build/test/development can occur on any of those machines simultaneously. etc. Everyone seems to be either comfortable with it or ignorant to this issue.

So I write proposals a few times to address those problems during the past few months. It is a good thing, every body says that, but the reality is no any privileged one would ever start any practical change.

I started feeling severe Monday blue. I am always depressed about how ignorant the colleagues are or raged about every commit message. Every technical post, software engineering principle reminds me that the legacy system is just a piece of sxxt, so are those who ignore this fact. I just went out of my mind.

One day I found that I am near burnout, so I took a vacation back home and stay a few days with my parents, none of whom knows anything about software. However, it was still good to hear their wisdom about workplace things. At the time I knew that they are proud of me that I had the courage to address old problems, the rage inside me started to decrease. And now I can see the whole picture from a more neutral perspective.

Nobody wants to change, especially those who uses them in a daily basis. For any slight changes, call it enhancement/refactor/rewrite whatever, the man in charge will ask you to provide explanation from the draft to all details. The best case is that you have a PoC to demonstrate your idea, but again the scale of PoC may be challenged.

In conclusion, I tried to understand the rootedness in such legacy system, and at the same time tried to relieve myself from the pressure of "you must do something with this!" at the same time, but it seems pessimistic to me that the two goals are contradict to each other. I am still trying to promote my solution to the crazy mess, but less motivate and proactive than before. Since this is only a minor project in my whole career life, I don't have to care it so much that I can't feel anything other than anger to my colleagues. Now I feel better.

Reply on news.ycombinator.com