What I want to say is that I've seen what happens in a team with a history of quick fixes and inadequate architecture design to support the complex features. In that case, a proper bugfix could create significant rework and QA.
What I want to say is that I've seen what happens in a team with a history of quick fixes and inadequate architecture design to support the complex features. In that case, a proper bugfix could create significant rework and QA.
The joys of enterprise software. When searching for the cause of a bug let you discover multiple "forgotten" servers, ETL jobs, crons all interacting together. And no one knows why they do what they do how they do. Because they've gone away many years ago.
and you just look at this and thinks: one day, all of this is going to crash and it will never, ever boot again.
And then comes the "beginner's" mistake. They don't seem to be doing anything. Let's remove them, what could possibly go wrong?
Move fast and break things is also a managerial/cultural problem in certain contexts.
You can only say with a straight face that if you're not the one responsible to clean up after Musk or whatever CTO sharted across the chess board.
C-levels love the "shut it down and wait until someone cries up" method because it gives easy results on some arbitrary KPI metric without exposing them to the actual fallout. In the worst case the loss is catastrophic, requiring weeks worth of ad-hoc emergency mode cleanup across multiple teams - say, some thing in finance depends on that server doing a report at the end of the year and the C-level exec's decision was made in January... but by that time, if you're in real bad luck, the physical hardware got sold off and the backup retention has expired. But when someone tries to blame the C-level exec, said C-level exec will defend themselves with "we gave X months of advance warning AND 10 months after the fact no one had complained".
Also with enterprise software a simple bug can do massive damage to clients and endanger large contracts. That's often a good reason to follow the Chesterton's fence rule.
It's not in the C-level's job description to manage the daily operations of the company, they have business managers to do that. If there's an expensive asset in the company that's not (actively) owned by any business manager, that's a liability -- and it is in the C-level's job description to manage liabilities.
said C-level exec will defend themselves with "we gave X months of advance warning AND 10 months after the fact no one had complained"
And that's a perfectly valid defense, they're acting true to their role. The failure lies with the business/operations manager not being in control of their process tooling.
Of course you can get lost on the way but worst case is you learn the architecture.
Cost efficient for your team’s budget sure, but a 1% chance of a 10+ million dollar issue is worth significant effort. That’s the thing with enterprise systems the scale of minor blips can justify quite a bit. If 1 person operating for 3 months could figure out what something is doing there’s scales where that’s a perfectly reasonable thing to do.
Enterprise covers a while range of situations there’s a lot more billion dollar orgs than trillion dollar orgs so your mileage may very.
In a reasonable organization only very minor systems can be undocumented enough to fall through the cracks.
Stuff that’s been working fine for years is easy for a team to forget about, especially when it’s a hidden dependency in some script that’s going to make some process quietly fail.
> Stuff that’s been working fine for years is easy for a team to forget about
That's why serious companies have a documentation system describing their processes, tools and dependencies.
Documentation on every possible system that could use the resource would need to be accurate, complete, have someone locate and actually read it, remember, and communicate it with someone in a relevant meeting which may be taking place multiple levels of management above the reader here. As part of that chain when a new manager shows up and there’s endless seemingly minor details, so even if they actually did encounter that information at some point theirs nothing that particularly calls out as worth remembering at the time.
That’s a lot of individual points of failure which is why I’m saying in the real world even well run companies mess this stuff up.
1) I was (temporarily) the only one still at the company who knew why it was there
2) I only knew myself because I had reverse engineered it, because the person who put it there had left the company
Now, some of those things had indeed become unnecessary over time (and thus were removed). Some of them, however, have been important (and thus were documented). In aggregate, it's been well worth the effort to do that reverse engineering to classify things properly.
____
[0] https://open.substack.com/pub/lunduke/p/the-scream-test
(Obviously not the first description of the technique as you’ll read, but I like it as a clear example of how it works)
But you hit on a point that seems to come up a lot. When a user story takes longer than the alloted points, I encourage my junior engineers to split it into two bugs. Exactly like what you say... One bug (or issue or story) describing what you did to typify the problem and another with a suggestion for what to do to fix it.
There doesn't seem to be a lot of industry best practice about how to manage this, so we just do whatever seems best to communicate to other teams (and to ourselves later in time after we've forgotten about the bug) what happened and why.
Bug fix times are probably a pareto distribution. The overwhelming majority will be identifiable within a fixed time box, but not all. So in addition to saying "no bug should take more than 2 days" I would add "if the bug takes more than 2 days, you really need to tell someone, something's going on." And one of the things I work VERY HARD to create is a sense of psychological safety so devs know they're not going to lose their bonus if they randomly picked a bug that was much more wicked than anyone thought.
Wish there were more like you, out there.
Knowing all of those aspects and where an issue lands makes it possible to prioritise it properly, but it also gives the developer the opportunity hone their investigation and debugging skills without the pressure to solve it at the same time. A good write-up is great for knowledge sharing.