Nobody ever gets credit for fixing problems that never happened (2001) [pdf]

There is a simple but undeniable element of genius to the idea of, instrumenting the number of validations triggered. I would hazard a guess that 90+% of teams would focus on an "order success rate" type of metric and call it a day, instead of a "# of times bad data was handled" metric, which is exactly how you escape this trap of good work going unnoticed.

Thanks for sharing! Shamelessly stealing and applying this concept at work next week!

Well, you could dust off the old email and 'mistakenly' send it again. Though it will seem petty, a small fraction might rethink the past few months.

Performing some heroics once in a while shows someone can be relied on. However, if heroics are the norm, it’s either bad work or bad management, and it should be looked at with some scrutiny.

I also find it more impressive when someone fixes someone else’s problem. I’m not one to shower praise on someone for fixing their own mistake, nor do I expect any praise if I fix my own mistake. I would be apologizing to everyone for screwing up in the first place.

I can this the "tragedy of software development": the pyromaniacs are the firefighters.

Easy - half a$$ your code and when it breaks - swoop in, "fix things" (actually do it right) and play the role of hero! (I've seen so-called "Rock Stars" at places I worked do this over and over)

See also: The Locksmith's Paradox - https://medium.com/@pk.patrick.kelly/the-locksmith-paradox-6...

Life and its rewards aren't perfect. Work with honest, intelligent people; genuinely do your best; your days will be much better and the odds will be with you.

> Don't know how to deal with this paradox.

Think that it is the management's fault if they misjudge you. They have the chance to judge you correctly, yet they misjudge you. Their fault entirely.

One approach to showing value is as follows:

1.) Create a spreadsheet with all of the features of your group's/company's products(s) listed in rows.

2.) Create a column for every team member in the group and highlight the lead developers for each feature.

3.) Then ask each team member to add a checkmark in their column for every feature for which they would be willing to be on hook for 24x7 triage pager duty.

Over the long term - the most valuable contributor(s) on the team will be the one(s) with the most checkmarks next to the features they led development on (i.e. they write understandable code and document well) combined with the most checkmarked rows in their column (i.e. they proactively seek to understand other peoples codebases).

Really good bosses make up for this. They push teamwork and collaboration, but at the same time they know the details of what every individual is doing and how they each contribute to the whole, and they can ~accurately compensate/promote/terminate. This prevents demoralization of team members. Team members psychologically need to be recognized for their individual contribution.

These bosses tend to be competent ICs who became team leads, they are best positioned to judge the ICs they manage because they themselves are masters at the craft.

Modern “agile” systems don’t work well with you. They want you to deliver X points of features in 2 weeks. Sounds like you’d be better off as a consultant? But then you need to do all that biz dev stuff! Or find a great CTO to work for (or be the CTO) who understands. You only need one job.

I think an underrated way to deal with this phenomenon is proper self-marketing. Talk endlessly what you have done to prevent catastrophe. Describe the avoided catastrophes vividly so that people get a clear picture.

Work for yourself. The fewer bugs you create, the faster you iterate. And the faster you iterate, the better your chances of finding product/market fit.

The solution is to be a shameless self-marketer of all the work you do.

Value is in the eye of the beholder is what I discover. Got to play the game.

If you are an IC, you play it up with the right people. Those right people fight for you instead of the other way around. When the “we need to cut the fat” talks come around, suddenly you are not on the chopping block. But that also assumes the people fighting for you are not on the chopping block.

Soft skills across the industry is highly under valued. Knowing when to pick a fight or hold your tongue is also important (ie, reading the room).

I hate it but you got to do what you got to do to get that cheddar

Or worse: People get stuck quite frequently and ask you for help pretty quickly. You get everyone unstuck, but your own work falls behind and when your boss's boss asks for metrics on developers you have few points delivered and few LOC changed. Your boss tries to explain, but your head is the one that rolls next when layoffs happen.

Quit your job and work for yourself.

I've done that as an hourly contractor, I fixed their 6 month problem on my first day, I hoped they would then hire me for more hours on other stuff but they were like nope, that was all we needed. They did tell some other companies that I was good at this kind of stuff but nothing came of it. My first and last day in contracting for small companies.

No one knows your true value but yourself. Move accordingly.

Me neither. And I've had a similar issue with the first case. You do your work and help someone finish theirs, people still think you only did you job (even if people do credit you, you still have normal impact).

The psychology of workplace is quite subtle and almost backward. There's a motto "squeaky wheel gets the grease" .. to the point I'm thinking of trying some games. Like designers who produce crappy options and one good to toy with higher ups thinking they decide. There are some ideas in that vein so you flip the relationship and benefit from it instead of bleeding.

It's mostly a question of finding organizations that recognize and value that sort of contribution. It also helps if you track who you work with and what you help out on every day, so when review time comes along you can list not only your 'regular job' wins, but all the wins you helped teammates get.

That's not true, your value is clear just subtle.

If your solutions tend to be more stable from the get-go, it will get noticed at most places. There are always going to be that odd company where that's not true, but in my experience, simply doing higher quality work gets appreciated.

The problem with doing something right the first time is nobody appreciates how hard it was.

Unfortunately one of the best ways is to toot your own horn. If you work with good folks, it will be appreciated for you to present a tricky problem you solved, or to present a design you put forth that would save everyone time.

Think of it this way: when a principal engineer is hired from outside the company, what is the forecasted value of that engineer? How was that determined, since they have yet to do anything? If Andrej Karpathy joined your AI startup, people won't say this person has unclear value. It's because of the impact they've had previously and what other people have to say about that person. Value is your reputation.

> Don't know how to deal

Good leaders. Or, worse: Consultants. That is, it takes either an outsider or a self-critical leader to affect change? Those doing the planning are always optimistic [0] about their decisions, processes, evaluations, and progress. https://en.wikipedia.org/wiki/Planning_fallacy

[0] not always bad: https://en.wikipedia.org/wiki/Hiding_hand_principle

It's always worth making some effort at promoting your value; no one else is likely to do it for you.

My anecdata is to be vocal about your works, like having short knowledge sharing sessions to show it to your team/manager. Don’t be afraid of little things, your team would learn something from it.

I hate to ask the obvious question, but maybe your job title is wrong?

Play to your leadership. Exceptions apply, but female bosses tend to appreciate averted crises more than male ones.

If your boss is male, let shit fail. You'll get hero points for responding to incidents.

A female boss will quickly suspect incompetence if things keep breaking (i.e. see how far half-assed DIY home repairs get you with your wife before she loses patience with you). Your hero points will come from mitigation.

This paradox comes up a lot in security. IME in this particular field the gender stuff is less relevant since everyone is paranoid. But when we run stuff up to C-levels, it's only the female execs and lawyers that really stop to consider possible issues-- the men just dismiss everything until it happens.

I think there’s a “this is life” aspect and a “this organization is immature aspect”.

A mature organization respects the process that prevents getting stuck for 3 months. But… they may be more stable and less nimble. Boring orgs don’t like heroes.

The problem may be in the relative value contribution. A senior eng that rights a project on 3 days which had been going on for 6 months is incredibly high value/he work.

A Sr eng who spends 6 months coaching a junior engineer has much lower value/hour.

Presumably that's not -all- you do? Or did you sit around for 3 months going quietly mad waiting?

Assuming you are doing other things make sure that is visible.

Also why did your colleague wait 3 months to ask you for help? A couple days would be OK, a month would be crazy. You should poll them and not wait for an interrupt.

Security/insurance paradox: when you do some jobs perfectly, it feels like you're doing nothing. Sadly, the only (ethical) way to counter such narrative is to spend a good amount of your time making reports showing how well everything is working and how it's not something you can just operate autonomously without disaster (to answer the inevitable "well the job is done, what are we paying you for?)

An adjacent problem is how to know that you've really prevented issues/others from getting stuck with whatever you were working on. No system is perfect and engineering is also an art of knowing when to skip which corners. If nobody sees value in your preventive work then maybe there isn't any?

Other problem on similar spectrum, when you have a good performance and your boss notice you, their expectations can increase and increase until you unable to deliver, then suddenly you're bad.

Having a carefree life of no technical debt is not its own reward? Firefighting is stressful :)

I went on holiday for 2 weeks. Came back to find the team had spent a week on a 5 minute task that had been clearly documented. Nothing had moved forward. Spoke to my manager that day and from that day forward it was clear what my value was. Eventually lead the team and helped the team develop processes and initiative so they could be productive in my absence.

Up until that point I knew my value but I don’t think anyone else quite got it.

At least you're likely not a person who works at a freelance agency where you purposefully do poor work or break things, so then you can bill more time "fixing" the problem - until the client gets the wiser at least, or there's another more oblivious client you can grift much more easily who doesn't know better?

The above seems to be how the top billing agencies on Upwork function, with fewer staff you can grift more people because the hours being billed aren't honest, and so you can have more clients and reach the "top" earnings position faster and more easily; with the same practices existing at least on one of the two prior platforms before oDesk and Elance merged.

If they don't pay you to fix those problems then don't fix those problems until they start paying you for it. You'll have to deal with the psychological torture of watching things go horribly wrong. So it's up to you how to deal with it.

Don't help anyone too much until a manager asks you to help. If someone is really pressing help, tell them that you are busy, but they can ask the manager to tell me to help. This way, you always get maximum credit. Every single manager that I have worked for claimed they were "different" and did not suffer from it. Usually, I give them many chances to see my value. After I am overlooked too many times, I revert to this selfish mode explained previously. In all such cases, my recognition from the manager improves. Sigh.

Time.

Your value will emerge with time.

Just keep calm and carry on.

Good and bad times will do their usual sinusoidal dance but your slope will be pointing up if you zoom out.

There will be unfair promotions outside of your team, reductions or freezes in yours (companies have tendency to throw people at problems = inflate dysfunctional parts; if you have well performing team - they won’t let you grow short term; just wait it out, they will eventually).

The educational system i was put through was set up to teach that outcome scales more or less linearly with effort and time. The first lesson after graduation is that this is not so. Increased effort and time most likely primarily yields more effort and time being expected of you with lagging compensation.

Value and opportunity are chaotic processes in effort and time.

All we can do is to try to maintain the levels of workload such that we have clarity of mind to seize opportunities when they reveal themselves. Honest and balanced colleagues help there, but that is ultimately a missions for yourself only.

If the headline were actually true...

Agree with your point, solving problems gets you points, avoiding them doesn't. My cynical view of the headline is, that a lot of people do get credit for solving problems that never really existed, simply either fabricating (intentionally or not) easy to solve problems or vastly overblowing problems they just happen to have a "solution" for.

If ypu successfully fight a fire, you are a hero, if you prevent fires it is just normal and nothing special.

It's possible to phrase the value proposition to align with what you want to deliver.

You can phrase it as "I help teams deliver on time by making sure they don't get stuck." Or "I increase a team's velocity by preventing mistakes that stop work."

It's how you sell yourself and then, how you tell the story of what happened. Being able to tell a compelling story is important.

Your value gets revealed in your absence. To leave and the company tanks- is a expression of the value shadow you throw.

It’s dealt with by knowledgeable managers that recognize and award your skill according.

If you don’t have such a manager, perhaps find them. Or become them.

Your managers need to understand the work that you're doing. When I review someone's code and I see them fixing something potentially catastrophic but that never happened, I definitely remember that and congratulate them. When I see a team member ask a question that makes me or the team rethink implementation strategy. That is very valuable. When a code reviewer finds a potentially critical bug in my code, or just a nuanced bug, I remember and appreciate. Because I fully understand that those are potential p0 or p1 issues.

And great engineers are consistently good at this type of passive, keeping the lights on work, but it does not reflect on their quantifiable work, so orgs do not include this in performance reviews. It is up to your manager to recognize this and advocate for you.

I am dealing with similar. I can provide solutions 10x faster than anyone. Everyone looks up to me, and are dependent on me for solutions to everything.

However it is clear that noone is improving and that the process IS me.

What I've been trying to do is make sure that my scope and role is fully clarified, and any "extra" activity that I perform is documented and flagged. Anything that becomes a "common" activity implies a missing part of the process - be it a role that is missing, or a skillset that is lacking.

Before you start thinking that it is pretentious or self serving, it's perhaps the opposite - you owe it to the process and the team/business/organisation to make them see what you're putting in, else they fail to find the gap.

It's not paid off yet, but hopefully it will yield results within the next 6 months.

Depending on your context, if you are at the top of the foodchain as described here, minus the recognition for it, one option is to simply add grist to the mill. Creative problems you can leave for the team to sweat and for you to come and slamdunk.

1. Donald Rumsfeld was right when he said “A’s hire A’s, B’s hire C’s”. 2. Environments where people can’t tell the difference between 1x and 1000x engineers lead to performative work and arsonist firefighting. 3. When folks in your organization can’t recognize (and reward) high impact, it’s probably time to get out, largely due to the first two points.

The lean mantra "Make work visible" is not only good at an organization level but also as professional feedback level. Although if used openly for promotions it will cease to be a good metric.

Go into consulting.

> When I work there the whole time, and nobody ever gets stuck for 3 months, my value is unclear.

Well, a problem that hasn't happened yet is a risk, right? So you can apply risk assessment math to it:

Value = Estimated risk of the problem occurring * Cost if it happens

I'm currently in an unrelated field, but with a similar tension of unclear value while things work. The way to deal with this has been to keep a tally of checks on important systems, whether it's daily, weekly, monthly, with links to undesired outcomes should the process that has been checked fail. This list of you checking the processes is one way to "prove" your value, at least for ppl who wouldn't necessarily understand naturally.

There's a ton of reactionary legislation on the books that exists pretty much entirely because politicians wanted to be seen doing something. It's mostly crap.

The thing is: it's very easy to play up the likelihood and severity of problems that are entirely imaginary. This can be just a bad habit, but also a deliberate tactic. In either case, a lot of effort, time, money, etc. is wasted.

Waiting for something to actually happen before allocating resources to preventing it is (to some degree) a rational policy.

> Another variation on the problem is the over-allocation of resources to preventing problems which actually happened once, versus those that are more severe but haven't happened yet.

I've heard this called "institutional scarring" in a blog post somewhere. The idea is a small wound can be replaced with tough inflexible tissue. The jist of the blog post was that just because something happened, doesn't mean you have to change things to ensure it never happens again because that can be an over-reaction that really burdens your future. Accept that loss and that it just might happen again, but that may be better than onerously preventing it with certainty.

This is basically how all bureaucracy comes about. In startups everything is so new that problems haven't had time to happen. In big tech, because they have a tremendous knowledge base of previous incidents and the resulting safeguards, every single step seems mired in bureaucracy.

I have worked in finance tech all my career, so I'm not sure how other orgs work, but this is extremely pertinent. Large investment banks react exactly like this.

I spent a miserable year trying to convince people that they were over-reacting to an outage and there was a very simple solution to the exact problem that occurred. But when senior managers see their jobs at risk because of a repeat, they'll mandate that the entire department review their code for similar issues and remediate. They'll also listen to the loudest voices who somehow come up with massively over-engineered solutions.

Another example, we had a password expire which caused an outage on our trading stack. The amount of effort that went into stupidly convoluted hand-crafted solutions ensuring this "didn't happen again" was laughable. And in the end, after more than a year of work, the whole thing was abandoned in favour of a much simpler centralised solution that should have been done from the start.

> Scrum

I always thought this just meant to scrumble every week to get things done with a weekly standup?

I found the idea Scrum being efficient when done right was true at one company I worked at. Everyone was committed to the process, the Scrum team included debt as a priority for 20% of the effort and everyone had a fairly accurate velocity that we could bake in an additional 20% of your work as your own interests, so stakeholder priorities filled that remaining 60%. Then we would pivot in some Sprints if any epic/team goals required a push to get done or other emergencies/bugs that required a change in priority.

> Scrum is a worse version of the planning processes it meant to replace!

You say this as if it was a coincidence.

I'm allowed to say such things. Some of my best friends are scrum masters.

I would call it "the hellhole that corporate performance assessment is turning into". Scrum is just a tool

I've found it works better when you "build up" to Scrum by incrementally adding processes when you identify issues and relaxing the process if things are working well.

Adding a bunch of processes because you want to add processes doesn't provide value.

Agile seems to have ended up as a way for PMs to report upward to senior managers who also need to report upward.

You can see why. These people have to decide what will be worked on out of all the potential things that can be worked on. Someone has to make that decision. Will this feature make us money? What about that bit of work that doesn't add a feature but reduces resource costs? What about tech debt that I'm told is building up and slowing the ability to deliver features?

I'm not a senior manager, but ultimately someone up the chain is responsible for the company surviving and making money and paying our salaries. They are just like you and I, trying to make decisions based on what little information that can glean. So part of that is "what will this cost and how much will it be worth" vs "what will that other thing cost and how much will it be worth".

To that end, they need some way to estimate this. They latched on to agile as it was being promoted by tech as a way to do this. Whose fault is that?

And so with that came all the frequent estimations, are we on track, rituals, etc. Some people don't believe this should naturally follow. I agree. But somehow all those rituals have become part of the cult.

We abondanded scrum. We abandoned refinements and estimation of stories and story points etc. Now we meet with a PM once a month formally and as a team perform a t-shirt size estimate on where we are. Otherwise we update her as frequently as she asks (which isn't often) or we want. This gives power to us but because of that we're conscientious and make sure to inform her timeously when things are looking sketchy or whatever. Yes, we still have to give "estimates", because ultimately, senior management want to make decisions, but it is otherwise quite lightweight.

It is so liberating.

I think that over time people realize who has a fire drill every month, and who quietly just gets things done.

I still don't quite buy that. Surely it's because computers mostly use epoch time for dates, not dd/mm/yy?

Guess we'll find out in 2038.

Depending on which Y2K bug you're referring to, this very much happened and is an easily believable scenario. I don't think this is a comparable situation.

Another example is ozone depletion.

This is a very important case. There was an enormous investment in fixing Y2K, starting long before the day in question (e.g. financial instruments with expiration dates after Y2K had to be fixed before they became due), so on the day there was only a smattering of minor residual bugs.

There were a few jokes about it in the newspapers but by and large the public just ignored it.

I work on climate and hope / hoped that the same would happen, but it's looking like it will soon get everybody's attention regardless.

What, what do you mean nothing happened? I'm in my underground bunker, do you think it would be safe to come out and look around?

You guys are all in your Y2K underground bunkers still, right?

Yeah. I've heard folk who should know better (looking at you, John C. Dvorak) call Y2K a nothingburger and suggest the effort to prevent problems was a total boondoggle.

Disclaimer: I was part of that effort. The funny thing was that I was brought back to a previous client to fix something that my previous efforts had literally caused. Took me 20 minutes to fix it once I saw what the problem was. And then there was a "while you're here, can you take a look at this ..." That lasted about two years until they closed the department and moved it to NY, NY.

At least I got credit in terms of billable hours.

the readiness paradox:

If you are prepared, nothing interesting happens, life moves on, people opened a notebook and pressed a few buttons.

If you are not prepared, the texas grid freezes, people die, people lose their savings, and "nobody could have imagined it would be this bad".

Not "probably". I was involved at ground level fixing code for BP in '98. Can say without a doubt that bad things would have happened. And not because my salary depended on it - there was no shortage of opportunities to work on other things, obviously. It was legitimately an issue that would have crippled the energy industry (major players, and all their many, many dependents). I infer from this experience that the impact would have been the same on many other industries (e.g. finance, resource development) directly and indirectly.

But it is a great example. I still run into people who recall Y2K as an example of much ado about nothing. No, no!! It wasn't an issue for you because many people worked hard to prevent the problems.

Those problems were not terribly complex just pervasive and critical, requiring a high LOE. It wasn't like a huge moon-landing engineering problem to be held up as some accomplishment of humanity, more akin fixing a lot of dumb Challenger o-ring problems before a blow-up.

I know of several other issues like that. I myself made sure the highscore in a video game I was working on would work :-)

The thing that annoys me is that this is best case scenario when it comes to climate change. If we do in fact manage to avoid the apocalypse, all the "climate deniers" are going to feel vindicated.

Wow, really puts things into perspective.

The book "The Black Swan" by Nassim Nicholas Taleb contains this very example. It is a great read on why the improbable events have the biggest impact.

and the related discussion: https://news.ycombinator.com/item?id=32466346

I wonder if this is why some managers seem to go out of their way to generate chaos. I have no patience for it, and judge them harshly for it. They don’t get any credit if my book for “solving” problems they pulled out of their back side or willfully ignored planning for.

Only take. No give. - Dog

You guys get awards?

Reminds me that one time I found 17 databases in our org without backups configured. All I got was a thanks from some devops guy and no manager even heard of the problem...

Award everybody with a bonus that can be deducted with every bug report

Don't take it personally, managers at bigger companies have to evaluate their teams on a bell curve. The whole team can't score a 5 and someone has to be worse than the group.

Your manager probably just doesn’t like you, find a new one if you can, there’s no point swimming against the current - you’ll just get tired and get nowhere

Are these magic rocks for sale?

And people who comment their code. Even if it is just a little. I swear, 90% of my time in coding is just fiddling around with functions to see what they do so that I can actually pull them together in the right way. If there was just a little documentation this would greatly reduce my time. It should also be in every team's and company's best interest because people hours are expensive. It should also be in the best interest of an open source project because more people will be able to build on your stuff. It should also be in your own interest because taking the 30 seconds to a minute to write those lines interrupts you and allows you to rethink and verify your method. So not doing it is only in the interest of moving fast and writing spaghetti code. But you're not actually moving fast. You move fast at that moment, but not in the race.

As a software engineer who pays more attention on security than average, I often feel my work didn't get recognized. For example, if I successfully prevented a supply chain attack, nobody would say thanks to me for a thing that didn't happen, even when they see a competitor product gets attacked. Similarly, most C/C++ programmers do not really care about integer overflow. But I know we are no longer in the world that computer viruses are everywhere(though ransomware are still common). The, who made it better? The people who are not satisfied with stuff that "just works".

This is a great insight, the idea that management has to look at leading indicators for the org itself. Makes me think that beyond line management, a manger’s job is to behave as an investor and steward of the org, instead of getting involved in the details.

Not a management book per se, but Working Backwards popularised an Amazon approach of thinking about the organisation in terms of mechanisms. You might find that useful.

I’ve always wanted to dig into Scarlet Ink, Dave Anderson’s blog, but never have, and I suspect it’ll be quite useful for you too.

> This happens in organizations that don't measure and respect precursors or leading indicators of success/failure

This happens in organizations that become over reliant on metrics and indicators of success/failure instead of recognizing that these are fuzzy concepts and indicators are just that... indications, not answers.

The High Velocity Edge had a lot about this subject.

The most memorable example was at a hospital, nurses would occasionally pick up the wrong vial. They always noticed before injecting, but there were lots of close calls that didn’t get tracked anywhere. Once they started tracking that as a leading indicator, it became pretty obvious that the vials needed a better design to prevent even the possibility of grabbing the wrong one. Also, if the hospital never realized this problem and someone got hurt, they would’ve ended up with a much worse solution like “double check really close”

I beg to differ. If you have a metric on "time since last outage", people will be incited to hide problems.

There is extensive writing on this subject http://rachelbythebay.com/w/2021/06/01/count/

I’ve seen it mean the department head is told to cut staff. If things run smoothly with 100 people, let’s see how runs with 90, then 80, and so on. The preventative measures start taking a backseat as the staffing issues are felt. The cuts keep coming until something falls over and the department head is pushed out after failing to maintain the service.

But then you need upper management that appreciates the "risk potential" of every area, and what is required to keep that risk at bay.

>One could frame the entire US military budget as preventative spend.

A decent percentage of the population thinks the country should reduce military spending, because they are undervaluing the preventative spend.

A better way to think about it is that you can trust the code you test. Y2K was a problem because it affected _everything_ because anywhere a date was used had to be checked. For mortgages, yes, banks did fix that code earlier but they did it because someone noticed a problem in 1980 and fixed that specific problem, not because they transformed how they wrote code and checked all of the millions of lines of other code which nobody had reported problems with.

Roll ahead a couple decades, and now you have even more code with lots of interconnected assumptions and the guy who originally wrote it retired to the beach at Margaritaville with zero interest in going back to Cleveland to talk about date math. If you’re lucky, a pile of cash could change that. If you weren’t, they passed in a boating accident last month.

My first tech job was working for a COBOL vendor in the mid 90s and heard a lot from customers and other people in the industry. Much of it was cosmetic (rolling from 1999 to 1900 on a display, etc.) but a lot of it would have had real consequences: not accepting credit cards with dates in the new millennium, calculating interest incorrectly, sending people incorrect notices or failing to send correct ones, etc. I remember a someone at Mastercard saying they wouldn’t have been able to processed credit card transactions at all after midnight if they hadn’t done several years of careful work.

The two things which helped were that it got enough attention to get the business people to pay for remediation, and there were enough dates in the future that people got wake-up calls in different industries. Mortgages were the earliest but closer in you had things like credit card expiration dates which caused enough problems that people got serious about deploying updates.

These days, I’m wondering how 2038 will go. We have a LOT more devices floating around and there are still plenty of new devices shipping with 32-bit embedded Linux which may never get updated. Hopefully most of that cheap IoT stuff won’t last another decade but I’m kind of something like an automotive company getting publicly outed for skimping on technical debt management and some people getting locked out of their smart houses.

So become a management consultant? ; )

So if you have a headache and you take an aspirin and the headache disappears you're going to conclude it's from the aspirin, right?

spot on. As a general rule of thumb I keep a form of activity/issues log and send a monthly summart to the finance guys and head of dept along with "expected issues". Letting things crash and burn is a thing I learned from one of my managers a while back. It's great for raises, bouses and rep. Takes a good deal of effort not to fix things I know will break though.