Hacker News Reader

Comment by bloopernova | Hacker News Reader

We're dealing with accumulated tech debt burying us right now.

For years the team was told to deliver features, that the upgrades to node dependencies weren't of a high enough priority. Now some of the dependencies are so old they're complicating upgrading past node 12.

Yet the cybersecurity people are demanding that vulnerabilities be fixed.

Thankfully we have a stronger product owner now, and spent most of the last 6 months getting things updated. Yet there are still those in senior leadership who openly question why our team hasn't delivered the features they want.

We're working on educating them. And something pithy and catchy is useful, so I'm definitely going to mention this in future: "fixing tech debt has a price, not fixing it has a cost."

It's the kind of thing that if you can get a "cool" or influential senior leader to repeat, the rest will fall in line.

View on HN

Node 12? Holy s** that is only 4 years old. Meanwhile in the real world most applications are running on 10+ year old stacks - Java 8, Python 2, Angular.js etc etc

We just last month upgraded away from a dependency that seems to have been abandoned since about 2008.

> Meanwhile in the real world most applications are running on 10+ year old stacks - Java 8, Python 2, Angular.js etc etc

Source: trust me, bro

The sky is blue.

Source: trust me, bro

The earth is round.

Source: trust me, bro

We retired our last Angular.js app literally last week, and have a project ongoing to replace the final Python 2 backend. And someone seriously suggested in a meeting last week to keep part of it around...

We're running .net 4.5.2

Source: trust me, bro

Many babies were thrown out with .net core though

We develop for systems that at of this time are still stuck on XP/Vista. We'd love to at least be able to progress to 4.8.1 but as of now we can't.

Do you think Oracle is supporting Java 8 until 2030 for funsies?

If you work at tech companies led by tech people who think of their companies as tech companies, yah, you'll probably be on new(ish) versions.

Most software isn't written at companies like that; it's written at boring business companies (like banks and utilities) that are led by business people who say "what do you mean to want to update java? Isn't it working fine? That would cost us a huge amount of money and be super risky and it's not even end-of-life. Go write more Java 8."

> Do you think Oracle is supporting Java 8 until 2030 for funsies?

> "... it's not even end-of-life. Go write more Java 8."

Sounds like Oracle is enabling this response by not EOL'ing it. Just like IE didn't die completely until Microsoft stopped supporting it.

Actually we have a mission critical application running Java 6. 8 remains a dream.

Our bank migrated to Java 17 year ago and plans to upgrade to Java 21 are in motion already.

Of the three, I’m somewhat surprised by myself thinking that Python 2 is actually easiest to upgrade!

> the real world […] Python 2

I think what kills me most is when you encounter these codebases in the real world and you're like "wait … this entire startup is younger than Python 3 significantly", and younger than the point where there was good support from the ecosystem…

Double bonus points these days if younger than the Python 2 deprecation date…

(And yet every security team lists some variant of "keeping stuff updated" in their process docs, somehow.)

Oh gosh darn you kids. During peak Covid I got calls about COBOL code I wrote in 1981 for Ranger Insurance company because it's still used for some reason at Igloo for ice buckets after the companies merged 40 years ago.

And don't get me started on the Cold Fusion code I wrote in 1997 that probably still is buried somewhere in the Epic software that generates mycharts for HIPAA compliant doctor/patient reporting. Last year I was almost arrested for hacking when my real name triggered a security flag that I might be doing something evil when I really just needed a colonoscopy.

Think the internet is forever? Try being scowled at for innocent code you wrote as a kid.

And stay off my lawn...

Tell me more about your lawn

> Yet the cybersecurity people are demanding that vulnerabilities be fixed.

Talk to them. They may have your back. It was always a happy day when the security audit we were going through at that moment wanted us to attest that all our dependencies were sufficiently updated. That meant we had an external mandate to do the work we'd been wanting to do for ages anyway.

When I was a CISO, I would've been glad to "order" one of my colleagues to do that if they asked me to.

> Yet there are still those in senior leadership who openly question why our team hasn't delivered the features they want.

Is this _really_ so hard for stakeholders to understand? I've found some limited success by dual-boating -- upgrading as features are created. "We can't build this feature without an upgrade" carries a lot of weight -- but sometimes you just have to bite the bullet and demand an upgrade.

Are stakeholders completely oblivious to the existential threat to their bonus if their company is hacked and driven to bankruptcy in lawsuits? Sure some bigs can avoid this, but not the mids and smalls.

Is this _really_ so hard for stakeholders to understand?

for decades now, every generation of programmers has been larger than the previous one.

every generation of engineering managers has probably grown in a proportionate way.

so, for decades now, every generation of engineering managers has been learning the same basic "how software differs from manufacturing" lessons over and over again.

(partly because business schools don't seem to have learned this lesson yet even once.)

Wait, is this the "amount of programmers in the industry doubles every ${not many} years, therefore software development is dominated by fresh juniors who know fuck all about anything" argument, but applied to engineering management?

Can't find the sources for the OG argument about programmers, but it does sound plausible.

Sorry but do you have an example of a company that got hacked and driven to bankruptcy by lawsuits?

Most of what people see in open are companies that get hacked, share price dips for a month or two if it is software company, if it is non software company share price does not even dip.

Okta went down quite a lot since 2022 but still far away from bankrupt and latest "hack" did not move it that much really.

Where in my opinion (as a tech person) they should be out of business already even after slightest "hack".

Microsoft Azure latest hack did not really affect their stock...

Code Spaces, for one. Larger companies can survive a hack yes, but many smaller companies get buried. Not always due to lawsuits but due to critical infrastructure failures.

https://en.m.wikipedia.org/wiki/Vastaamo_data_breach

Not sure if the actual cause of bankruptcy was lawsuits, action by authorities, or just loss of business caused by the publicity.

Cool story and nice part that GDPR being toothless is not true:

*In April 2023, Helsinki District Court did sentence the ex-CEO of Vastaamo, Ville Tapio, to a three-month suspended sentence. It found him guilty of a data protection crime mandated in the General Data Protection Regulation (GDPR).*

I was working for a company, where the engineering team was doing this kind of refactoring, test development and what not mainteinance coding with large team for about 2 years. Meanwhile not many new business features were being developed and the competition took over, and eventually everyone got fired. I think the leadership was just too lazy to actually push the development of the product, but let the engineers do this pussy work of endless refactoring was big part of the problem. Not only one of course.

That's exactly it: you need balance. Not so much focus on tech for tech's sake that the competition gets to bury you but also not so little that you end up burying yourself.

When senior leadership is non-technical, they can't tell when non-customer visible changes are useful or they just have mediocre engineering. They also can't tell when their software stack is such a mess it's beyond fixing.

That's the most important job of a CTO. Communicate the cost and implications of any development done by tech team. This could be feature required by product/sales/CEO or internally by tech team.

That's a great point! We don't have commercial competitors because we're an inhouse project, but there's still a balance to be made, you're right.

IMO the best way is just to spend part of your time refactoring and part of your time adding features. It basically solves all the problems. You still deliver features so your company hopefully doesn't go under. And you can reduce technical debt without having to awkwardly justify not delivering features.

In my experience in the medium term (~2-5 years) it doesn't even slow down feature delivery since high technical debt can have an absolutely catastrophic effect on development speed.

At the previous company I worked in I'd say we delivered work at about 50% of the potential rate because the infrastructure they'd developed was so janky. A big Python/YAML/Makefile mess with zero static types. CI time was over 2 hours and used hundreds of CPU-hours, even for fixing a typo.

> It's the kind of thing that if you can get a "cool" or influential senior leader to repeat, the rest will fall in line.

sympathetic-lol at the desperate copium.

If I am honest with myself, it's about 40% copium and 60% experience with the directors in question.

There's 2 "cool" directors who were tech people but are also influential in that peer group. You can't trust them to represent you fairly, but you can trust that they'll listen to your argument.

The 40% copium stems from knowing that some of the "only features have business value" group are stubborn and think everyone should just work more.

> why our team hasn't delivered the features they want.

Good work takes time.

But I think it is an optimization problem. Rather, that different people are optimizing different things. Your team is optimizing long term profits. The senior leadership who's frustrated is optimizing short term profits. I think one way to handle these situations is to clarify which is the more important optimization problem. That makes points like "opportunity costs vs marginal cost" discussions. The same way we turn down immediate profits to go to school, because in the long run we make more money (and for other reasons). But our systems often push us to focus on short term goals because these are far easier to measure.

I agree clarity is needed but i would caution that it is not ethical to let executives that get to make that determination.

I don't think we as engineers can ethically ever let management optimize our work for the short term. They will ALWAYS want the short term improvement and by even giving them that decision we abdicate our own judgement. That abdication is totally unethical.

It’s okay we can’t get past Node 8

Ik going to switch off our last node 6 instances in a few weeks time.

The "debt" metaphor is a good one already though, and one that senior leaders should well understand.

A bottom up approach to tech this way obviously does not work long term. Startups will eat your lunch.

Node 12? That's cute.

We have an application that runs on Java 6.

The issue isn't tech debt, its tech interest.

Looking at the historical maintenance cost of the stacks when choosing them and prioritizing ones with a better history of backwards and forwards compatibility in their ecosystem does far more to prevent tech debt pile up than trying to reach a strict habit of bending over backwards to update dependencies.

Reply on news.ycombinator.com