Hacker News Reader

Comment by Fr0styMatt88 | Hacker News Reader

The whole technical debt discussion often makes me wonder if there's a fundamental distrust (or even disrespect) of software engineers as domain experts.

Do people in other professions have the same issues? If the salespeople say this thing must be done in this particular way, or if the lawyers say this thing must be done in this particular way, are they questioned and overridden by their managers? I honestly don't know; it feels like software engineers are given a special level of distrust but it just might be that I'm perceiving that because I haven't experienced the other jobs.

FWIW where I currently work, we have an amazing middle-manager / product owner who has to fight the same battles with upper management with regards to process improvement and technical debt. She gets it, but has to fight the same battles we do. Sales and C-level want everything now with as little investment as possible.... I wonder if I just answered my own question?

View on HN

I distrust my fellow software engineers when they say they need time to pay tech debt.

There were many times where I refactored code that never received any additional updates or to have the feature removed. I wasted a lot of time without any real gain for it.

There are real costs to paying off tech debt and it is easy for engineers to spend their time on low value refactoring.

Software engineers are not always good at prioritizing the business value side of their work so they aren't trusted there.

If you can show you can prioritize business value then it is easier to convince management to give you two weeks to clean up tech debt.

> If you can show you can prioritize business value then it is easier to convince management

If your management needs 'convincing' then there's a problem.

Either the manager is technical enough to know whether a particular set of work is the right thing to do or not, in which case they don't need convincing. Or they're not, in which case they should defer to the engineer and also not need convincing.

The only managers who need 'convincing' on technical matters are technically inept individuals who nonetheless attempt to micromanage technical decision making. They are no more able to link business value to technical decisions than the engineers you say you distrust. Fire them.

> Engineers don't generally enjoy paying off technical debt, it's a chore. But they recognise that it must be done sometimes.

I disagree. Most engineers I work with like refactoring. It’s usually well defined work with clear metrics for success. It can also reduce daily pain if it makes delicate/messy code much easier to work with.

The problem with refactoring is that its only value is in improved execution of future projects. So it can be extremely valuable or completely useless. It depends on how likely the refactored things are to be worked on again.

> Software engineers are not always good at prioritizing the business value side of their work so they aren't trusted there.

This.

Software engineers are focused on the implementation side of things and how today's work will impact their work in the future. Yet, sometimes a quick and dirty solution that takes close to nothing to implement is more than enough to put a functionality in production. Sure, the whole thing will need to be scrapped in the near future when rewriting the feature to meet all design requirements, naturally the quick and dirty solution underperforms, and of course implementing it properly will take 3 or 5x the time. Isn't it obvious that the team is wasting it's time by implementing a quick and dirty solution?

Yet, the quick and dirty feature does shorten the time to market to a fraction of what it would otherwise take to implement properly, and perhaps that is good enough for the project for the time being.

It's for this reason that I've often tried to get the business side to give opinions on how shoddy/quickly they'd like the feature to be done. Curiously a lot of the time they appear very reluctant to do so. It's probably CYA kicking in maybe... many managers don't like to say "do a shoddy job" if it ends up biting them in the ass later... they'd often rather just hint and exert unspoken pressure to rush it out and then deny everything later when the CEO starts asking questions a bug that fucked everything up.

There are exceptions, of course, and I'm pretty happy to have worked with them, but in general about 90% of managers I've worked with have resisted my attempts to get them to be explicit about the level of quality that they want.

Quality is so ridiculously subjective that I'm not sure a non-technical manager should approach the question, and a technical manager should refuse a direct answer and steer the conversation towards structure and deliverables.

% of time spent on refactoring is completely objective. Whether that results in quality is another matter, but it's a dial that a product manager has control over that (usually) has direct impact on the quality of the end product.

My take is that product managers care mostly about what functionalities were delivered and if the requirements are met. If the development teams deliver a quick and dirty solution and it meets some requirements then they check the requirement as delivered, and that's it.

When this happens then the devteam's negotiating position changes completely, because they are no longer arguing about how to allocate resources to deliver a functionality and instead they are asking to waste time rewriting something that was was already delivered and is supposed to be up and running.

Do you not consider overall product quality to be a product decision?

> There were many times where I refactored code that never received any additional updates or to have the feature removed.

Since we're throwing out anecdotes, I've lost count of the number of companies I've seen that were essentially zombies due to technical debt. They had aging, rotting codebases that were impossible to add features to at the rate needed to continue making sales. No senior engineers who could improve the situation wanted to set their careers back by working for them.

I think you and grandparent post's author are both right. I've seen both. It depends on the type of engineers that inhabit your team, and therefore are not mutually exclusive.

The main problem is that many developers will increase technical debt when they try to fix it. For example, some will try to fix technical debt by writing a new framework. Then the new framework doesn't handle all edge cases, so you still need the old solution. Now you have two different conflicting styles creating more technical debt. The next developer will come in, see the mess, and promptly start working on a framework to consolidate everything...

So whenever you consider fixing technical debt, ask yourself if the team is skilled enough to actually create a cleaner solution. If they can't then it is just a waste of time. The best predictor of this is that they have created a cleaner solution elsewhere, if they haven't then don't let them work on technical debt unless they have an extremely well thought out plan.

Management's solution is often to hire more software engineers.

Ironically in big corporations I think that this actually plays into their underlying desires - giving a boost to headcount gives them more importance and having more developers means that the leverage each engineer has is commensurately less.

I've worked on more than a few projects which would have proceeded 4x faster with 2 good developers on a clean code base than the 7 they actually had.

Indeed the entire software startup industry is probably reliant upon this dynamic.

Not the software startup industry, unless you're flush in easy VC cash. Most of the startups which are trying to solve real problems that I've known/interviewed for are extremely conservative with hiring, for the very good reason that bad hires are not just a 0 value add, then can be a negative value add.

Uber is one of the outliers in this regard. But if you look at most successful others (Instagram, Whatsapp and a bunch of others I don't want to name since I want to stay anonymous) all had very small teams right until they were acquired.

I meant that the startup industry would be hollowed out if large corps didn't exhibit this dysfunction and thus create space in the market for them, not that startups do it themselves (although occasionally they do too).

Ah, that does makes sense. Apologies for misunderstanding.

My experience is that these companies have the worst kind of technical debt: a prematurely optimised and poorly abstracted codebase, due to non senior engineers refactoring just in case. Non abstracted codebases are boring to work with because the same code is duplicated, but they are not that difficult to change or add feature to. Extreme DRY on the other hand are the kind of codebase everyone is afraid to touch. Sure maybe they had 100% test coverage in their days, too bad adding a new feature with a small model change would break every interwined object factory.

It's not just extremely DRY stuff that can be problematic.

I work in a non-abstracted code base, but all that really means is every time someone wanted to do something they simply wrote their own way of doing it. Needless to say it's suboptimal to work in on the best of days.

People like to deride one way of doing things vs another but in reality pretty much every approach to software development and architecture can be amazing or hellish depending on the aggregate quality of the developers, their management and the lines of communication between them.

TL;DR - Bad code is bad no matter the paradigm and all paradigms can be made awful.

"Continuous attention to technical excellence and good design enhances agility."

Very few people seem to actually believe this in their bones. All talk about being "agile" and "pragmatic" becomes a joke when it's posed as somehow being in contradiction to design and anything that goes deeper than finding a place to insert your new if statement.

It’s because there are literally infinite ways to solve similar problems in software. There is no physical assets to measure the differences making very hard as relatively speaking it’s still a very young domain. I still go back to my freshmen year in college when in my dorm we had a mechanic, chemical and material science engineers. Each of them complained because they were not senior enough they could not get time in the expensive lab to run their end of year projects... it was at that moment they I looked at my crappy dell laptop and realized I had access to the most advanced systems in my field computer science and it was just sitting here in my dorm... I mean a few years later a guy on similar crappy laptop or desktop created Facebook... and yet there are so many options to build Facebook using php, c#, c++, Java, python etc... maybe to you one of those ways is inferior or less good then the other. Maybe using cloud servers or physical would be better?? Maybe a class or a function... css or Scss... the space is infinite

I think it’s about perception. Broadly speaking, professions like Doctors and Lawyers are seen as prestigious and important, and so they are allowed to manage themselves.

I think software engineers tend to overestimate how prestigious and valuable people outside of tech think software engineers are.

I work in computational biology, and even in a fairly tech related field, you wouldn’t believe how many people think software engineering is just glorified IT. I’ve had a lot of conversations where the crux of it is “why should we pay them so much money, it’s the computers that are doing the work. Or, on building a distributed computation engine to run for 2 years straight without failing: “it’s all just grunt work we’ll just get an undergrad to do it”.

> why should we pay them so much money, it’s the computers that are doing the work.

Why should we pay the blacksmith so much money? It's the armor doing all the work. Why should we pay the stone mason so much money? It's the arch that's doing all the work.

Yes that’s essentially the problem. Engineers/developers aren’t seen as fundamentally making anything. They’re just seen as the interface to the computer.

If you cheap out on your smith, good luck coming back without a hole in you.

Doctors operate with enormous amounts of regulation and overhead; I don't think they're allowed to really manage themselves as all.

Organizations like the AMA are themselves run by doctors.

> The whole technical debt discussion often makes me wonder if there's a fundamental distrust (or even disrespect) of software engineers as domain experts.

Yeah.

I think, sadly, this is mostly on software developers though. Look at all the pop culture stereotypes of programmers: whether those stereotypes are true or not, that is what culture thinks of us. Would you take seriously what a big-bang theory character would say? The problem is, I don't see most software engineers valuing social skills to better relate to other professions. Instead, there's a lot of whining about being misunderstood. And we are misunderstood! But man we're not helping it at all.

Contrary to most stereotypes, most adult software engineers are fairly well functioning adults. They may not be the best at public speaking or other social skills, but they're not mostly like characters from the Big Bang Theory.

I agree, but if the split is 90% well functioning, and 10% people with the manners of RMS/Torvalds, well, people are going to notice those 10%. Stereotypes obviously don't need to be real to shape things. My concern more is that our industry needs to stick up for itself and act like professionals, instead of intentionally marginalizing itself and then acting indignant about it.

I don't even think that 10% really needs to change. We need the RMS' and Torvalds of the world. But we also need to, individually, act better than they do and advocate for ourselves as adults.

I constantly see coders deriding any decision that places the business over their own individual interests, and then we get indignant that the decision makers don't listen to us. But if we don't care about their interests, and the collective best interest of the company, why should they care about our interests?

From my experience, managers tend to start to distrusts software engineers after the first complete system rewrite, that was supposed to solve the big problems, actually didn’t solve them and brought new ones instead.

Maybe that’s one more reason why, paraphrasing Spolsky, you don’t do big system rewrites and instead carefully refactor the system bit by bit?

The issue is neither has crossed the bridge. The non-technical manager can’t understand the risks, and the engineers can’t quantity it in the dollars and cents that CFOs understand. Salespeople are good at dollars and cents. So are some lawyers. But they get overruled all the time by execs.

“I don’t care how you do it, increase sales by 20% in Q4” creates the sales and marketing version of tech debt. (Channel stuffing, excess discounting, etc)

I'm glad you've raised this, because it's something I've felt for a long time.

A few friends of mine work in accredited professions, and it's not something they have to deal with all that much, most likely because they are all accredited. They are billed by the hour, and typically if someone isn't happy with the cost the response is "okay, we'll down tools". Inversely, companies that bid on contracts will bid with a fixed cost in mind, and when estimates are taken that go over the requirements things will be fudged to fit the requirement in for that price. This means one of three things: corners will be cut, developers have to work faster, or requirements are cut. In my experience, the last point is a rarity.

I know that we software engineers like to think of ourselves as thought workers, and on the same level as the likes of lawyers and chartered accountants, but the way our industries operate are wildly different. I prefer to compare software engineering to a trade like carpentry, plumbing, etc. We share many similarities, such as easy barrier to entry and working on fixed-cost projects, but unlike trades, we're always willing to negotiate on our next pay day. You can say it's down to a difference in costs, but even on an industrial level where the person doing the sale isn't a trades-person, if a job requires £x and the client can only pay £y, then the requirements change or the job isn't taken on.

People in other professions have similar issues. Some 'smart' folks feel as if they should be able to understand everything. It probably often works for them, after all, how did they become 'smart'? But, it's a fine line between genuine interest and distrust per se. Try to assume good intent, consider it interest and grow your ability to explain. At some point you might have to call it as a poor use of time and move on, but if you can use the time to grow on both sides of the discussion - bonus.

This is what I suspected, with my perception being tainted by the "grass is always greener".

I also can't say we aren't collectively to blame, or at least somewhat - as a bunch we can be poor at political negotiation/skill, lack of standardization, etc. I kick myself every time I miss an estimate that I've given, because I know at some level that has to erode trust. Or when my manager does press for more details as to why something should be done and I can't come up with a really strong justification at that moment, etc.

> if there's a fundamental distrust (or even disrespect) of software engineers as domain experts.

Physicians enjoy a similar problem. Everyone goes to Dr. Google an then brings their misunderstandings to the clinic. Anything goes wrong, it couldn't possibly have been the fact that the patient was a brittle diabetic with hypertension. It must be the care they received in the hospital after they had a heart attack while driving and ran into a tree. Nor is anyone else in the hospital possibly making mistakes. Clearly it's the board-certified physician who doesn't know what they're doing.

Off-topic, but the ‘Dr Google’ thing is interesting. Something that makes it more nuanced in that particular case is that a patient will have better insight into the state of their own body in many ways than a physician will.

I’ve been watching a series called ‘Mystery Diagnosis’ about people with rare diseases and how they were eventually diagnosed. Something you hear most of them say is along the lines of ‘Finally I met this doctor that actually listened to everything I had to say’. It’s a really interesting series — and damn does it remind you how lucky you are to be mostly healthy!

Yes, business stakeholders often mistrust software and IT people, because relative to other industries we do a poor job of meeting our commitments.

McKinsey did a study recently which found that large IT projects ran on average 45 percent over budget and delivered 56 percent less value than expected [1].

An executive is interacting with people from multiple industries or business functions and this is on the high side. It's much worse than a function like accounting, and it's also worse than many industries with bad reputations, like construction. So over time they are going to come to regard estimates from the tech guys with more suspicion than most. To the extent that other industries have this problem, yes you will find similar trust issues.

If you are a PM/product manager then the best way to handle technical debt is to focus on the benefits that come from paying it down. You are not scheduling time for technical debt, you are scheduling time to improve reliability, security, scalability, etc. and from day one you are lobbying for more time to do every feature so that it's done right in the first place. Yes, sometimes one of these things has to go wrong before the business people start listening, but if you were calling it out from day one they will respect your input that much more.

If you are a developer it's important to understand what work has business value and what doesn't, and make or inform decisions appropriately. Frankly there are developers out there whose attitude amounts to "Your job is to pay my salary, my job is to make whatever technical decisions I want without oversight, and if we don't deliver on time/on budget it's not my problem."

Not all tech debt is worth paying down, the decision generally should be made based on the "interest rate" of the debt (how hard is this going to burn us in the long run?) and the term length of the "loan" (will this component be around for one year or ten?).

[1] https://www.mckinsey.com/business-functions/mckinsey-digital...

> McKinsey did a study recently which found that large IT projects ran on average 45 percent over budget and delivered 56 percent less value than expected [1].

Is that fault of IT people or rather did business people over-promised? Because my repeated experience is that we give estimates, they are considered big and shorter estimates are sold. And it has nothing to do with IT people over-promising. I have yet to see a team where it is tech people pushing for shorter deadline and management being like "yeah, it is too risky, lets go with bigger estimate".

There are situations of me making short estimate, but I also make estimates too big in other situations. However, somehow the estimate that makes its way toward contract is always short due to negotiating pressure.

PM are all too often non-technical, it is quite unfair to blame "IT people" on mistakes of general management and project management. Project management blaming the these contracts being basically frauds on anyone else then themselves is literally them blaming scapegoats for their own faults.

This approach fails spectacularly. The benefits of paying down tech debt are often massive but they are extremely diffuse (i.e. they tend to affect everything in a small way) and hard to recognize (good luck proving that feature X that hasn't even been dreamed up yet will take 16 days instead of 25 because of Y refactoring you're doing now).

Every time I've ever seen the approach of "create a business case for each refactoring" put into practice it has made the problem spectacularly worse because the developers are reluctant to manufacture sales pitches for things where the payoff is hard to recognize.

Hell, developers are reluctant to manufacture sales pitches period.

They're typically awful salespeople. If they're good at making business cases, and few are, they're probably not working for you. Making the quality of your product contingent on the skills of your developers at creating sales pitches is a brain dead own goal.

The best approach is to hire good engineers, allocate a rough proportion of their time that proxies the level of quality you want (e.g. 30%) and trust them - let them figure out how to allocate that time. Coz if you can't trust them you're probably fucked anyway.

>If you are a developer it's important to understand what work has business value and what doesn't, and make or inform decisions appropriately. Frankly there are developers out there whose attitude amounts to "Your job is to pay my salary, my job is to make whatever technical decisions I want without oversight, and if we don't deliver on time/on budget it's not my problem."

I can't begin to explain how incredibly damaging this attitude is. It's the philosophy of a micromanager.

Seriously, anybody who wants technical oversight of developer decisions should either become a developer or they should fuck off.

I think one of the problems is that code is a black box to non-programmers. If I tell you that your car needs $2000 worth of work in order to avoid a $10K problem down the line, how do you evaluate that? Keep in mind that as far as you can tell the car is performing exactly as you expect at the moment.

Another problem is that especially in large organizations there are people who are literally trying to scam their way into money. You always have people aggrandizing their work, or belittling others. Especially as companies become profitable, they attract people who are only out for their cut. There are people who do nothing but attend meetings and try to scoop money into their pockets. This is the norm.

Now marry the two issues and you can see the result: someone says they have to do "special work" that will delay the project and that they are actually saving everybody else's asses. Nobody that isn't involved in doing the work can evaluate the claim. Are they just avoiding work/blame? How do you know?

I find that usually in organisations I work with, there is trust up to the D-level manager (i.e. those who directly manager developers), but there is often very little trust from other parts of the organisation (sales, upper management, or sometimes even program managers).

The best way to engender trust is to work as tranparently as possible. In the best of worlds you would deliver something every day and it would be obvious what's in the queue. Things would be in the queue for a very short time (say 2 weeks max). Work like refactoring, writing tests, improving build systems, etc would be just things you do every day and nobody would ever mention it. Of course, not everybody lives with an ideal situation, but I think most teams should strive to move their situation as close to this as they can (there are definite exceptions, though).

In a sense, technical debt is only a big problem for large monolithic codebases. Software that is primarily composed of small manageable components means that technical debt doesn't matter as much.

When each component is roughly one week's worth of code, it becomes easy to delete a component with a model that no longer fits the problem domain, and rewrite a new component with a new more appropriate solution.

"If you can rewrite a piece of code in a week, you can understand it in a week." - Greg Young

https://maximilianhoffmann.com/talks/programming/the-art-of-...

Except when you large monolithic codebase is composed of a whole bunch of screwy distributed components that have been thrown together haphazardly. And, yes, I know you can't have a "monolithic codebase" with "distributed components". It's just that it's actually all the same dish with a different sauce. Distribution of processing doesn't create better code bases -- it's decoupling of the computation that does it. You can decouple code and a very large number of ways. It's also just as easy to write a rat's nest of confusing spaghetti with components that hold each other's state as it is to do it without "components" (I speak from experience on this front, as much as I hate to admit it).

Well if components are holding each others states then that's failing the decoupling test.

But I agree, it's definitely not easy to keep things properly decoupled. Requires some level of discipline or some savvy choices up front in the overall architecture of the whole app..

Non-decoupled components = tech-debt

Speaking collectively, given how much we fight against standardization of our profession, we should be distrusted, IMHO.

We distribute senior titles to people with three years of experience. We task junior developers with setting up and managing critical cloud infrastructure. Management has no idea how it all works.

It's kind of like we're asking for it.

And we often hire people who have no clue how things work even for low-level management positions. Can you imagine in construction, the lowest level manager being someone with no clue about construction but is just supposedly really good at "project management"? No way that would ever happen. The "management has no idea how it all works" issue is real but it's not because the CEO doesn't know how it works. The CEO doesn't know how things work in any industry.

In SD we have people with 3 people under them who are clueless. There is no construction company in the world where a guy who has 3 people under him doesn't understand the actual work being done.

No wonder there is a trust issue when there are such huge knowledge gaps, it's not uncommon that someone's first direct superior already understands very little about what they do.

Although I have a feeling that this issue is not present at "top" companies. I have a suspicion that at a company like Microsoft, you'd have to go quite a few levels up the chain before you get to someone who is technically clueless. I doubt they'd hire some guy who never wrote a line of code and have him lead the Visual Studio team or whatever. This is something shitty companies do, it's not an unavoidable fact of life.

Basically there should never be a situation where someone's job is to write code, but their boss can't understand that code. The knowledge gap between boss and employee is too large. A boss always needs to be able to understand the work their employee is doing. If you're at a higher management level where your direct reports do not write code, then sure, you don't need to understand code.

I think it's a bit different in software because it's fairly easy to change things and update them.

As for the distrust, generally there is can be a lot of distrust in things you don't fully understand. Legal I think is a great example. There are a lot of contracts that screw people over and legalese is so hard to read, I have a hard time trusting what is going on. But one big difference is it is very difficult to go back and change legal terms. I was changing an LLC to a c-corp in order to receive investment and it took 3 months to fix the "legal debt" I had incurred by starting with an LLC with outside parties who had a say in what the new terms were. And it's hard to say whether we did all the right moves in that transition because I'm not a legal expert. Just as some people doubt the decisions I make in software.

I think this is related to how badly things go wrong if there is a fuckup. You really do not want to push the engineers responsible for determining your $500M office building will stay standing.

So what kind of fuckup in software is going to have to happen before this is also true? My fear is that it will happen one day, if not already (see eg- 737 MAX). I'm not looking forward to that day.

The 737 Max was not a software fuckup. The concerns raised by the engineers were overruled by management, and then buried during certification.

> Do people in other professions have the same issues?

There are very few professions that have as much leeway as we do to begin with.

> Do people in other professions have the same issues?

Yes, any domain which doesn't produce a product, conceivable to a layman in one way or another. You need conceivable results to be understood by managers and other people. For example domains producing tangible results are doing much better.

I used to be electronic engineer, and it was way simpler to explain laymen around you that you actually doing something. Blueprints, boards are conceivable more or less. Lines of code are not, especially hard to explain that less lines of code is better, since laymen don't see architecture, they see just text.

There is a widespread misunderstanding of what Technical Debt actually is and why it comes about. It's not about poor coding standards, it's a rational pursuit of commercial advantage through deferral of costs. I've written a few words about it a while ago here: https://datalanguage.com/features/reassessing-technical-debt

It might have something to do with (1) how expensive it is to make software and (2) the fact that software engineers are known to sometimes be purists or aesthetes or susceptible to hype.

A good comparison might be (building) architects: you know that if you let an architect run wild, there's a good chance you'll end up with something that is beautiful but wildly impractical and over budget.

(And to a lesser extent, every expert wants to do a good job, not a good enough job.)

> The whole technical debt discussion often makes me wonder if there's a fundamental distrust (or even disrespect) of software engineers as domain experts.

LOL, seriously? Guess you haven't worked that much with other practitioners, for example lawyers.

There is some distrust always with business relationships, and I would say that it makes sense not to trust everyone 100% all the time. In the opposite, better not to trust everyone fully, even yourself.

Here's what normal people see and hear when you enter the room: Nerd opens mouth: "Blablabla quibberish, you're all idiots, hear me express your problem way overcomplicated. Also, I need more budget and cannot promise to actually deliver a solution anytime soon because of more quibberish."

If you want to be trusted start acting like normal people. Stop talking in technical quibberish. Stop behaving like a nerd. Shape up. Get some social interaction with normal people. Stop complaining all day long and start delivering what was asked on time and on budget.

Reply on news.ycombinator.com