Development slowness in big and legacy applications | Hacker News Reader

Development slowness in big and legacy applications

michaelscodingspot.com

110 pointsolvy095 comments

A big pile of all things of kinds of things.

Big Project Complexities - has nothing to do with software business. It's an organization challenge which has been known way before anyone wrote anything remotely complex with software. See planning fallacy, optimism bias, read about how car and structural companies has done it. Look up "GM, Toyota and NUMMI" as an example.

Cross Team Dependencies. Again, this is an organizational challenge. Cognitive loads, poor communication structures based on industrial org designs and army-based leadership with control and command. Hierachies etc. Read Fred Brooks and Mel Conway on these for a more specific software approach.

Meetings, meetings, meetings. Of course there is a "going around it" solution for this. And it's called a manager. No manager should allow any team member to attend meetings if no thourough agenda is available, no actions for attendees, no proposed actions afterwards and someone who controls the meeting. Meetings will then turn into writing instead of verbal.

Programmers: Skip the damned meetings!

Also programmers: Who and why decided that we should be doing this thing this way?!

Or: Why is the other team doing things this way?!

The point is not in not having meetings, but in having useless and inconsequential meetings.

It's much like requiring that any computation should either return a value or produce an effect. Without that, it's just waste.

> but in having useless and inconsequential meetings.

only know to be useless after having had the meeting, and the decisions or discussion were either moot, already known, or completely expected.

So how do you know whether a meeting is required, before organizing and/or attending one?

A few tricks that can catch meeting that a more or less guaranteed to be useless (in IT):

* Does it have less than 10 people attending?

* It there an here an agenda?

* Is there more technical people than managers?

* Do you know why you've been asked to attend?

If you can answer "No" to any of those questions, you're going to have a bad time. Now you can have meetings where you answer yes to all of the above, and it's still useless, but the chances are lower.

I think only the first and last are really important. I've had very effective meetings where I was the only technical person and we didn't have a formal agenda. Often meetings with stakeholders where I wanted to know what I could do for them; what they needed. So I guess that's still an agenda, but more important is that I was in control and the meeting was small.

Being a bystander in a big meeting, that is the worst. If it doesn't affect me and my input is not needed, why am I there?

Don't you mean more than 10?

A meeting with more than 10 people tends to waste more people's time than a meeting with less than 10.

One rubric I use is "is there one or more questions to be decided on in this meeting?" If no, it's a pointless meeting.

The problem is that some people, engineers included, think that way more people should be involved in decisions than necessary.

One structural cause of that is the “nobody owns anything, everything is shared and employees are replaceable cogs” paradigm. Because then it’s suddenly a real problem that people do things a bit differently. You have to not only agree on what to do (the interface) but also the how (implementation). It would be the same problem in a band if you decided to rotate the instruments every time you jam.

I enjoy working at companies where things get written down so you don't have to be at the meeting, unless you specifically want to participate in the real-time discussion.

managers: why is this taking so long? I'm scheduling a daily two hour meeting and inviting everybody so I can yell at you for taking so long!

And said managers refuse to attend the daily standup.

>> No manager should allow any team member to attend meetings...

I say this as a manager: who do you think is scheduling many (or even most) of these meetings?

Indeed. That’s why the role of the scrum master exists – the thinking goes that because managers can’t be trusted to protect the team from unproductivity due to complicated conflicting interests, you need a "neutral party" to do it instead.

Great! Let's schedule a meeting to discuss your thoughts!

There is also a different way around this; Skip all the meeting and ship code into production, and iterate into good solutions to complex problems based on feedback

> Skip all the meeting and ship code into production

This works until you have an outage, disaster, or embarrasment of some kind, at which point everyone is banned from doing that without approval.

This leads to a kind of "trauma driven development"; when you suggest a change, you see a flash of fear in the eyes of people around you. If you punish people for improvements often enough, you don't get any improvements.

Imagine a surgeon would work like this: no prior diagnostics, no procedures to follow, just cut people open and take it from there!

Honestly, you shouldn't deploy into production without testing. Development and deployment needs to be tracked and scheduled. Because worst case, it has to be reversible. If you start deploying directly, without coordination with others (skip all the meetings, right?), you are begging for desaster.

There's always a good case for slowing down and adding more rules. You can even build a whole career out of it, and if you do it right you'll be harder to remove than a tick - simply ensure anything that might threaten you requires your approval and review.

Convincing everyone that a mistake deploying software is equivalent to a botched surgery, though, that's a good one! I've not run into that one before but I imagine you could get a lot of mileage out of that at the right company.

I work in aerospace, last time a company botched software hundreds of people died. Worse than botched surgery.

Others rely on whatever software is produced, co-workers, customers, users. I think those people deserve a good quality product. And that requires at least some rules and procedures. Unless, of course, it is a single dev thing. In that case you won't have middle management and performance reviews neither.

Obviously the degree of caution necessary depends on the nature of the problems you're solving. But your typical consumer app doesn't need the same degree of caution as aerospace, and using a broad brush to say "everyone must be as risk-adverse as the most risk-adverse company" isn't a great approach.

I'd be willing to bet that the most recent mistake during a deployment that an aerospace company made didn't result in many people dying. Maybe the last one you read about in the news, but that biases the selection a bit.

That said, if you're sending people to the moon, and are actually working on the system that sends them there rather than the website, feel free to have a more rigorous testing procedure. But if you aren't, consider maybe you don't need it.

An guess why the fuck ups killong people are so rare?

Not every industry has to follow aerospace, or life science, standards. Every company owes its customers a good product. And with complexity, product, organisation, requirements, that requires coordination, communication and rules.

Ignoring that, cowboy style, not going to meetings, deploying to production directly, not understanding requirements and blaming all of that on management is simply unprofessional.

I'm still not convinced yet that more meetings makes better software, but I appreciate that you have shared your position so passionately and I feel like I learned a lot.

You're lumping a lot of things together here. It's entirely possible to test changes and have reversible deployments with them being scheduled or really coordinated with anyone, and "tracking" deployments can be trivially automated as a log of git refs deployed without any real bureaucracy or human involvement.

ship code into production, and iterate into good solutions to complex problems based on feedback

It makes me sad that this is even considered a serious possibility. It implies that as an industry we have so successfully convinced our customers to accept and pay for junk - as if software being junk is somehow normal or inevitable - that making an effort to competently build a good product doesn't generate enough competitive advantage to be a reliable winning strategy.

Partially, I think, this is due to fast and ubiquotous internet. I remember the days when games were sold on physical media, played single player on a nost of the time offline computer. Same for other software, updates and patches were cumbersome and expensive. Hence, software had to be complete when shipped.

Fast forward, today everything can be patched and fixed OTA. It seems that possibility made it possible to ship pre-Beta stuff. That all those OTA updates are an additional revenue stream didn't help.

I am absolutely certain you're right. The forcing function to achieve a respectable level of quality before you ship isn't there in an always-online world.

The other big factor IMHO is the amount of speculative money that has poured into the industry in recent times. That has allowed a lot of businesses to survive for longer than they naturally could despite shipping junk and that in turn has taught users/customers that expecting better software is unrealistic. Even most unicorn tech businesses (FAANG etc.) still produce a lot of junk and little that is any good - relative to the almost unimaginable scale of resources they have available - because usually they reached that unicorn status by having one or at most a few massive successes that they can defend and that then subsidise all the bad decisions and junk products just as effectively as VC funding rounds.

Couldn't agree more. We can lament all day long so, it won't change, will it?

When it comes to stuff like that, I feel old!

I find it's good to work outside the bubble of modern web/SAAS development from time to time. It doesn't always pay as well but it does maintain a sense of perspective and an awareness of what is possible. Changing cultures is hard but working somewhere with a healthier culture can be easier.

I would argue that the reason companies ship junk is they can’t really afford the costs of slowing down and doing it right.

Even mentioning the true cost would shut down most deals.

That is certainly a popular claim but I question how true it really is. I find modern development processes with this emphasis are often comically inefficient. In particular a lot of people who are used to working under those processes seem to dramatically underestimate how much drag the rushed decisions and accumulating tech debt will add to their entire project and how quickly that drag effect will be felt.

I don’t think it is related to design decisions necessarily — I think it is more related to coding to the most common case when there are many more edge cases exposed when you scale the number of users.

I mostly witness hurried scenarios in software contracting where each contractor tries to do the bare minimum to meet the definition of demoable. They then leave the project and mark it as another success.

Limiting scope at first is reasonable and often necessary. It's the basis of the MVP idea that has stood the test of time.

I wouldn't describe a well-made product with limited scope as junk though. By junk I mean compromising quality and taking on excessive tech debt, not just starting small because you have to start somewhere.

Alternatively, skip all the meetings anyway and then fire your middle manager. Or, if that fails, go work for a company that values your time.

If you skip all the meetings, your middle manager will fire you and you will have to look for another company.

Oh no, what will become of my beautiful performance management reviews

You found a company of more than 5 people without those? Good for you!

Any company of under 200 with a middle management layer who enforces meeting attendence through the threat of firing and executes a formal performance review process should be considered de facto sociopathic imho.

Counter point: Any employee who refuses to go to meetings and coordinate with others is toxic.

Meeting attendence is important, because

- others need input or have relevant information to share

- activities have to be coordinated

- decisions have to be met and shared

Getting the balance right is trickey, just not showing up is simply not acceptable so.

All of those things are better done in writing so that:

- There’s a record of decisions and the thought process behind them

- When asked for input I can take time to consider a response and therefore give a higher quality reply

- I can refer back to conversations or search them when I inevitably forget some detail

- I don’t miss key information because I was out sick or had to take a bathroom break

- There’s a paper trail you can copy/paste when there’s a disagreement in understanding

- In meetings it can be frustrating to try to get a word in: there are often too many people or just one really chatty bastard who won’t stop

- It’s easier to scroll past chatter about weekend plans and sports than it is to sit through it in a meeting

The problem with meeting notes is they only reflect one person’s understanding, and often the person taking notes only has half an understanding anyway. Meeting notes are like JPEG compression set to the lowest possible quality.

I work with a team that spans +9h and +16h from me and it’s great because there are few meetings and lots of written communication. It’s very easy to search past communications to find old materials or decisions, and that’s not just hypothetical: I do it ALL the time.

We’ll use meetings, but for more tactical purposes: troubleshooting a specific problem, or doing a demo of some feature (although even then I would suggest demos are better as a recorded video).

Most objections are really something like “wah, I don’t want to read or write, it’s too haaaard.” Reading and writing were some of the earliest inventions in human civilization and everyone learns them starting in pre-school. It’s a low bar for a grown professional.

Now this is what I would call a professional approach to work.

Agree to disagree maybe. But consider trying an experiment - take notes so anyone who wasn't there for whatever reason, intentional or not, can read them later, and don't threaten to fire anyone or insult them as being toxic who didn't make it. If you end up in a meeting by yourself, just make a decision and write it down. I'd personally at least be very surprised if all of a sudden you were unable to coordinate activities, make decisions, or share information.

May I remind you that you started with

>> Alternatively, skip all the meetings anyway and then fire your middle manager. Or, if that fails, go work for a company that values your time.

Imho you'll find that a middle manager who spends his days in meetings by himself wishing he could fire the "toxic" people who didn't come is non-essential, but I'm happy to leave that journey of discovery as an exercise to the reader.

You uave to be a real pleasure to work with...

Thanks! Your kind words just made my whole day. I appreciate that even when people disagree they can be civil.

We are totally not on the same page, but in all seriousness, I love your sarcasm. So maybe we actually would get along rather well at work!

Not going to meetings and not coordinating with colleagues are two different things.

One of the most damaging things for productivity is not being able to run the stack locally. This generally happens due to laziness and corner cutting.

If developers have to commit and wait for a build + test + deploy pipeline to see if their code works, that should be priority #1 to fix.

> laziness and corner cutting

My experience is that it just isn't prioritized, and you have to do it early to enforce good engineering solutions to keep it this way. It's much easier to keep something locally runnable than try to set it up that way six months into a project.

If you can't run the stack locally, it makes it harder to set up performance and acceptance testing later. Running locally also makes long build times more noticeable and it's easier to address these issues earlier.

Being able to run locally looks low value at first, but if you do it, and sustain it, you get extremely large project-wide benefits later.

> My experience is that it just isn't prioritized

my experience as well and it blows my fucking mind every time I see it. It tells me developers have _no clue_ what it is they're doing that is helping or hindering them.

I took a job at a place that was into "serverless". That's fine, but there was no way to run anything locally due to how the application was built (wrong framework, no framework, etc.) Also, they failed to give each developer their own environment, so we had people editing the same code in the AWS console. I've never seen anything like it. Everywhere else I worked prioritized local first.

"Can you actually run your software on your laptop?" has become one of my interview questions for that reason.

I think a lot of dev shops just genuinely don't realise it can be done.

I've never been in a startup right at the beginning, but I don't understand how a stack would ever be allowed to transition away from being locally-runnable.

As soon as you choose to use any SaaS that doesn’t have a local equivalent, it’s over.

We were fine until we started using Snowflake. The SQL doesn’t quite transpile to Postgres, and anyway we’re using some Python functions where the equivalent SQL would be nightmarishly complicated. I hate that we can’t iterate fast on that part of the stack.

Could we build something that works nearly identically, but locally? Yes, but we’d need to submit some patches to sqlglot, which nobody has time for.

Could we avoid using Snowflake altogether? Only with a lot more work. It solves a few real problems for us.

That's a good question to ask. I've always assumed nobody would ever develop without local. Been lucky I guess.

Just start writing firmware.

This so much.

Current project I work on, everyone looked at me like I was made of cheese when I asked how I spin up the project locally. They only do unit and component tests locally, the rest is assumed to work when deployed.

Another killer is no contribution guide as a living doc for the team. "Code reviews" end up being a weird gate-keeping mechanism to exercise your authority over others by pointing out things they haven't done "correctly" when there is no shared consensus on what is correct and why.

At my job it's a weird situation where it isn't really possible to even compile the code locally so we end up editing locally and then uploading the code to another system to compile it. Then we can remote into a hypervisor to spin up a VM with the code.

So in a way it's "local" because we have private environments to build & test the code (and don't need to go through a build system) but it is still a relatively slow code-compile-test loop.

VSCode remote-ssh (or emacs TRAMP) solves the remote issue.

I have had it were devs could run locally, but their machine was not setup like production and they included some library, dll, or other dependency that was on their laptop into the project and then it blows up in prod....

fun times...

even more fun when that project is a GUI app installed on lots of computers and now you have a new dependency to roll out to every system because it would be too hard for the dev to revert and fix it where the code did not need that dependency...

Even more fun when that dependency comes from licensed software that you only have a limited number of licenses for

With the advent of microservices, it has exacerbated the situation where many dev shops only build, unit test their own little service and call it a day without verifying the end to end functionality -- the "works in my microservice" effect.

You can see the effect in many large SaaS/tech products in the form of latency in the double digits to forever looping redirects of their service to Narnia and back.

> unit test their own little service and call it a day without verifying the end to end functionality -- the "works in my microservice" effect.

You really shouldn't have to do more if the contracts are strong-enough.

If you really are that fragile you have a micro-monolith not micro-services.

If you have to run everything, what is micro? The individual deployable binaries (but their sum is now even bigger...)?!!

With proper instrumentation, the main speed difference between running your binary locally or in the cloud is the additional scheduling delay. You still need to build your code and start it.

Startup time is the main productivity issue in our stack. We load lots of datasets into memory on startup, which can take minutes.

Getting the time from “everyone agrees the change looks good” to “it’s in front of users” down is so satisfying. Work that goes into reducing this tends to have wonderful knock on effects all over, from faster builds, quicker rollbacks, smaller CI bills (yes it’s possible!). The biggest difficulty I’ve seen in practice is going from “somebody hits the button” to “thing happens”.

But even just making sure that “somebody hits the button” is a 1 button thing instead of a 20 button thing can be a major win! Totally valuable work

On the other side of the coin: having good observability into prod (talking mainly about APM stuff and tracing) gives devs way more confidence in the changes they’re making as well. Make it easy to deploy stuff, but also make it easy for systems to log things, and make it easy for developers to poke into those logs (my favorite flavor of this right now is honeycomb, but mainly cuz you can do perf analysis and answer questions from PMs about usage quite easily)

The nice thing about all of this if you’re a mid sized team is that you can totally outsource this with the right kind of person. Somebody who comes in, has the right to write some automation code (maybe it’s Zapier even!), and is not part of office infighting about process. The mandate being “make this pipeline smoother”. Probably among the easier problems to fix compared to deeper business or product problems. And it’s also super visible to other teams (“wow the bug is already fixed?”)

EDIT: one thing I’ve seen done that “never works” (I have not seen everything) is splitting up a repo into smaller repos. If you find yourself thinking “with smaller repos things could be smoother” it’s likely that what you want is not multiple repos but a build tool like Bazel or Buck, that will let you describe multiple projects in one repo, without causing inter-repo friction come review time. Integrating these tools can save you loads of money on CI and enforce decent test suite separation. The unfortunate flip side is that at least Bazel is very idiosyncratic, and most other tools are based on Bazel. I have heard good things about Buck 2 but have yet to see it in prod

At Apple, we did indeed use many small repos. Some were even in different SCM systems! It was indeed a nightmare to integrate with “everything on main is broken, hold off on pulling” being a common multi day experience.

At Google we use Blaze which is Bazel and it works very well, as you say.

At Meta (where I was before Google) we used Buck2 and it was the best. Much faster and more laptop-friendly than Blaze. It was like Christmas when they switched us over from Buck1.

Meta is very very good at “20,000 contributors to this repo”. Google is about 85% as good.

Getting work done (especially on mobile apps that take 7 hours to build from scratch) at that scale without these tools would make me jump off the building.

That's a good summary, I agree with basically everything.

I think a lot of it boils down to strong engineering, which is not getting mauled into submission by product / business. Strong engineering will naturally gravitate to optimizing long processes.

The other big factor which has many consequences is the overall organization culture - are people encouraged to be helpful, reach out to other teams without organizing meetings in the calendar? Is there a blaming/finger pointing happening? Are many disputes largely about ego, dominance, politics?

The "Meetings" section might be controversial, but I agree as well. There's just no getting away from lots of communication/alignment in large organizations, and synchronous/low latency communication should be encouraged. Yes, that means disruptions, but being on the same page/unblocking others is usually more important. The amount of lines of code produced is not the largest cause of productivity loss in the large organizations.

> The "Meetings" section might be controversial

I can't really say I found it all that controversial. If I wanted to be controversial I'd say that most recurring meetings and any meeting with more than 5 people are a waste of time.

I quite often ask someone who messages me out of the blue with a problem to book time in my calendar (which is usually at least 50% free). Because having a specific time to look at something is usually much better than me either dropping what I'm doing to think about something else or forgetting about their message and getting chased by them 3 days later.

> Because having a specific time to look at something is usually much better than me either dropping what I'm doing to think about something else

There's no hard and fast rule, but if you can help them out in 10 minutes, then it's much healthier for the culture/organization to unblock them immediately (within let's say 30 minutes) than ask them to organize a meeting.

If I wanted to be controversial I'd say that most recurring meetings and any meeting with more than 5 people are a waste of time.

You're probably right but it's sad that this is regarded as controversial rather than self-evident. Maybe any meeting with more than 5 people being a waste of time is a stretch and most again would have been a more defensible claim? But the spirit is right. Good meetings happen when there is a good reason to have a meeting. Recurring meetings are often symptoms of other management problems that have better solutions available.

Yes, that means disruptions, but being on the same page/unblocking others is usually more important.

The real problem is often that people are getting blocked so easily in the first place. I'm sure I've seen every excuse imaginable for why organisations need a constant stream of pings via Slack or Teams or whatever they use. There is usually some underlying claim that everyone should respond to those pings quickly and accept the endless random interruptions that cripple their individual productivity because it's somehow better for the team as a whole. But if you make the effort to plan things and coordinate properly to begin with and you allow your people to concentrate and get on with their work then sudden, completely blocking problems should be unusual and your processes can be chosen accordingly.

> But if you make the effort to plan things and coordinate properly to begin with

That assumes something like this is even possible, won't be completely wrong immediately after the plan is finished, and won't take weeks / months to finish. This is what we used to do in waterfall - architects created a detailed specification which was obsolete / wrong / pipe dreams the moment they hit the Send/Publish button.

That's true, it does assume that. I have never seen a project where this common hypothetical actually became reality so until proven otherwise I'm OK with assuming that Bigfoot is a myth.

I have however seen many projects where management did understand their product and their market well enough to plan strategy more than 24 hours in advance. Their developers could then be reasonably confident about what they'd be working on for a while and act accordingly. If your organisation is functional on this basic level then getting the right people together early to gather information and consider alternatives and sketch out a design is useful for the same reasons that getting peer review when the code is done is useful or that having more than one person writing tests or doing QA is useful.

It's far more efficient to identify and correct problems early in software development. If you skip the up-front discussions and planning entirely then you miss several useful opportunities to do that. Typically in that scenario the problems are either identified and corrected during code review at the earliest or - worse - they are not identified and corrected at that stage either because the right people didn't have the right information to see the problem or because they did see it but it was judged too expensive to fix by then - and so the resulting bugs and tech debt make it through.

> Encouraging a culture of helpfulness is, well, helpful. If you incentivize developers to assist each other, they will do so. On the other hand, if you encourage a culture of competitiveness, they might feel less likely to help. They might even sabotage each other’s efforts.

Yes! "Stack ranking" and inter-departmental competition and all these other dog-eat-dog things turn your employees against each other. People will waste unlimited amounts of company money to prevent another team improving their situation.

> One idea to encourage helpfulness is to include it in a company’s mission statement. Or better yet, add helpfulness to every employee’s performance review goals. If money depends on it, you can be sure people will make an effort.

Noooooo! You've gamified a metric! You know what happens when you gamify a metric? You destroy its value as a metric. And encourage people to cheat it, again. You've made money contingent on appearing helpful, not on actually being helpful. This is a great way to get interdepartmental meetings that achieve nothing but that everyone can cite on their "helpfulness score".

>Long build times

This is one of the major ones for me. It's a multiplier for everything else. local builds, CI builds, feedback cycles. If you get your build times down, everything else gets faster.

Migrating to Bazel was a godsend, while Bazel isn't perfect, it's one of the best build systems for large C++ (or mixed) code bases I've seen. buck2 looks promising, too, in this regard.

Having used both I vote for Buck2 being a bit better in dev experience for performance, resource usage, and sensible config language.

I'm intrigued by Buck2, do you have good resources on learning it? Last time I checked it was still a bit early and I had issues finding out how to migrate our Bazel build to Buck 2.

The single phase build is amazing as a concept.

It was an internal (not yet open source) tool when I used it, so the docs and people behind it were much more available to me than it would be now as a public project.

Ah, gotcha. I tried installing it and the first thing it gave me was an error message. I googled it, couldn't find the answer. Got an answer now, it worked for a small toy example at least. But I haven't gotten around to try it out further.

I feel like the dev experience - without internal help for initial setup - isn't quite there yet, especially regarding error messages and documentation.

I stated my carer developing systems in an enterprise setting but have since moved into the wonderful space of companies transitioning from startup to enterprise. So I’ve worked a lot in this space, and while I’m not sure that I necessarily agree that old complex software is unmanageable legacy, there is a lot of truths in this article. In my experience software doesn’t really have to be “legacy” to encounter these issues. In fact a lot of 30+ year old systems are both better and more maintainable than things developed within the past 5 years. Which is probably an interesting case on its own, but let’s keep on topic!

The way I like to handle these systems is to slowly transition things into “microservices”. I put in in quotes because it doesn’t necessarily have to be what many think of when they read the word “microservices”, and in truth they aren’t at all since they often build upon complex shared data models with all the “fun” that comes with that. Eventually I like to build things in to actual microservices, but it’s often a loooong if not impossible struggle to get there within most organisations which view IT as a cost center on par with HR but without the communication charms. Anyway, what I like to do is to separate complex business logic into their own “microservices” so that anything related to X lives in a single smallish services instead of a big complex system. One which is example is how we’ve got a “budget upload and computation”, a ”liquidity (not entirely sure that’s the correct English word)”, and, a “bank account” service all of which used to be part of this huge complex system, but are now separated completely in terms of the responsibility for their particular businesses logic.

Eventually this breaks complex systems into a ton of minor services, which comes with its own set of issues and is obviously against a lot of what you’re taught in academic computer science because often you’ll have similar code in multiple systems, but it’s sooo much more maintainable and it’s so much easier to move fast and implement what the particular teams in an enterprise organisation needs for their particular business logic. In my example, we’ve gone from five teams having to wait literal months for changes to being able to implement them within a week, and often the same day they are requested. We’ve also lowered our incident reports on problems from less than 60% to above 90% each month. I’m not sure why we call it lowered when higher is better, but whatever.

> In fact a lot of 30+ year old systems are both better and more maintainable than things developed within the past 5 years.

Just would be true just through survivorship bias, no? The systems that were already unmaintainable 30 years ago could not be maintained and either fell over or got replaced, while the maintainable systems from 30 years ago are the only ones left.

I don't think I've ever seen a system fall over and be replaced. Systems don't fail and get replaced - companies do. Systems at profitable companies keep on chugging along.

Occasionally someone suggests a major rewrite but that usually fails when it doesn't reach feature parity, nevermind meet the expectations of all the new features which were thrown at it to justify the expenditure.

”Features are often added with minimal effort and without refactoring the system to make the abstractions reflect the new feature set. So these additions, hacks, and patches are added on top of each other until the codebase becomes a big ball of mud… When a developer faces the choice of completing a task in 1 day with a patch versus asking for two additional weeks to refactor, the result will usually be the first. Multiply that by 1000 over five years, and you’ll get a typical enterprise software project.”

Well put!

Good article. I highly recommend the book "Kill it with Fire: Manage Aging Computer Systems (And Future Proof Modern Ones)" by Marianne Bellotti if you've ever had to deal with these kinds of legacy systems.

It all boils down to unmanaged technical debt that can have many forms - code, standards, architecture, documentation, tooling, compliance. There are two major indicators of tech debt - slow development pace and high amount of unexpected bugs/problems. Big projects have big capacities for such debt so it's visibility is also higher. Replacing "legacy" solution with a shiny new thing is declaring bankruptcy and starting fresh - it does not change the cause of poor tech debt management.

One thing that has always puzzled me is how many managers are seemingly less uninterested in low hanging fruit which would speed up development and more interested in creating metrics to measure the slowness.

How do you know it's low-hanging fruit if you can't measure the impact?

Ask your developers who can feel the impact.

How does a manager argue for funding to make improvements without some sort of metric?

You can't just say it's self evident, because the people making these decisions haven't complied code in years.

Low hanging means you dont need budgetary approval.

I find it both puzzling but also sad that we as managers have to argue with somebody for giving us the time and space to make improvements.

Who is that parental figure? Why have we reached this state as an industry? Why do we have to beg for time and space just so we can do our job right?

To me, the problem is in today's management methodologies there are too many people loosely related to software who nevertheless participate in the process of software development.

Scrum master bothering my teams with some rituals that loosely relate to software but are actually tailor fit for a Japanese factory floor from the 80's, and have a weird cult-y smell? Out of the door with scrum.

Product owner wasting the time of my teams with childish poker card games and refinements and sprints and calling the shots on priority, considering they are the least technically qualified in the team? Why do we give the decision what to build to the least knowledgeable person? This is insane! Out of the door with this guy.

Agile coach (yes, that exists) just talking and talking about how this is not the right flavor of scrum? Out with this guy also.

Managers or sales attending the dev meetings trying to pressure them into delivering something faster? Throw them out and very sternly explain to them to stop with that, things will be done when they are done.

But wait, how do developers then know what the business wants, you ask? Well, I am glad you brought it up.

Engineers should decide what's the best way to build something awesome. They can do that by requesting feedback from sales or business representatives, talk to clients, view usage metrics, whatever. Whatever it is, it's their business and responsibility how to do it, what features to develop and how long each feature takes. They should also damn make sure the product is stable, free of bugs and reliable. Nobody else should have a vote in making engineering and build decisions. That's it. End of story. C łevel dudes better take a note of that, cause me and my teams, we are paid to build an awesome product, so leave that to us and don't waste our time trying to interfere.

If anyone who is not an engineer has objections, I don't care. I don't go to your sales meetings (those things are noisy and chaotic anyway) and teach you how to do your sales magic, right? I don't make you play poker cards trying to squeeze out of you how much time till you make a sale, right? And I damn right don't tell you what client to focus on and that each client has to be acquired in a 2 week period, or you somehow failed the sprint. And who can sprint for two weeks, no rest, and sprint again? That's just harmful bollocks. So why should I accept a non engineer having any say of how and what engineers are building?

I am at a fairly high executive position in a scale up, but that has been my approach and line of thinking for all my career. Remove anyone not engineer from having any say on what engineering does and you will build good, stable, reliable and sellable software. Even the guys who pay our salaries, they hired you not to execute their instructions like a trained monkey but to use your skills and expertise to do it yourself. So if you find that some management types are sniffing around you and your teams trying to make you make too many compromises, remind them that road leads to you using only 5% of your potential, and if they don't listen, find another place to work.

We can build good software if we focus on that and push away all constraints trying to make us do otherwise.

Well, let me tell you that you are my first favorited comment not containing technical info since... July 2022.

Also I'd love working with you (programmer with 22 years of experience, 43 y/o).

My question to you is: do you manage to implement the policy you are advocating for?

> Well, let me tell you that you are my first favorited comment not containing technical info since... July 2022.

Thank you! I feel like it had to be said.

> Also I'd love working with you (programmer with 22 years of experience, 43 y/o).

Gladly! Message me to <my-nickname-> at protonmail dot com although I gotta warn you my hiring budgets are frozen atm.

> My question to you is: do you manage to implement the policy you are advocating for?

Yes! I got lucky enough to join a startup early-ish in my career and scale it up over the years. In my position (VP of Eng)I managed to have enough of influence on the the money people to be able to run the engineering org properly and without much interference. I even managed to pull off a 4 day work week for the developer teams with no reduction of salary. People've been happy, rested, our retention is very high for the industry (+4-5yr) and productivity (I hate that word) didn't suffer a single bit. Who knew? (Apart from 100s of studies).

Anyway, all that came with a cost of me not being very popular with the C-suite of the company, but that's life. You can't win all people's hearts, you just gotta try to do what you think is right.

> Thank you! I feel like it had to be said.

Yes. Regularly and repeatedly.

> I gotta warn you my hiring budgets are frozen atm

That's quite fine, I just started a new job beginning of December 2023 and it's going to be a while until I start looking around (likely part-time / weekends but who knows, we might disagree on the promotion when the time comes in 3-6 months).

> all that came with a cost of me not being very popular with the C-suite of the company, but that's life. You can't win all people's hearts, you just gotta try to do what you think is right.

I wonder if "popular with the C-suite" is as good as many people make it out to be. If you achieve results then the fact that you're not singing everyone else's tune should not matter... but I've lived long enough to know that many people will view you as an enemy of some sort; a wildcard that must be tamed and molded and reshaped to be like everyone else. Because frak your personal approach that actually works, right?

Humans' social behavior is both savage and puzzling to me. Appreciation of personal style very rarely exists in work.

Anyhow, huge topic. I'd love to discuss it after I ping you over the email. :)

It all comes back around to having cross-functional teams that can act without dependencies.

Reply on news.ycombinator.com