After every single project, the org comes together to do a retrospective and ask "What can devs do differently next time to keep this from happening again". People leading the project take no action items, management doesn't hold themselves accountable at all, nor product for late changing requirements. And so, the cycle repeats next time.
I led and effort one time, after a big bug made it to production after one of those crunches that painted the picture of the root cause being a huge complicated project being handed off to offshore junior devs with no supervision, and then the junior devs managing it being completely switched twice in the 8 month project with no handover, nor introspection by leadership. My manager's manager killed the document and wouldn't allow publication until I removed any action items that would constrain management.
And thus, the cycle continues to repeat, balanced on the backs of developers.
Thats what we call blameless culture lol
Why let their own credibility get dragged down for a second time, third time, fourth time, etc…?
The first time is understandable but not afterwards.
But I don’t think a self respecting person would do that over and over.
People will do crazy things for just $100. Including literally get fucked in the ass by a stranger.
7 figures? Ho boy. They’ll use way fancier words though for that.
A man approaches a girl and asks, "Would you sleep with me for $1 million?”
She responds, “Yes, of course!”
Excited, he then asks, “What about for $1?”
She indignantly replies, “Who do you think I am?”
To which he responds, “We already established who you are; now we’re just discussing the price.”
I think it fits there. There is surprising amount of people believing that they have some Values, but just to a point when they were offered to sell them for a right price.
Cy Porter's home inspection videos... jeez. How these "builders" are still in business is mind-blowing to me (as a German). Here? Some of that shit he shows would lead to criminal charges for fraud.
I'll also say the obvious here in Sinclair's quote about salaries: you can indeed pay for someone's self respect.
(Thus commanding a rate similar to a more competent person who doesn’t package it to sell.)
Shit rolls downhill and there's a lot more fuss when an engineer calls out risks, piss-poor planning, etc. than any actual introspection on why the risks weren't caught sooner or why the planning was piss-poor.
Can we also address the fact that “software spend” is distributed disproportionately to management at all levels and people who actually write the software are nickel and dimed. You’d save billions in spend and boost productivity massively if the management is bare bones and is held accountable like the rest of the folks.
(as an aside, this contrasts diametrically with Amazon, where i worked for a year for healthcare, not needing to because of Apple years' savings, but after a genomics startup i had joined ran out of funding, and wanting a new challenge; there skilled engineering types were presumed to be fungible assets for (not kidding) at least 7 layers of do-nothing bureaucrats making huge salaries...they could survive because sales on the amazon store extract something like the 30% royalty to amazon)
TL;DR your realistic options are snake oil that doesn’t work, or nothing.
Keep that in mind next time anyone’s talking about managing through metrics & data or whatever bullshit. All that stuff’s kayfabe, companies mostly run on vibes outside a very-few things.
That said, I think I would agree with your main concern, there. If they question is "why did the devs make it so that project management didn't work?" Seems silly not to acknowledge why/how project management should have seen the evidence earlier.
170 years ago is 1855.
Operation Just Cause? Desert Storm?
And, depending on how you look at it, the US won the war in Afghanistan and Irak, but lost the peace afterwards.
One minor win, every major operation being a loss doesn't change the conclusion though imo.
I think it's also instructive to look at each of these operations and note that the two that were won were small, had clear objectives, and were executed quickly to meet those objectives and had no scope creep.
I would agree that the US is notably terrible at occupations and getting involved in civil wars, at least since WWII, but Desert Storm was pretty much an unqualified slam-dunk take-a-victory-lap success against one of the top armies in the world that wasn’t an ally or a nuclear state—carried out on the other side of the planet from the US, to boot. Like I think Iraq was ranked top-10 at the time by many ways of reckoning military strength, and that wasn’t enough to effectively resist the US effort at all, really.
If that war seems small, it’s only possible for it to seem that way from the victor’s perspective, and only because we did such an amazingly good job of totally destroying Iraq’s substantial capacity to fight in a matter of weeks. In terms of deployed and engaged men and materiel it was really big, just fast because it was so very one-sided, and “cheap” in terms of casualties on the US side for the same reason.
I consider Desert Storm an unqualified victory in an engagement that is in the same conversation as a Vietnam or Korean war, but still not quite to a WW level in scope or complexity.
I think the reasons this hasn't happened is (a) tech has moved too fast for anyone to actually be able to credibly say how things should be done for longer than a year or two, and (b) attempts at professional organizations borrowed too much from slower-moving physical engineering and so didn't adapt to (a). But I do think it can be done and would benefit the industry greatly (at the cost of slowing things down in the short term). It requires a very 'agile' sense of standards, though.. If standards mean imposing big constraints on development, nobody will pay attention to them.
For instance by and large the role of organizing to not to get more money but rather to reduce indignities... Wasted work, lack of forethought, bad management, arbitrary layoffs, etc. So it is much more about governing management with good practices than about keeping wages up; at least for now wages are generally high anyway.
there are also reasons to dedend jobs/wages in the face of e.g. outsourcing... But it's almost like a separate problem. Maybe there needs to be both a union and a uncoupled professional standard or something?
>the role of organizing to not to get more money but rather to reduce indignities
agreed. And I think that's why it's going to really start taking hold as we enter year 4 of mass layoffs in the US (because outsourcing). Alongside overwork from the "survivors" and abusive PIPs to keep people on edge.
A lot of the layoffs appear to be about conserving cash for investment in AI. In many cases the jobs that are cut are not backfilled by workers in the US or abroad.
What objective measures would you use?
Oh, you meant innovative for the shareholders. Got it.
But regardless. Not like USA companies are open to EU citizens currently. Whether it's politics, compliance / legal, or tax-related, it does not matter much. Most US job ads I've seen lately are making it super clear that they want only US citizens.
Though I think Gen Z in general will be making waves in the coming years. They can't even get a foot in the door, so why should they care about "high salaries"?
This was even more apparent before the ACA passed, when insurance providers were not prevented from denying or charging more for customers with pre-existing conditions. Hate your job but have cancer? Better not leave, you might not even be able to get insured at your next job.
If you see an engineer is out of his depth then change his position, no need to fire them since like others have pointed out, that can have severe consequences in their personal lives and most of the time they can still be useful if they get more adequate work.
I also think people here misunderstand what unions do. Unions are inherently conservative (small c) institutions that aim to protect the status quo. Improving processes is not a fundamental goal to unions. We saw this with the ILA that fought to essentially ban automation in the ports that would drastically increase efficiency because of the belief that this would reduce union jobs. It's foolish to think software unions wouldn't end up becoming like this.
In MANY other countries there is already WAY more regulations regarding layoffs and firing employees that has nothing to do with unions.
In Germany there is a probationary period in which you can just fire somebody for no reason basically. That time can be like half a year (in my case) and in most cases it becomes clear if the new hire fits your team or doesn't.
All unrelated to unions though. The big unions in Germany for example have a lot of power and if you are just a simple welder for example you'd have no chance getting anything done without a union.
When your scope is Europe ... The US is not the exception in the world, it's Europe which is.
The US has a dynamic job market where it's easy to lose your job, but easy to find another one. In Europe, and that's true for most EU countries, it's really hard to lose your job, but it's also really hard to get one for the very reason it's hard to get fired - and when you get a job, you will have to compromise on compensation and other benefits. It's not black and white here. While the European market is appealing to some people, the US market is preferable to others.
I agree with that, it's a very individual topic. I'd say for high paying "high performance" jobs the US model definitely has an advantage but for low-wage jobs it's quite the opposite.
But that made me curious, and answering my own question, it's this guy https://en.wikipedia.org/wiki/Peter_Hummelgaard who is indeed a Social Democrat .. So much about workers rights, funny ...
Re-reading your comment again, I'm not sure that I understand it: "Let's see whether Denmark remains competitive." What do you mean?
For software that's actually safety critical, like avionics, there are already sufficient regulatory controls.
But I totally agree, I think people are too compliant and fear banding together to have influence on higher ups. I'd argue in most places the engineers have far more power than they realize since they are in high demand due to shortages of qualified personnel. (at least in many countries in Europe)
There are tons of factors in play though that I believe contribute to this like some employees being friends with their higher ups not wanting to hurt their careers, not wanting the tough discussions, the repercussions if management says "no" etc..
"Just". Come on, man, you know better than that. I too like my sci-fi to be over the top unrealistic.
Truth is, nobody ever thinks about their rights before it's too late. The paycheck shows up on time every month and people just don't want to rock the boat. Not to mention all the opportunists that will tell on you immediately to gain the favor of the upper class.
These things are well-known and apparently nothing ever changes before the guillotines start working. I don't think anything will ever improve in our area. Nobody is bothered. The people who are have zero power. And so it goes.
I honestly think it's just people who don't mind these problems and are fine working under those conditions, or they just quit once they are too annoyed. Change is way harder just than just leaving, I think that's also part of the reason why this keeps going on.
It's extremely sad. I did not want to live in such times but oh well, good thing anyone asked us, right?
>a sophisticated attempt at building a professional organization that can spread simple standards which organizations can clearly measure themselves against.
We have that as a form of IEEE, but it really doesn't come up much if you're not already neck deep in the organization.
That's maybe in Europe. Plenty of US developers those days have a litany of ~1-2 year stints at FAANGs and startups du jour in their CV.
The little people are going to do what they need to do to survive. If these multi-billion or even multi-trillion dollar companies feel some type of way about it, well, they're the ones with all the power, not us. They're more than welcome to change the culture at any time.
> If standards mean imposing big constraints on development, nobody will pay attention to them.
Unless there are penalties for not doing so.
> tech has moved too fast for anyone to actually be able to credibly say how things should be done for longer than a year or two
But that's just it. If things are moving so fast that you can't say how things should be done, then that tells you that the first thing that should be done is to slow everything way down.
I think introducing more competition at higher levels may be better than eliminating it below. This should be happening because I’m pretty sure most PMs could be replaced by an LLM.
Most PMs I've ever met had zero clue what they are doing. And no it's not only N=1 sample, same anecdote is heard from many, many other people.
But sadly enough, undeniable human incompetence there is not even the biggest problem. The "we will never make more reasonable deadline" is.
Most managers, not just PMs, are an objective net negative. As any ruling class always does, they get complacent and think that just putting their foot down is going to magically change reality.
Most people say that because they have no idea what a PM's job is and are upset that the PM isn't doing what they want.
Fact on the ground is that they usually optimize the entirely wrong indicators and never ever optimize for avoiding future problems.
In my consulting and contracting stints I made it a habit to write down what crisis is looming on the horizon and making bets with my wife which one is going to arrive first. A fun little game.
Watching train wrecks in slow motion stopped being fun with time.
What PM's job is is above my pay grade. I want them to enable me and the team, not be a mouth piece of people with zero understanding of product _and_ of engineering. They are just there to ask you how is stuff going.
Fact on the ground is they don't optimize for _your_ indicators and have to compromise on what problems can be addressed now or later. They only appear to be optimizing to the wrong indicators because you don't have all the information.
Your balanced take is nice and I wish I inhabited that reality. So far I have not. Maybe it's the region market or maybe my marketing sucked.
If you talk about an upcoming shitshow you are generally seen as not entrepreneurial, if you mitigate a risk before it becomes a crisis that crisis is not viewed as crisis. However, if you let a crisis happen and "successfully" manage it you are seen as a hero.
9/10 PM's work is literally avoiding any decisions and responsibilities whatsoever.
Concerned dev/ops/po: Dear PM, feature FOO-123 will not be merged before we start activities for Gate 5.2. FOO-123 is required for Gate 6.0 and if we start with Gate 5.2 activities without it, Gate 6.0 will be delayed by at least 5 weeks. Team FOO is projecting verification of FOO-123 being done by the end of week, which would delay Gate 5.2 by a week. Shall we delay our activities until FOO-123 is merged or start regardless?
PM: Gate 5.2 is extremely important for the project timeline and no delays are acceptable.
<- few moments later ->
PM: I was informed by Compliance that FOO-123 is mandatory, does that affect timelines for Gate 6.0?
Disgruntled employee: Either we start over Gate 5.2 activities with FOO-123 included, which would delay Gate 6.0 by at least 9 weeks, or you get team FOO to backport FOO-123.
<- few moments later ->
PM: gets promoted for successful handling of stressful situation with FOO-123, limiting project delay to 15 weeks and only overrunning projected costs of Gate 6.0 by 30%.
Since successful businesses continue to employ them, I'm going to err on the side of they do serve a function I don't always understand rather than they must be stupid.
On a less snarky note: Depending on organization type PMs might do a ton of paperwork on behalf of the teams and act as some sort of central point for business-product communication, however that has exactly nothing with management in any typical definition.
And yes most PMs are useless, regardless of how snarkily you attempt to push that element aside.
Executives are generally incompetent everywhere and of course they'll introduce a reality distortion field where none of them are ever accountable. That should be obvious to anyone. Question is why do all working people keep allowing it to happen. But the answer to that is also known and quite depressing, too.
Waste of my bloody time. Project completed, taking twice as many devs for twice as long, great success, PM promoted. Doesn’t do that basic thing that was the entire point of it. Nobody has ever cared.
Edit to explain why I care: there was a very nice third party utility/helper for our users. We built our own version because “only we can do amazing direct integration with the actual service, which will make it far more useful”. Now we have to support our worse in-house tool, but we never did any amazing direct integration and I guarantee we never will.
This is why software projects fail. We lowly developers always take the blame and management skates. The lack of accountability among decision makers is why things like the UK Post Office scandals happen.
Heads need to be put on pikes. Start with John Roberts, Adam Crozier, Moya Greene, and Paula Vennells.
This came about because our work isn’t too diverse but the requirements are wildly diverse and many of the customers have no idea how to achieve the proper level of readiness. I do management in an enterprise API project for a large organization.
People who desire infinite power only want it because it gives them the power to avoid consequences, not because they want both the power and the consequences.
The people who believe that with great power comes great consequences are exactly the people who don't want great power because they don't want the weight of those consequences. The only people who see that bargain and think "sign me up!" are the ones who intend to drop the consequences on the floor.
This leads to higher and higher towers of abstraction that eat up resources while providing little more functionality than if it was solved lower down. This has been further enabled by a long history of rapidly increasing compute capability and vastly increasing memory and storage sizes. Because they are only interacting with these older parts of their systems at the interface level they often don't know that problems were solved years prior, or are capable of being solved efficiently.
I'm starting to see ideas that will probably form into entire pieces of software "written" on top of AI models as the new floor. Where the model basically handles all of the mainline computation, control flow, and business logic. What would have required a dozen Mhz and 4MB of RAM to run now requires TFlops and Gigabytes -- and being built from a fresh start again will fail to learn from any of the lessons learned when it was done 30 years ago and 30 layers down.
To do a new job, build afresh rather than complicate old programs by adding new "features".
[1] http://antipatterns.com/lavaflow.htm
I've been managing, designing, building and implementing ERP type software for a long time and in my opinion the issue is typically not the software or tools.
The primary issue I see is lack of qualified people managing large/complex projects because it's a rare skill. To be successful requires lots of experience and the right personality (i.e. low ego, not a person that just enjoys being in charge but rather a problem solver that is constantly seeking a better understanding).
People without the proper experience won't see the landscape in front of them. They will see a nice little walking trail over some hilly terrain that extends for about a few miles.
In reality, it's more like the Fellowship of the Rings trying to make it to Mt Doom, but that realization happens slowly.
And boy to the people making the decisions NOT want to hear that. You'll be dismissed as a naysayer being overly conservative. If you're in a position where your words have credibility in the org, then you'll constantly be asked "what can we do to make this NOT a quest to the top of Mt Doom?" when the answer is almost always "very little".
Wrote to my management: "It is, by all means, great when a navigator is able to take over an incapacitated pilot and make an emergency landing, thus averting the fatality. But the conclusion shouldn't be that navigators expected to perform more landings or continue to be backup pilots. Neither it should be that we completely retrain navigators as pilots and vice versa. But if navigators are assigned some extra responsibility, it should be formally acknowledged by giving them appropriate training, tools and recognition. Otherwise many written-off airplanes and hospitalized personnel would ensue."
For all I know the only thing this writing might have contributed to was increased resentment by management.
You are 100% correct. The way I've tried to manage that is to provide info while not appearing to be the naysayer by giving some options. It makes it seem like I'm on board with crazy-ass plan and just trying to find a way to make it successful, like this:
"Ok, there are a few ways we could handle this:
Option 1 is to do ABC first which will take X amount of time and you get some value soon, then come back and do DEF later
Option 2 is to do ABC+DEF at the same time but it's much tougher and slower"
Working teams are good for a project only, then they are destroyed.
When I was in grad school ages ago, my advisor told me to spend a week reading the source code of the system we were working with (TinyOS), and come back to him when I thought I understood enough to make changes and improvements. I also had a copy of the Linux Core Kernel with Commentary that I perused from time to time.
Being able to dive into an unknown codebase and make sense of where the pieces are put together is a very useful skill that too many people just don't have.
It's more about being good at juggling 1000 balls at the same time. It's 99.9% of the time a management problem, not a software problem.
Large projects I've worked on failed simply because nobody wanted the solution in the first place.
In government I've seen many millions spent on projects that were either forgotten about or the politician that requested it lost office.
The worst was a custom-built copy protection scheme that was built by a former (?) software cracker who designed it to be difficult for someone of his skills to reverse engineer. It took me a week tracing through the code to understand what it was actually doing so I could extend it to add more options.
Because such failures are so common management typically isn’t punished when they do so it’s hard to keep interests inline. And because many producers are run on a cost plus basis there can be a perverse incentive to do a bad job, or at least avoid doing a good one.
Suffice to say, projects are significantly more likely to succeed when the power in the project is held by people who are competent /and/ understand the systems they are working with /and/ understand the problem domain you are developing a solution in. Whether or not they have a title like "engineer" or have a technical degree, or whatever other hallmark you might choose is largely irrelevant. What matters is competency and understanding, and then ultimately accountability.
Most large projects I've been a part of or near lacked all three of these things, and thus were fundamentally doomed to failure before they ever began. The people in power lacked competency and understanding, the entire project team and the people in power lacked accountability, and competency was unevenly distributed amongst the project team.
It may feel pithy, but it really is true that in many large projects the fundamental issue that leads to failure is that the decision makers don't know what they're doing and most of the implementers are incompetent. We can always root cause further to identify the incentive structures in society, and particularly in public/government projects that lead to this being true, but the fact remains at the project level this is the largest problem in my observation.
This statement is terribly incongruent.
In what way? A condition being required but not sufficient for success is a perfectly logical statement.
Do your two decades of experience cover both sides?
Yes.
I appreciate both sides and have a wealth of experience in both. The challenge is that all the non-technical problems cannot be solved successfully while lacking a technical understanding. Projects generally don't fail for technical reasons, they fail because they were not set up for success, and that starts with having a clear understanding of requirements, feasibility, and a solid understanding of both the current state and the path to reach your desired outcomes, both politically/financially and technically.
I was an engineer for more than a decade, I've been in Product for nearly a decade, and I'm now a senior manager in Product. I can honestly say that I have the necessary experience to hold strong opinions here and to be correct in those opinions.
You need technical people who can also handle some of the non-technical aspects of a project with the reins of power if you want the project to succeed, otherwise it is doomed by the lack of understanding and competency of those in charge.
Do you mean the problem of wanting to build something without knowing how/having the skills, to build something?
I do not think it is the only reason. The world is complex, but I do think it factors into why software is not treated like other engineering fields.
If we took the same approach to other engineering, we'd be constantly tearing down houses and rebuilding them just because we have better nails now. It sure would keep a lot of builders employed though.
This is almost exactly what happens in some countries.
If the residents die and someone new purchases the land, the old house is (generally) torn down and a new one built.
On the other hand Microsoft and taceboook did collude to keep salaries low. So who knows.
It was more tech companies in collusion than many people realize. 1) Apple and Google, (2) Apple and Adobe, (3) Apple and Pixar, (4) Google and Intel, (5) Google and Intuit, and (6) Lucasfilm and Pixar.
It was settled out of court. One of the plaintiffs was very vocal that the settlement was a travesty of justice. The companies paid less in the settlement than the amount they saved by colluding to keep wages down.
https://www.mercurynews.com/2014/06/19/judge-questions-settl...
There's also the complexity gap. I don't think giving someone access to the Internet Explorer codebase is necessarily going to help them build a better browser. With millions of moving parts it's impossible to tell what is essential, superfluous, high quality, low quality. Fully understanding that prior art would be a years long endeavor, with many insights no doubt, but dubious.
> While hardware folks study and learn from the successes and failures of past hardware, software folks do not. People do not regularly pull apart old systems for learning.
For most IT projects, software folks generally can NOT "pull apart" old systems, even if they wanted to.
> Typically, software folks build new and every generation of software developers must relearn the same problems.
Project management has gotten way better today than it was 20 years, so there is definitely some learnings that have been passed on.
Once you've worked in both hardware and software engineering you quickly realize that they only superficially similar. Software is fundamentally philosophy, not physics.
Hardware is constrained by real world limitations. Software isn't except in the most extreme cases. Result is that there is not a 'right' way to do any one thing that everyone can converge on. The first airplane wing looks a whole lot like a wing made today, not because the people that designed it are "real engineers" or any such BS, but because that's what nature allows you to do.
And yet we scale the shit out of it, shifting limitations further and further. On that scale different problems emerge and there is no single person or even single team that could comprehend this complexity in isolation. You start to encounter problems that have never been solved before.
> In distributed systems there is no real shared state (imagine one machine in the USA another in Sweden) where is the shared state? In the middle of the Atlantic? - shared state breaks laws of physics. State changes are propagated at the speed of light - we always know how things were at a remote site not how they are now. What we know is what they last told us. If you make a software abstraction that ignores this fact you’ll be in trouble.[2]
[1]: “The Mess We’re In”, 2014 https://www.youtube.com/watch?v=lKXe3HUG2l4
You can horizontally scale cargo transport by running more cargo planes until you're constrained in some way.
> Result is that there is not a 'right' way to do any one thing that everyone can converge on.
Are you trying to say there is in hardware? That must be why we have exactly one branch predictor design, lol
> The first airplane wing looks a whole lot like a wing made today, not because the people that designed it are "real engineers" or any such BS, but because that's what nature allows you to do.
"The first function call looks a whole lot like a function call today..."
I'll be that 'well akshually' guy. IIRC the AMD and intel implementations are different enough that spectre/meltdown exploits were slightly different on each manufacturers.
Source: wrote exploits.
Only superficially. What's actually happening varies radically by language. See for instance tail call optimization.
Seems more accurate to say they are obsessed with copying "what sounds good". Software industry doesn't seem to copy what works, rather what sounds like it'd work, or what sounds cool.
If they copied what works software would just be faster by default, because very often big established tools are replaced by something that offers similar featurage, but offers it at a higher FPS.
I know a lot of people on here will disagree with me saying this but this is exactly how you get an ecosystem like javascript being as fragmented, insecure, and "trend prone" as the old school Wordpress days. It's the same problems over and over and every new "generation" of programmers has to relearn the lessons of old.
Systems that "work" tend to have some way of correcting for or mitigating the principal agent problem by aligning incentives.
I'd also point out that hardware is a much older discipline, in terms of how long it's been operating at scale. It's had more time to formalize and crystallize. Intel is 56 years old. Google is 27.
First, hardware engineers are dealing with the same laws of physics every time. Materials have known properties etc.
Software: there are few laws of physics (mostly performance and asymptotic complexity). Most software isnt anywhere near those boundaries so you get to pretend they dont exist. If you get to invent your own physics each time, yeah the process is going to look very different.
Engineering is the intersection of applied sciences, economics and business. The economics aspect is almost never recognized and explains many things. Projects of other disciplines have significantly higher costs and risks, which is why they require a lot more rigor. Taking hardware as example, one bad design decision can sink the entire company.
On the other hand, software has economics that span a much more diverse range than any other field. Consider:
- The capital costs are extremely low.
- Development can be extremely fast at the task level.
- Software, once produced, can be scaled almost limitlessly for very cheap almost instantly.
- The technology moves extremely fast. Most other engineering disciplines have not fundamentally changed in decades.
- The technology is infinitely flexible. Software for one thing can very easily be extended for an adjacent business need.
- The risks are often very low, but can be very high at the upper end. The rigor applied scales accordingly. Your LoB CRUD app going down might bother a handful of people, so who cares about tests? But your flight control software better be (and is) tested to hell and back.
- Projects vary drastically in stacks, scopes and risk profiles, but the talent pool is more or less common. This makes engineering culture absolutely critical because hiring is such a crapshoot.
- Extreme flexibility also masks the fact that complexity compounds very quickly. Abstractions enable elegant higher-level designs, but they mask internal details that almost always leak and introduce minor issues that cause compounding complexity.
- The business rules that software automates are extremely messy to begin with (80K payroll rules!) However, the combination of a) flexibility, b) speed, and c) scalability engender a false sense of confidence. Often no attempt is made at all to simplify business requirements, which is probably where the biggest wins hide. This is also what enables requirements to shift all the time, a prime cause for failures.
Worse, technical and business complexity can compound. E.g. its very easy to think "80K payroll rules linearly means O(80K) software modules" and not "wait, maybe those 80K payroll rules interact with each other, so it's probably a super-linear growth in complexity." Your architecture is then oriented towards the simplistic view, and needs hacks when business reality inevitably hits, which then start compounding complexity in the codebase.
And of course, if that's a contract up for bidding, your bid is going to be unsustainably low, which will be further depressed by the competitive bidding process.
If the true costs of a project -- which include human costs to the end users -- are not correctly evaluated, the design and rigor applied will be correspondingly out of whack.
As such I think most failures, in addition to regular old human issues like corruption, can be attributed to an insufficient appreciation of the economics involved, driven primarily by overindexing on the powers of software without an appreciation of the pitfalls.
If people want to know why Microsoft hated DOS and wanted to kill it with Xenix, then OS/2, then Windows, and then NT it would be vital to know that it only came about as a result of IBM wanting a 16bit source-compatible CP/M which didn’t yet exist. Then, you would likely want to read Dissecting DOS to see what limits were imposed by DOS.
For other stuff, you would start backwards. Take the finished product and ask what the requirements were, then ask what the pain points are, then start digging through the source and flowcharting/mapping it. This part is a must because programs are often too difficult to really grok without some kind of map/chart.
There is likely an entire discipline to be created in this…
You'll learn all of this for yourself, on the job, just via experience.
Your company's root priorites are probably at odds with writing good software.
One Japanese company, not going to name names, kept trying to treat software as a depreciating asset. I didn't really understand well but the long and short was that fixing things that were supposed to be "done" was bad for the accounting. New things, however were good.
How can you run a software company like that? But they did and got the kind of outcome you'd expect. Japan made the laws this way and gets software to match.
That's why we see every now and then "new" programming paradigms which were once obsolete.
The retro community is huge and varied. If it exists, someone is really into it.
Hardware folks just follow best practices and physics.
They're different problem spaces though, and having done both I think HW is much simpler and easier to get right. SW is often similar if you're working on a driver or some low-level piece of code. I tried to stay in systems software throughout my career for this reason. I like doing things 'right' and don't have much need to prove to anyone how clever I am.
I've met many SW folks who insist on thinking of themselves as rock stars. I don't think I've ever met a HW engineer with that attitude.