Trillions spent and big software projects are still failing

That's what works for products, not software systems. Gradual growth inevitably results in loads of technical debt that is not paid off as Product adds more feature requests to deliver larger and larger sales contracts. Eventually you want to rewrite to deal with all the technical debt, but nobody has enough confidence to say what is in the codebase that's important to Product and what isn't, so everybody is afraid and frozen.

Scale is separately a Product and Engineering question. You are correct that you cannot scale a Product to delight many users without it first delighting a small group of users. But there are plenty of scaled Engineering systems that were designed from the beginning to reach massive scale. WhatsApp is probably the canonical example of something that was a rather simple Product with very highly scaled Engineering and it's how they were able to grow so much with such a small team.

The accounting, legal and business process requirements are vastly different at different scales, different jurisdictions, different countries, etc.

There's a crazy amount of complexity and customizability in systems like ERPs for multinational corporations (SAP, Oracle).

When you start with a small town, you'll have to throw most of everything away when moving to a different scale.

That's true for software systems in general. If major requirements are bolted on after the fact, instead of designed into the system from the beginning, you usually end up with an unmaintainable mess.

While I think this is good advice in general, I don’t think your statement that “there is no process to create scalable software” holds true.

The uk gov development service reliably implements huge systems over and over again, and those systems go out to tens of millions from day 1. As a rule of thumb, the parts of the uk govt digital suite that suck are the parts the development service haven’t been assigned to yet.

The Swift banking org launches reliable features to hundreds of millions of users.

There’s honestly loads of instances of organisations reliably implementing robust and scalable software without starting with tens of users.

What works at small scale possibly won't work at a huge scale.

The dominant factor is: there is a human who understands the entire system.

That is vastly easier to achieve by making a small, successful system, which gets buy in from both users and builders to the extent that the former pay sufficient money for the latter to be invested in understanding the entire system and then growing it and keeping up with the changes.

Occasionally a moon shot program can overcome all of that inertia, but the “90% of all projects fail” is definitely overrepresented in large projects. And the Precautionary Principle says you shouldn’t because the consequences are so high.

See also Gall's Law:

"All complex systems that work evolved from simpler systems that worked"

> If you need to make a national payroll, you have to use it for a small town with a payroll of 50 people first, get the bugs worked out, then try it with a larger town, then a small city, then a large city, then a province, and then and only then are you ready to try it at a national level.

At a large box retail chain (15 states, ~300 stores) I worked on a project to replace the POS system.

The original plan had us getting everything working (Ha!) and then deploying it out to stores and then ending up with the two oddball "stores". The company cafeteria and surplus store were technically stores in that they had all the same setup and processes but were odd.

When the team that I was on was brought into this project, we flipped that around and first deployed to those two several months ahead of the schedule to deploy to the regular stores.

In particular, the surplus store had a few dozen transactions a day. If anything broke, you could do reconciliation by hand. The cafeteria had single register transaction volume that surpassed a surplus store on most any other day. Furthermore, all of its transactions were payroll deductions (swipe your badge rather than credit card or cash). This meant that if anything went wrong there we weren't in trouble with PCI and could debit and credit accounts.

Ultimately, we made our deadline to get things out to stores. We did have one nasty bug that showed up in late October (or was it early November?) with repackaging counts (if a box of 6 was $24 and if purchased as a single item it was $4.50 ... but if you bought 6 single items it was "repackaged" to cost $24 rather than $27) which interacted with a BOGO sale. That bug resulted in absurd receipts with sales and discounts (the receipt showed you spent $10,000 but were discounted $9,976 ... and then the GMs got alerts that the store was not able to make payroll because of a $9,976 discount ... one of the devs pulled an all nighter to fix that one and it got pushed to the stores ).

I shudder to think about what would have happened if we had tried to push the POS system out to customer facing stores where the performance issues in the cafeteria where worked out first or if we had to reconcile transactions to hunt down incorrect tax calculations.

> https://www.amazon.com/How-Big-Things-Get-Done/dp/0593239512

This is what https://www.amazon.com/How-Big-Things-Get-Done/dp/0593239512 advocates too: start small, modularize, and then scale. The example of Tesla's mega factory was particular enticing.

Came here to say this. I still think that Linus Torvalds has the most profound advice to building a large, highly successful software system:

"Nobody should start to undertake a large project. You start with a small trivial project, and you should never expect it to get large. If you do, you'll just overdesign and generally think it is more important than it likely is at that stage. Or worse, you might be scared away by the sheer size of the work you envision. So start small, and think about the details. Don't think about some big picture and fancy design. If it doesn't solve some fairly immediate need, it's almost certainly over-designed. And don't expect people to jump in and help you. That's not how these things work. You need to get something half-way useful first, and then others will say "hey, that almost works for me", and they'll get involved in the project."

-- Linux Times, October 2004.

Yes. Also the same applies to companies. There should not be companies that are growing to $100 million revenue while losing money on a gamble that they will eventually get big enough to succeed. Good first, big later.

You will never get to the moon by making a faster and faster bus.

I see a lot of software with that initial small scale "baked into it" at every level of its design, from the database engine choice, schema, concurrency handling, internal architecture, and even the form design and layout.

The best-engineered software I've seen (and written) always started at the maximum scale, with at least a plan for handling future feature extensions.

As a random example, the CommVault backup software was developed in AT&T to deal with their enormous distributed scale, and it was the only decently scalable backup software I had ever used. It was a serious challenge with its competitors to run a mere report of last night's backup job status!

I also see a lot of "started small, grew too big" software make hundreds of silly little mistakes throughout, such as using drop-down controls for selecting users or groups. Works great for that mom & pop corner store customer with half a dozen accounts, fails miserably at orgs with half a million. Ripping that out and fixing it can be a decidedly non-trivial piece of work.

Similarly, cardinality in the database schema has really irritating exceptions that only turn up at the million or billion row scale and can be obscenely difficult to fix later. An example I'm familiar with is that the ISBN codes used to "uniquely" identify books are almost, but not quite unique. There are a handful of duplicates, and yes, they turn up in real libraries. This means that if you used these as a primary key somewhere... bzzt... start over from the beginning with something else!

There is no way to prepare for this if you start with indexing the book on your own bookshelf. Whatever you cook up will fail at scale and will need a rethink.

That sounds like the way nature handles growth and complexity: slowly and over long time scales. Assume there will be failures, don't die and keep trying.

When you bite off too much complexity at once you end up not shipping anything or building something brittle.

> If you need to make a national payroll, you have to use it for a small town with a payroll of 50 people first, get the bugs worked out, then try it with a larger town, then a small city, then a large city, then a province, and then and only then are you ready to try it at a national level.

You could also try to buy some off-the-shelf solutions? Making payroll, even for very large organisations, isn't exactly a new problem.

As a corollary I would also suggest: subsidiarity.

> Subsidiarity is a principle of social organization that holds that social and political issues should be dealt with at the most immediate or local level that is consistent with their resolution.

(from https://en.wikipedia.org/wiki/Subsidiarity)

If you solve more problems more locally, you don't need that many people at the national level, thus making payroll there is easier.

I work at a small shop, I'm a big advocate of giving customers the 0.1 version and then talking it out what they want. It's often not exactly what they asked for at the start ... but it often is better in the end.

It's hard to hit the target right the first time.

You just need: Plan -> Implement -> Test -> Repeat

Whether you are creating software, games or whatever, these iterations are foundational. How these steps look like in detail of course depends on the project itself.

Not saying you're wrong, but I wonder what is the differentiating factor for software? We can build huge things like airliners, massive bridges and buildings without starting small.

Incremental makes less sense to me when you want to go to mars. Would you propose to write the software for such a mission in an incremental fashion too?

Yet for software systems it is sometimes proposed as the best way.

That's the ideal, but a lot of these big problems can't start small because the problem they have is already big. A lot of government IT programs are set up to replace existing software and -processes, often combining a lot of legacy software's jobs and the manual labor involved.

If you have something like a tax office or payroll, they need to integrate decades of legislation and rules. It's doable, but you need to understand the problem (which at those scales is almost impossible to fit in one person's head) and more importantly have diligent processes and architecture to slowly build up and deploy the software.

tl;dr it's hard. I have no experience in anything that scale, I've been at the edges of large organizations (e.g. consumer facing front-ends) for most of my career.

While I like the "start small and expand" strategy better than the "big project upfront", this trades project size for project length and often that is no better:

- It gives outside leadership types many more opportunities to add requirements later. This is nice is they are things missed in the original design, but it can also lead to massive scope creep.

- A big enough project that gets done the "start small and expand" way can easily grow into a decade-plus project. For an extreme example, see the multi-decade project by the Indian rail company to gradually replace all its railways to standard gauge. It works fine if you have the organisational backing for a long duration, but the constant knowledge leaks from people leaving, retiring, getting promoted, etc can be a real problem for a project like that. Especially in fields where the knowledge is the product, like in software.

- Not every project can feasibly start small.

> A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.

Gall’s law wins again.

There is no solution because these projects are not failing because of technical reasons.

They are failing because of political scheming and bunch of people wanting to have a finger in the pie - "trillions spent" - I guess no one would mind earning couple millions.

Then you have "important people" who want to be important and want to have an opinion on font size and that some button should be 12px to the right because they are "important" it doesn't matter for the project but they have to assert the dominance.

You have 2 or 3 companies working on a project? Great! now they will be throwing stuff over the fence to limit their own cost and blame others while trying to get away with as least work done cashing most money possible.

That is how sausage is made. Coming up with "reasonable approach" is not the solution because as soon as you get different suppliers, different departments you end up with power/money struggle.

> I'm afraid that the solution is: build something small, and use it in production before you add more features.

Gall's Law:

> A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.[8]

* https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law

Imagine if the only way to build a skyscraper was to start with a dollhouse and keep tacking extensions and pieces onto it until. Imagine if the only way to build a bridge across San Francisco bay was to start with pop sickle sticks.

I do get concerned when the solution is to be more strict on the waterfall process.

I used to believe there were some worlds in which waterfalls are better: where requirements are well know in advance and set in stone. I’ve since come to realize neither of those assumptions is ever true.

>It's a great article, until the end where they say what the solution would be. I'm afraid that the solution is: build something small, and use it in production before you add more features.

I think that is true for a lot of projects. But I'm not sure it is realistic to incrementally develop a control system for a nuclear reactor or an air traffic control system.

The very specific example you chose: payroll, shows how it can be difficult to incrementally step from small to huge. As you grow from town to national, you will run into all the disadvantages without really hitting the advantages. I feel that incremental does help you move from one level to one just a few above. But only if there are enough customers at these starting levels exactly.

When developing for towns, you will have all small random subsets of the variations imposed by year after year of legal changes BUT small sales. You will have to implement niche variations in arbitrary aspects for all the towns you have to support AND you will not have the customer size on which to amortize this work. Each new customer will bring a new arbitrary set of legal aspects to be met. Each new customer may be arbitrarily difficult to support.

By the time you reach national, you will have already covered most of the historical legal quirks - but that will have been done in one kludgy manner after another - and then you will hit one more set of legal quirks at the level of national organizations (some of them will have their very own laws). You will now have a very large budget to finalize things but you will be burdened by an illogical software base.

So I agree that you will need experience and subject matter experts that have worked at the various levels. BUT, now that you have this experience you now know the degree of flexibility that is required (you know where and what needs to be variable and quirk-friendly and how far the quirks can go = "any size") as well as size-related issues (mailing, transaction, user support volume) and you can now plan for all this AS YOU restart a new development from scratch. Because at this new "master" level you need both systematic flexibility AND relience at size.

Payroll is exactly the kind of topic where "adding features" will be "fun" - I mean bewildering - while you learn, but probably economically difficult to manage, until it kills you "as you climb up"?

You will be killed by a large software project that can afford to hire out a bunch of your subject matter specialists (or hires new ones) and uses them in a "from scratch" project. If you are lucky, this large project will be from the same company but only if you are lucky.

Now. AFTER you have done the one top level project - for one country -, you will probably be in a good situation to sell service to all kinds of organizations. Because you now have a system in which you can implement ridiculous quirks without breaking everything. And if you have done the job just right, you can onboard smaller customers (towns) economically enough that they can afford your solution.

That's different from where you deploy your solution first. Sure, deploy a national-design solution first at a subset of the target employees - although that does impose more requirements still: now you need to coexist with the legacy solutions. Which would be another hard to meet handicap when developing for towns first.

This is one part of the issue. The other major piece of this that I've seen over more than two decades in industry is that most large projects are started by and run by (but not necessarily the same person) non-technical people who are exercising political power, rather than by technical people who can achieve the desired outcomes. When you put the nexus of power into the hands of non-technical people in a technical endeavor you end up with outcomes that don't match expectations. Larger scale projects deeply suffering from "not knowing what we don't know" at the top.

Agree 100%.

I know a lot of people on here will disagree with me saying this but this is exactly how you get an ecosystem like javascript being as fragmented, insecure, and "trend prone" as the old school Wordpress days. It's the same problems over and over and every new "generation" of programmers has to relearn the lessons of old.

I was so annoyed when I found out the OTP library and realized we’ve been reinventing things for 20+ years

I have a theory that the churn in technology is by design. If a new paradigm, new language, new framework comes out every so many years, it allows the tech sector to always want to hire new graduates for lower salaries. It gives a thin veneer of we want to always hire the person who has X when really they just do not want to hire someone with 10 years of experience in tech but who may not have picked up X yet.

I do not think it is the only reason. The world is complex, but I do think it factors into why software is not treated like other engineering fields.

I've also considered a side-effect of this. Each generation of software engineers learns to operate on top of the stack of tech that came before them. This becomes their new operating floor. The generations before, when faced with a problem, would have generally achieved a solution "lower" down in the stack (or at their present baseline). But the generations today and in the future will seek to solve the problems they face on top of that base floor because they simply don't understand it.

This leads to higher and higher towers of abstraction that eat up resources while providing little more functionality than if it was solved lower down. This has been further enabled by a long history of rapidly increasing compute capability and vastly increasing memory and storage sizes. Because they are only interacting with these older parts of their systems at the interface level they often don't know that problems were solved years prior, or are capable of being solved efficiently.

I'm starting to see ideas that will probably form into entire pieces of software "written" on top of AI models as the new floor. Where the model basically handles all of the mainline computation, control flow, and business logic. What would have required a dozen Mhz and 4MB of RAM to run now requires TFlops and Gigabytes -- and being built from a fresh start again will fail to learn from any of the lessons learned when it was done 30 years ago and 30 layers down.

I work at $FANG, every one of our org's big projects go off the rails at the end of the project and there's always a mad rush at the end to push developers to solve all the failures of project management in their off hours before the arbitrary deadline arrives.

After every single project, the org comes together to do a retrospective and ask "What can devs do differently next time to keep this from happening again". People leading the project take no action items, management doesn't hold themselves accountable at all, nor product for late changing requirements. And so, the cycle repeats next time.

I led and effort one time, after a big bug made it to production after one of those crunches that painted the picture of the root cause being a huge complicated project being handed off to offshore junior devs with no supervision, and then the junior devs managing it being completely switched twice in the 8 month project with no handover, nor introspection by leadership. My manager's manager killed the document and wouldn't allow publication until I removed any action items that would constrain management.

And thus, the cycle continues to repeat, balanced on the backs of developers.

There are rational explanations for this. When software fails catastrophically, people almost never die (considering how much software crashes every day). When hardware fails catastrophically, people tend to die, or lose a lot of money.

There's also the complexity gap. I don't think giving someone access to the Internet Explorer codebase is necessarily going to help them build a better browser. With millions of moving parts it's impossible to tell what is essential, superfluous, high quality, low quality. Fully understanding that prior art would be a years long endeavor, with many insights no doubt, but dubious.

I would boil this down to something else, but possibly related: project requirements are hard. That's it.

> While hardware folks study and learn from the successes and failures of past hardware, software folks do not. People do not regularly pull apart old systems for learning.

For most IT projects, software folks generally can NOT "pull apart" old systems, even if they wanted to.

> Typically, software folks build new and every generation of software developers must relearn the same problems.

Project management has gotten way better today than it was 20 years, so there is definitely some learnings that have been passed on.

"While hardware folks study and learn from the successes and failures of past hardware, software folks do not." Couldn't be further from the truth. Software folks are obsessed with copying what has been shown to work to the point that any advance quickly becomes a cargo cult (see microservices for example).

Once you've worked in both hardware and software engineering you quickly realize that they only superficially similar. Software is fundamentally philosophy, not physics.

Hardware is constrained by real world limitations. Software isn't except in the most extreme cases. Result is that there is not a 'right' way to do any one thing that everyone can converge on. The first airplane wing looks a whole lot like a wing made today, not because the people that designed it are "real engineers" or any such BS, but because that's what nature allows you to do.

... are you saying that hardware projects fail less than software ones? just building a bridge is something that fails on a regular occurence all over the world. Every chip comes with list of erratas longer than my arm.

In my experience, a lot of the time the people who COULD be solving these issues are people who used to code or never have. The actual engineers who might do something like this aren't given authority or scope and you have MBAs or scrum masters in the way of actually solving problems.

I think part of it is that reading code isn't a skill that most people are taught.

When I was in grad school ages ago, my advisor told me to spend a week reading the source code of the system we were working with (TinyOS), and come back to him when I thought I understood enough to make changes and improvements. I also had a copy of the Linux Core Kernel with Commentary that I perused from time to time.

Being able to dive into an unknown codebase and make sense of where the pieces are put together is a very useful skill that too many people just don't have.

> While hardware folks study and learn from the successes and failures of past hardware, software folks do not

I've been managing, designing, building and implementing ERP type software for a long time and in my opinion the issue is typically not the software or tools.

The primary issue I see is lack of qualified people managing large/complex projects because it's a rare skill. To be successful requires lots of experience and the right personality (i.e. low ego, not a person that just enjoys being in charge but rather a problem solver that is constantly seeking a better understanding).

People without the proper experience won't see the landscape in front of them. They will see a nice little walking trail over some hilly terrain that extends for about a few miles.

In reality, it's more like the Fellowship of the Rings trying to make it to Mt Doom, but that realization happens slowly.

Most of the time, there's no need to study anything. Any experienced software engineer can tell you about a project they worked on with no real requirements, management constantly changing their mind, etc.

Some consequences of NOT learning from prior successes and failures: (a) no more training for the next generation of developers/engineers (b) fighting for the best developers, and this manifests in leetcode grinding (c) decrease in cooperation among team mates, etc.

I think this is a downstream of effect of there being no real regulation or professional designations in software which leads to every company and team being wildly different leading to no standards leaving no time for anything but crunching since there are no barriers restricting your time, so nobody spends time doing much besides shipping constantly.

Software folks treat their output as if it's their baby or their art projects.

Hardware folks just follow best practices and physics.

They're different problem spaces though, and having done both I think HW is much simpler and easier to get right. SW is often similar if you're working on a driver or some low-level piece of code. I tried to stay in systems software throughout my career for this reason. I like doing things 'right' and don't have much need to prove to anyone how clever I am.

I've met many SW folks who insist on thinking of themselves as rock stars. I don't think I've ever met a HW engineer with that attitude.

As someone who's learning programming right now, do you have any suggestions on how one would go about finding and studying these successes and failures?

Indeed.

That's why we see every now and then "new" programming paradigms which were once obsolete.

I think there is a ton more nuance, but can still be explained by a simple observation, which TFA hints at: "It's the economics, stupid."

Engineering is the intersection of applied sciences, economics and business. The economics aspect is almost never recognized and explains many things. Projects of other disciplines have significantly higher costs and risks, which is why they require a lot more rigor. Taking hardware as example, one bad design decision can sink the entire company.

On the other hand, software has economics that span a much more diverse range than any other field. Consider:

- The capital costs are extremely low.

- Development can be extremely fast at the task level.

- Software, once produced, can be scaled almost limitlessly for very cheap almost instantly.

- The technology moves extremely fast. Most other engineering disciplines have not fundamentally changed in decades.

- The technology is infinitely flexible. Software for one thing can very easily be extended for an adjacent business need.

- The risks are often very low, but can be very high at the upper end. The rigor applied scales accordingly. Your LoB CRUD app going down might bother a handful of people, so who cares about tests? But your flight control software better be (and is) tested to hell and back.

- Projects vary drastically in stacks, scopes and risk profiles, but the talent pool is more or less common. This makes engineering culture absolutely critical because hiring is such a crapshoot.

- Extreme flexibility also masks the fact that complexity compounds very quickly. Abstractions enable elegant higher-level designs, but they mask internal details that almost always leak and introduce minor issues that cause compounding complexity.

- The business rules that software automates are extremely messy to begin with (80K payroll rules!) However, the combination of a) flexibility, b) speed, and c) scalability engender a false sense of confidence. Often no attempt is made at all to simplify business requirements, which is probably where the biggest wins hide. This is also what enables requirements to shift all the time, a prime cause for failures.

Worse, technical and business complexity can compound. E.g. its very easy to think "80K payroll rules linearly means O(80K) software modules" and not "wait, maybe those 80K payroll rules interact with each other, so it's probably a super-linear growth in complexity." Your architecture is then oriented towards the simplistic view, and needs hacks when business reality inevitably hits, which then start compounding complexity in the codebase.

And of course, if that's a contract up for bidding, your bid is going to be unsustainably low, which will be further depressed by the competitive bidding process.

If the true costs of a project -- which include human costs to the end users -- are not correctly evaluated, the design and rigor applied will be correspondingly out of whack.

As such I think most failures, in addition to regular old human issues like corruption, can be attributed to an insufficient appreciation of the economics involved, driven primarily by overindexing on the powers of software without an appreciation of the pitfalls.

I’ve read one tech history book and I really enjoyed it. any you recommend?

How do you study software history? Most of the lessons seem forever locked away behind corporate walls - any honest assessments made public will either end careers or start lawsuits

Yes, and it's because there aren't very many textbook ways to do software engineering, because it's evolving too fast to reach consensus.

Software just feels so much more ephemeral than hardware. I haven't yet met a single 'old software enthusiast' in my life, yet there are so many enthusiasts for older hardware.

This is an interesting distinction, but it ignores the reasons software engineers do that.

First, hardware engineers are dealing with the same laws of physics every time. Materials have known properties etc.

Software: there are few laws of physics (mostly performance and asymptotic complexity). Most software isnt anywhere near those boundaries so you get to pretend they dont exist. If you get to invent your own physics each time, yeah the process is going to look very different.

I think this is too simple. First of all, hardware people have high incentive to fully replace components and systems for many reasons. Replacement is also the only way they can fix major design mistakes. Software people constantly do fix bugs and design mistakes. There is certainly no strong culture to document or dig up former mistakes made, but it's not like they don't learn from mistakes, it's just a constant process. In contrast to hardware, there is usually no point in time to retrospect. The incentives to rejuvenate systems are low and if considered often seem expensive. Software engineers self motivation is often ill-minded, new devs feeling uncomfortable with the existing system and calling for something "modern". But if the time comes to replace the "legacy" systems, then you are right, no one looks back at the former mistakes and the devs that know them, are probably long gone. The question is whether we should ever replace an software system or focus more on gradual and active modernization. But the latter can be very hard, in hardware everything is defined, most of the time backed by standards, in software we usually don't have that, so complex interconnected systems rarely have sane upgrade paths.

IME, "Why systems fail" almost always boils down to a principal-agent problem. This is another way of expressing the Mungerism "show me the incentive, I'll show you the outcome".

Systems that "work" tend to have some way of correcting for or mitigating the principal agent problem by aligning incentives.

I'd also point out that hardware is a much older discipline, in terms of how long it's been operating at scale. It's had more time to formalize and crystallize. Intel is 56 years old. Google is 27.

Do you mind sharing some good reads / what you wrote?

> Early in my career, I got into solving some of the barriers to software project management (e.g., requirements analysis and otherwise understanding needs, sustainable architecture, work breakdown, estimation, general coordination, implementation technology).

Was there any literature or other findings that you came across that ended up clicking and working for you that you can recommend to us?

Right, it's largely politically and ego driven; a people not a software problem.

I just fed the above into Claude Code and it one-shotted this in 5 minutes. Already doing $3B ARR after lunch.

This touches on the absolutely vital issue of domain knowledge. Everybody understands that you're not supposed to have the same people handle sewer maintenance and preschool teaching because these are two entirely separate skillsets. To an extent you can also treat kindergartens and treatment plants as black boxes that consume money and produce desired services.

For people who don't know much about programs it's sort of natural to assume that software engineering works the same way. Put in money and specs, get back programs. But of course it doesn't work like that, because software dev is not a single skillset. To write useful programs, you have to know how to code and understand the environment in which the program will be used.

To be fair, that’s a rare exception. Most government tenders are quite narrow in scope.

What I have found is that they’re written by people with zero knowledge of either the solution requirements or the technology! Combine that with zero profit motive and zero personal consequences, and you can end up with total nonsense even on projects with billion dollar budgets.

A state school department here put out a tender for wiring over two thousand schools with fibre, but the way the contract was stipulated only a single applicant could win the contract and most handle every single location across a thousand miles of territory. Hence, only the largest incumbent telco could possibly win… which they did… at 15x the cost of a bunch of local contractors doing the work. This cost something like a billion dollars to taxpayers.

The excuse of the guy writing the tender was “it’s easier for me to get one contract signed than fifty.”

He’s a public servant getting paid $50K. He’s got nothing else on, no other pressing needs or distractions, but he’s too busy, you see? So much easier to waste a billion dollars to save himself a few months of effort.

It is not just government. Private companies also have the same problem.

One reason why aws got so big is because it took months to get infrastructure to provision a virtual machine.

I think if we look at the lack of accountability it's obvious that one major problem is that many of these projects do heavily rely on contract work. No company or gov in the world can supply the perfect brain- and manpower necessary on day one (on a huge and complex project that requires expert knowledge). So there is an prevalent delusion that talent just spawns at project kickoff and those people even care about what they do.

Maybe this is some artifact we carried over from the industrial era. We expect that complex machinery is built by expert companies over night and they just work, with a bit of maintenance and knowledge transfer. But software doesn't work like that.

If software engineers want to be referred to as "engineers" then they should actually learn about engineering failures. The industry and educational pipeline (formal and informal) as a whole is far more invested in butterfly chasing. It's immature in the sense that many people with decades of experience are unwilling to adopt many proven practices in large scale engineering projects because they "get in the way" and because they hold them accountable.

> Software projects fail because humans fail. Humans are the drivers of everything in our world.

Ah finally - I've had to scroll halfway down to find a key reason big software projects fail.

<rant>

I started programming in 1990 with PL/1 on IBM mainframes and for 35 years have dipped in and out of the software world. Every project I've seen fail was mainly down to people - egos, clashes, laziness, disinterest, inability to interact with end users, rudeness, lack of motivation, toxic team culture etc etc. It was rarely (never?) a major technical hurdle that scuppered a project. It was people and personalities, clashes and confusion.

</rant>

Of course the converse is also true - big software projects I've seen succeed were down to a few inspired leaders and/or engineers who set the tone. People with emotional intelligence, tact, clear vision, ability to really gather requirements and work with the end users. Leaders who treated their staff with dignity and respect. Of course, most of these projects were bland corporate business data ones... so not technically very challenging. But still big enough software projects.

Gez... don't know why I'm getting so emotional (!) But the hard-core sofware engineering world is all about people at the end of the day.

> But I think those have the same flaw, in that there isn't enough of a consequence for each part of the process

If there was sufficient consequence for this stuff, no one would ever take on any risk. No large works would ever even be started because it would be either impossible or incredibly difficult to be completely sure everything will go to plan.

So instead we take a medium amount of caution and take on projects knowing it's possible for them to not work out or to go over budget.

Management is always a huge problem, but software engineers left to their own devices can be just as bad.

I very rarely hear actual technical reasons for why a decision was made. They're almost always invented after the fact to retroactive justify some tool or design pattern the developer wanted to use. Capabilities and features get tacked on just because it's something someone wanted to do, not because they solve an actual problem or can be traced back to requirements in any meaningful way.

Frankly as an industry we could learn a lot from other engineering fields, aerospace and electrical engineering in particular. They aren't perfect, but in general they're much better at keeping technical decisions tied to requirements. Their processes tend to be too slow for our industry of course, but that doesn't mean there aren't lessons to be learned.

Exactly this. Not just large software projects tend to fail often; also large architectural and infrastructure projects do. There are loads of examples, one famous one for instance is the Berlin Airport.

Management is bad at managing large projects. Whatever those projects are. In particular when third parties are involved that have a financial interest.

This is precisely the point of the article. I mean, it's right there at the top in that weird arrow-shaped infographic. It's _almost_ always a management issue.

Why do you find it offensive? It’s not personal. Someone who thought webvan was a great lesson in hubris could not have built an Instacart, right? Even evolution shuns experience, all but throwing most of it out each generation, with a scant few species as exceptions.

> the 10 NASA principles

Could you be more specific?

Unix and Linux would be your quintessential examples.

Unix was an effort to take Multics, an operating system that had gotten too modular, and integrate the good parts into a more unified whole (book recommendation: https://www.amazon.com/UNIX-History-Memoir-Brian-Kernighan/d...).

Even though there were some benefits to the modularity of Multics (apparently you could unload and replace hardware in Multics servers without reboot, which was unheard of at the time), it was also its downfall. Multics was eventually deemed over-engineered and too difficult to work with. It couldn't evolve fast enough with the changing technological landscape. Bell Labs' conclusion after the project was shelved was that OSs were too costly and too difficult to design. They told engineers that no one should work on OSs.

Ken Thompson wanted a modern OS so he disregarded these instructions. He used some of the expertise he gained while working on Multics and wrote Unix for himself (in three weeks, in assembly). People started looking over Thompson's shoulder being like "Hey what OS are you using there, can I get a copy?" and the rest is history.

Brian Kernighan described Unix as "one of" whatever Multics was "multiple of". Linux eventually adopted a similar architecture.

More here: https://benoitessiambre.com/integration.html

I find it kind of hard to define success or failure. Google search and Facebook are a success right? And they were able to scale up as needed, which can be hard. But the way they started is very different from a government agency or massive corporation trying to orchestrate it from scratch. I don't know if you'd be familiar with this, but maybe healthcare.gov is a good example... it was notoriously buggy, but after some time and a lot of intense pressure it was dealt with.

I heard Direct File was pretty successful. Something like a 94% reported it as a positive experience.

This is a noble and ambitious goal. I feel qualified to provide some pointers, not because I have been instrumental in delivering hugely successful projects, but because I have been involved, in various ways, in many, many failed projects. Take what you will from that :-)

- Define "success" early on. This usually doesn't mean meeting a deadline on time and budget. That is actually the start of the real goal. The real success should be determined months or years later, once the software and processes have been used in a production business environment.

- Pay attention to Conways Law. Fight this at your peril.

- Beware of the risk of key people. This means if there is a single person who knows everything, you have a risk if they leave or get sick. Redundancy needs to be built into the team, not just the hardware/architecture.

- No one cares about preventing fires from starting. They do care about fighting fires late in the project and looking like a hero. Sometimes you just need to let things burn.

- Be prepared to say "no", alot. (This is probably the most important one, and the hardest.)

- Define ownership early. If no one is clearly responsible for the key deliverables, you are doomed.

- Consider the human aspect as equally as the technical. People don't like change. You will be introducing alot of change. Balancing this needs to be built into the project at every stage.

- Plan for the worst, hope for the best. Don't assume things will work the way you want them to. Test _everything_, always.

[Edit. Adding some items.]

India's UPI (digital payments) is almost as big a scale as it gets, and it's pretty universally considered a success: https://en.wikipedia.org/wiki/Unified_Payments_Interface

I don't think you should focus on successful large projects. Generally you should consider that all big successes are outliers from a myriad of attempts. They have been lucky and you can't reproduce luck.

I'd like try to correct your course a bit.

DevOps is a trash concept, that had good intentions. But today it's just an industry cheatcode to fill three dev positions with a single one that is on pager duty. The good takeaways from it: Make people care that things work end to end. If Joe isn't caring about Bob's problems, something is off. Either with the process, or with the people.

Agile is a very loose term nowadays. Broadly spoken it's the opposite of making big up front plans and implement them in a big swipe. Agile wants to start small and improve it iteratively as needed. This tends to work in the industry, but the iterative time buckets have issues, some teams can move fast in 2 week cycles, others don't. The original agile movement also wanted to give back control and autonomy back to those who actually do stuff (devs and lower management). This is very well intended and highly functional, but is often buried or ignored in corporate environments. Autonomy is extremely valuable, it motivates people and fosters personal growth, but being backed by a skilled peers also creates psychological safety. One of the major complaints I hear about agile practices is that there are too many routines, meetings and other in person tasks with low value that keep you from working. This is really bad and in my perception was never intended, but companies love that shit. This part is about communication, make it easy for people to share and engage, while also keeping their focus hours high. Problems have to bubble up quickly and everyone should be motivated and able to help solving them. If you listen to agile hardliners, they will also tell you that software can't be reliably planned, you won't make deadlines, none of them, never. That is very true, but companies are unable to deal with it.

> If you're 97% over budget, are you successful?

I don't like this as a metric of success, because who came up with the budget in the first place?

If they did a good job and you're still 97% over then sure, not successful.

But if the initial budget was a dream with no basis in reality then 97% over budget may simply have been "the cost of doing business".

It's easier to say what the budget could be when you're doing something that has already been done a dozen times (as skyscraper construction used to be for New York City). It's harder when the effort is novel, as is often the case for software projects since even "do an ERP project for this organization" can be wildly different in terms of requirements and constraints.

That's why the other comment about big projects ideally being evolutions of small projects is so important. It's nearly impossible to accurately forecast a budget for something where even the basic user needs aren't yet understood, so the best way to bound the amount of budget/cost mismatch is to bound the size of the initial effort.

"Worse" won't even start to describe the economical crisis we will be in once the bubble bursts.

And although that, in itself, should be scary enough, it is nothing compared to the political tsunami and unrest it will bring in its wake.

Most of the Western world is already on shaky political ground, flirting with the extreme-right. The US is even worse, with a pathologically incompetent administration of sociopaths, fully incapable of coming up with the measures necessary to slow down the train of doom careening out of control towards the proverbial cliff of societal collapse.

If the societal tensions are already close to breaking point now, in a period of relative economical prosperity, I cannot start to imagine what they will be like once the next financial crash hits. Especially one in the multi trillion of dollars.

They say that humanity progresses through episodes of turmoil and crisis. Now that we literally have all the knowledge of the world at our fingertips, maybe it is time to progress past this inadequate primeval advancement mechanism, and to truly enter an enlightened age where progress is made from understanding, instead of crises.

Unfortunately, it looks like it's going to take monumental changes to stop the parasites and the sociopaths from making at quick buck at the expense of humanity.

Not really, by most indications AI seems to be an amplifier more than anything else. If you have strong discipline and quality control processes it amplifies your throughput, but if you don't, it amplifies your problems. (E.g. see the DORA 2025 report.)

So basically things will still go where they were always going to go, just a lot faster. That's not necessarily a bad thing.

The GOV.UK project is a rare success story for IT in the UK government. They need to take that team, give them big incentives to stay, and give them more projects. Why are we outsourcing to international companies that don’t give a shit when we have a wealth of talent at home? Why aren’t we investing in our own people?

Despite its overall failure, some parts of the infrastructure and national applications, such as the Summary Care Record and the Electronic Prescriptions Service, are considered to have survived and continue to be used

I don't know any other industry where non-technical unqualified people can just decide everything in the way that it's done with software/IT

The point is not that the growth of IT spending is bad. That was just to show the scale of spending. The point of the article is that, a billion spent on software could well lead to a loss of hundred billion.

Success rates imply a ratio. Constant dollars are adjusted.

Yes there is a lot more spending overall. But nothing improved quality wise, despite everyone in software somehow says they "make software better". (Which is phrased by people that don't do software, but own it.)

How many bridges have you used that have collapsed? How much software have you used that has been broken or otherwise not served your interests? If we built the rest of society like we build software, millions of people would be dead.

Many specialties were eliminated / absorbed over the past few decades. I started working almost 30 years ago. Today, I rarely see dedicated testers, just like I rarely see dedicated DBAs. Sysadmins went away with the "DevOps" movement. Now they are cloud engineers who are more likely to understand a vendor-specific implementation than networking fundamentals.

Oh, it's much more interesting than that. Phoenix started as an attempt to create a gun registry. Ottawa had a bunch of civil servants that'd be reasonably compotent at overseeing such a thing, but the government decided that it wanted to build it in Miramichi, New Brunswick. The relevant people refused to move to Miramichi, so the project was built using IBM contractors and newbies. The resulting fiasco was highly predictable.

Then when Harper came in he killed the registry mostly for ideological reasons.

But then he didn't want to destroy a bunch of jobs in Miramichi, so he gave them another project to turn into a fiasco.

> Canada failed to do a payroll system

New Zealand tried to do a new payroll system for teachers called Novopay which imploded spectacularly and is still creating grief. The system is now called EdPay (the government decided to take over the privately created system). The total cost of the debacle was north of $200M NZD. Somehow they managed to fail to replace a working system!

> de-computerization

I would think cloud-disconnectedness (eg. computers without cloud hosted services) would come far before de-computerization.

Can you please provide a few examples to get the gist of such trend?

Honest question.

My reaction also. 80K payroll rules!!! Without much prompt effort, I got about 350K Canada Federal Service employees (sorry if not correct).

Wasn't the Agile movement kicked off by a group of people writing payroll software for Chrysler?

https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compens...

Payroll systems seem to be a massively complicated beast.

From a dark storage area I may someday again get out an early Sceptre gaming monitor from the DOS days.

I held on to it throughout the 1990's precisely because it was not a plug & play monitor and it was real good to install Windows with so nothing would interfere with higher resolution alternative graphics you were going to install later.

Now by the 21st century it was seldom seen but these were well-made and it still worked, however the most obsolete feature that got the most interest was the sleek aftermarket plastic accessory unit attached to the side of the monitor with those sticky 3M tacky pads that are so tenacious.

Yes, you've all seen it and remember it fondly, the mouse holder.

Kind of like a custom cup holder that fits the standard mouse perfectly, it's obviously where you keep your mouse most of the time, except for those rare occasions when you dabble in a bit of software which actually supports a mouse.

You want to keep it out of the way of your everyday desktop activities :)

Nailed it, but I fear this wisdom will be easily passed by by someone who doesn’t already intuit it from years of experience. Like the Island de la Muerta: wisdom that can only be found if you already know where it is.

One of the problems is scale.

Large scale systems tend to fail. large centralised and centrally managed systems with big budgets and large numbers of people who need to coordinate, lots of people with an interest in the project pushing and lobbying for different things.

Multiple smaller systems is usually a better approach, where possible. Not possible for things like transport infrastructure, but often possible for software.

> and my recent hyper-fixation (seems to spark up again every few years), the sinking of the Titanic.

But the rest of your comment reveals nothing novel other than anyone would find after watching James Cameron's movie multiple times.

I suggest you go to the original inquiries (congressional in the US, Board of trade in the UK). There is a wealth of subtle lessons there.

Hint: Look at the Admiralty Manual of Seamanship that was current at that time and their recommendations when faced with an iceberg.

Hint: Look at the Board of Trade (UK) experiments with the turning behaviour of the sister ship. In particular of interest is the engine layout of the Titanic and the attempt by the crew, inexperienced with the ship, to avoid the iceberg. This was critical to the outcome.

Hint: Look at the behaviour of Captain Rostron. Lots of lessons there.

>Building 1,000 bridges will make the cost of the next incremental bridge cheaper due to a zillion factors, even if Bridge #1 is built from sticks (we'll learn standards, stable, fundamental engineering principles, predicable failure modes, etc.) we'll eventually reach a stable, repeatable, scalable approach to build bridges. They will very rarely (in modernity) catastrophically fail (yes, Tacoma Narrows happened but in properly functioning societies it's rare.)

Build 1000 JSON parsers and tell me if the next one isn't cheaper to develop with "(we'll learn standards, stable, fundamental engineering principles, predicable failure modes, etc.)"

>Software isn't scalable in this way. It's not scalable because it doesn't have hard constraints (like the laws of physics)

Uh, maybe fewer but none is way to far. Get 2 billion integer operations per second out of a 286, the 500 mile email, big data storage, etc. Physical limits are everywhere.

>It's also the same reason why enterprise software sucks.

The reason enterprise software sucks is because the lack of introspection and learning from the garbage that went before.

I agree with you on "don't do big software project" Specially do not fast scale them out to hundreds of people. You have to scale them more organically ensuring that every person added is a net gain. They think that adding more people will reduce the time.

I am surprised on the lack of creativity when doing these projects. Why don't they start 5 small projects building the same thing and let them work for a year. At the end of the year you cancel one of the projects, increasing the funding in the other four. You can do that every year based on the results. It may look like a waste but it will significantly increase your chances of succeeding.

You have to be able to turn away unsuitable customers.

... If this happens, the next hacks will be context poisoning. A whole cottage industry will pop around preserving and restoring context.

Sounds miserable.

Also, LLMs don't learn. :)

While I don't disagree, it would increase the cost of software by an extraordinary amount.

will be moderated by* increased demands.

Systemically, how would you solve homelessness, if I gave you a trillion dollars?

The article isn't really about AI (for a change).

And that often takes a particular form: The requirements never converge, or at least never converge on anything realistically buildable.

Licensing and the perceived rigor it signifies is irrelevant to whether something can be considered "professional engineering." Engineering exists at the intersection of applied science, business and economics. So most software projects can be YOLO'd simply because the economics permit it, but there are others where the high costs necessitate more rigor.

For instance, software in safety-critical systems is highly rigorously developed. However that level of investment does not make sense for run-of-the-mill internal LOB CRUD apps which constitute the vast majority of the dark matter of the software universe.

Software engineering is also nothing special when it comes to various failure modes, because you'll find similar examples in other engineering disciplines.

I commented about this at length a few days ago: https://news.ycombinator.com/item?id=45849304

Nuclear power plants usually only cost about twice as much as projected in phase II planning. IT projects are sort of open-ended. Interestingly, the simulator I was involved with (many decades ago) at a nuclear power plant came within about 10% of initial projections. The last "scan uploads for viruses" project I worked on was about 20x - 40x more expensive than projected. (Unfortunately the person who suggested we just pay a third party for this service was fired.) The bit with projecting cost and schedules for nuke plants is to ignore any initial costing and multiply the phase II planning estimate by 2 or 4.

> blaming the IT community for not doing better, when most businesses refuse to look past anything but shipping features

IT != software engineering. IT is a business function that manages a company's internal information. Software engineering is a time-tested process of building software.

A lot of projects fail because management thinks that IT is a software engineering department. It is not. It never was, and it never will be. Its incentives will never be aligned such that software engineering projects are set up for success.

The success rate of implementing software outside of IT and dragging them along later is much higher than implementing it through IT from the beginning.

No need to waste GPUs, a simple bash script that alternates between asking for status updates and randomly changing requirements would do

There are great big software projects and shitty ones. IRCTC, UPI being examples of great ones.

Insurance and RTO being shitty ones.

I had an insurance deadline very near and the payment was not showing up in the insurance providers dashboard so had to do it twice and now it was still not showing up.

Also I have faced huge problems with getting the learner's licence online.

Do let me know if you've faced anything same.

Easy, just imagine a 1GB array as a 2.5mm long square in RAM (assuming a DRAM cell is 10nm). Now it's physical.

I came to opposite conclusions. Technology is pretty easy, people are hard and the business culture we have fostered in the last 40 years gets in the way of success.

Bureaucracy is: scar tissue, someone else's moat, someone else's data model

Enlighten us.

The UK Post Office lied and made people kill themselves ... because of dev ego?

As far as I can see in the modern tech industry landscape, virtually everyone has adopted style guides and automatic formatting/linting. Modern languages like Go even bake those decisions into the language itself.

I'd consider managing that stuff essentially table-stakes in big orgs these days. It doesn't stop projects from failing in highly expensive and visible ways.

Ironically, the downvotes pretty much prove this is exactly correct.