Why it is important that software projects fail (2008)
berglas.org
berglas.org
This is a short-sighted and overly pessimistic view of the role of software. Let's bring in some sorely needed optimism that's especially relevant to the current pandemic:
https://blog.cleancoder.com/uncle-bob/2020/04/05/ANewHope.ht...
Another area where software is actually good for the world is accessibility. Lots of people with disabilities, including me and several of my friends, are able to live independent and happy lives largely thanks to software such as screen readers, speech synthesizers (useful for both blind people and people who can't speak), optical character recognition, and speech recognition. Not to mention that many products and services that are merely convenient for able-bodied people are life-changing for us.
So I feel bad for people who are so caught up in the world of software that merely increases bureaucratic complexity, that they've lost sight of all of the good that software is doing, or at least don't get to contribute to that kind of software.
This revised statement would then lead to an interesting discussion about which software projects are beneficial to society and which should be allowed to fail. I would agree that a lot of enterprise software tend to increase workflow complexity. I think that's because in a corporate setting, the people with purchasing powers often don't understand how software technologies works, and they aren't the ones using the system.
An old(er) post that is vaguely relevant: https://news.ycombinator.com/item?id=21848860 An interesting look at the end of what makes projects slow down, and possibly insights into why bureaucracies grow.
> Really? Who?
> The programmers.
Wow. This is the pinnacle of narcissism.
How about another group of people who don't get mentioned as often, say Chinese workers who slave away at foxconn factories.
Maybe this piece wouldn't read as bad if it wasn't written by a programmer.
It takes me 1 minute to review, all data is already filled in, my yearly income tax (not much ongoing a part from my salary). And, two clicks in my phone approve, sign and send the declaration to the tax agency. No writing in paper, no looking for numbers, no finding old mail with my payslips. I would avoid living in any country that makes it more complicated that this.
If your public administration software sucks, you are voting the wrong people into public office. Do not blame the software.
The USAs tax system where the online portion was essentially privatized by a single company (Turbotax) was pretty jarring.
It's usually filed correctly already. So many times you'll just be clicking next.
If you have other income, like dividends, there are a few more boxes to fill, but it's still fairly simple.
Once you start having serious money, you better have a tax lawyer doing this for you, but I suppose this is the case everywhere.
Four years, maybe eight in some cases. Not much more. That's all they have to prove their worth. Most complex projects easily take longer than that to plan and execute. Civil servants on the other are typically there until they retire. In many countries civil servants enjoy special job protections that ensure it is typically hard to fire them. And since they are large organizations, there's a bit of a rat-race to get promoted to the lucrative top positions. That requires pleasing the people holding the purse strings. Politicians in other words. And politicians need to get stuff done and that requires warm relations with the established civil servant leadership because otherwise nothing gets done and they lose the next elections.
This adds to the problem because politicians basically rely on an already compromised bureaucratic machinery that resists changes to make it smaller. So any new policy just translates into more budget, more people, and more bureaucracy. The old ones never go away.
You see the same pattern of behavior in large corporations except those periodically get forced by their shareholders and the market to do something about it. Sometimes that fails and they get acquired, lose market-share, or even just implode and disappear. If you look at the fortune 500 through the years, you'll see there's a lot of churn there. New companies take up spots vacated by older failing corporations. But pretty much all of them employ way more people than the competitors that ultimately displace them.
Banks were mentioned in the article a few times. Most banks have actually been reducing head count for years. It used to be that they had to have offices in every small city, village, and even neighborhoods. Not any more. Those days are long gone. There are now several startups disrupting the space with online only banks. Any bank looking to survive in that space is pretty much copying that organizational model at this point. And as consumer banking is a this point a low margin product, a lot of the older and bloated banks are clearly struggling.
1. It's very difficult for voters to know how much a given politician will prioritize digital services.
2. Even if a politician appears to have a good digital strategy, will they implement it well?
3. Most decisions about public admin software are made by bureaucrats who aren't elected anyways.
4. Most elections will likely have more important issues than public admin software. Although I agree it probably does have huge returns on investment.
Sort of but not really. Just keep voting in progressives that believe in government until the bureaucracy gets the message that it pays to be progressive and fix things.
If we elect politicians in favor of simplifying government operations, and they can actually succeed, then it follows that government software would naturally simplify as well.
Note: This isn't necessarily the same as reducing government's role.
In my experience, there is a correlation between parties that follow the science, invest in education, and create useful software.
My critique is towards people that gives up and says that governments cannot make good software. And, that is false. It will never be perfect, like anything else, but it can be good.
As a US -> UK immigrant:
1. This is true.
2. This is made harder or easier by the complexity of the underlying abstractions it is trying to model.
I suspect the software that the NHS has to calculate how much to charge medical patients is very simple.
And, most importantly, author apparently believes that the observation that efficiencies introduced by new processes are invariably erased by increases in associated bureaucracy was novel and useful in 2008. People have been complaining about that since at least the Roman Republic.
Not only failed projects can prevent bureaucratic complexity, they can also save lives
> automated system assigns a unique identifying code to each incoming call before passing it on — a method of keeping track of phone calls as they move through the system. But on April 9, the software responsible for assigning the codes maxed out at a pre-set limit; the counter literally stopped counting at 40 million calls. As a result, the routing system stopped accepting new calls, leading to a bottleneck and a series of cascading failures elsewhere in the 911 infrastructure.
Had that automated system failed at the development phase...
https://www.washingtonpost.com/news/the-switch/wp/2014/10/20...
Go figure.
This sounds like a 32-bit TXID exhaustion bug in a Postgres database, which is a notorious and very difficult to recover from problem in live systems. It is probably the #1 arcane Postgres cataclysm. It would be consistent with how this is a "single coding error" (i.e., the call to VACUUM), additionally why it was so difficult to recover from ad-hoc. I'm generously assuming that everything else went well, that people who are building 911 systems aspire to do a really good job, and that it didn't fail anyone how important their work was.
Which is funny, I mean what if Postgres "failed" for every TXID exhaustion cataclysm it had. Arguably the best database product in the world "failed" because of this. It probably provides more than a few billion dollars to our tech economy, on top of being the hub for research in databases - research whose goal is the same as the one as this paper, to use automation to reduce bureaucracy and improve society.
Anyway we always imagine these rinky-dink consultant shops as writing all this custom stuff, and in actuality they are almost always combining a bunch open source and off the shelf things - which I would like to not see fail.
Besides the well known Y2K mitigation, there were some problems on day 10,000 when four characters to represent days since the epoch overflowed, mid 90s for Unix derived systems and I believe VMS as well. None were civilization threatening.
We also had Second One Billion in early 2000s. Also no public catastrophes.
Next big one is 2038, when 32-bit Unix timekeeping overflows.
Thanks for the reminder! We had a small fun party to celebrate!
None of it could have been done before computing power became so cheap that it needed to be wasted on doing it.
Everyone complains about useless procedures, but I think the real culprit is that we have no framework for discussing value. By that I mean the fundamental economic problem of what should we do, who should do what, how should stuff be shared?
In effect our current idea of value is a sort of unstated default: whatever you can get someone to do is valuable, unless it's illegal. And what becomes illegal is also governed by never-stated assumptions.
So we end up with this society where anything that can take up productive capacity is done. There's always some sort of justification that might sound reasonable in some context - privacy of personal data is indeed important - but we end up not balancing the costs with the benefits, and you end up with absurd situations where schools can't send out mail to their alumni (yes, this actually happened to me).
Attempting to address this isn't a waste of time or useless.
It makes no sense to blame the people trying to solve this problem, rather than those who created it in the first place.
If anything, the GDPR didn't go far enough. Though, that's often the case with legislation trying to stop corporate malfeasance.
In my opinion - and speaking more generally, not just regarding privacy - not enough large companies have been obliterated for their harmful behaviours. They just pay a fine, or bribe some officials, and move on.
It is if you fail.
It makes no sense to blame the people trying to solve this problem, rather than those who created it in the first place.
Now we have two problems: the original problem that GDPR failed to solve and GDPR useless warnings.
You say warnings are unlawful, and that's even worse, because that means they've created an unenforceable law. Or what are they waiting to prosecute those violations?
I'm not against data protection. I'm again incompetence.
Are you also against government waste?
The consent forms are just a tip of iceberg and the stupid thing. But under everything it forces companies to do things they might have not done before. And it's also a tool for developers who want things to be better. As there is finally slight risk to point out.
Exactly. If it went far enough the solution could have been much simpler. But instead of the real solution, politicians preferred the appearance of the solution which created enormous complexity and didn't solve the problem.
Bureaucracies grow because people demand them to grow, it's just not that obvious. Something somewhere is always wrong and unjust, so people petition politicians for change. The tool of a politician is lawmaking, so more laws are created. Implementing a law requires a bureaucracy. A bureaucracy requires managers. Managers want to hire more subordinates, not less. Subordinates are paid by the hour and promoted according to plan, so there is no incentive to work faster. In effect, everyone involved is always busy with something and nobody appears lazy or irresponsible.
I would argue that the tax system of (in this example) Australia is exactly what the Australians asked for, the system that they deserve. They just don't know they asked for it.
So when the Aussies get their tax system, they must have asked for it. Why? Because they got it.
This is an absurdity. Have you ever complained about the proliferation of utterly terrible reality TV shows? Very few people think it's anything but mindless garbage. But it's there, so it must be wanted, because it's there.
Plenty of other examples in society of things that exist but nobody appreciates them.
Anyway, been reading Mazzucato recently.
It is not a circular argument, it is a connection between cause and distal effect.
> There is no extrinsic value in the economically orthodox description of value that you describe in the first line.
I'm not sure what to make of this statement.
> So when the Aussies get their tax system, they must have asked for it. Why? Because they got it.
This is a misrepresentation. By demanding a more just system of taxation, they asked for laws, which ask for bureaucracy, which ask for labor resources, which consume tax revenue. All of these are logical implications, there is no circularity at all.
Similarly, if I ask you to serve me a hamburger, there's the implication that someone somewhere has slaughtered an innocent cow. I don't want the innocent cow slaughtered, I just want to eat the hamburger. Yet, it would be fair to say that I asked for the cow to be slaughtered.
> This is an absurdity.
Not at all. Usually there's a plausible rational explanation for why things had to end up the way they did, but such explanations might be uncomfortable.
> Have you ever complained about the proliferation of utterly terrible reality TV shows?
I don't watch TV.
> Very few people think it's anything but mindless garbage. But it's there, so it must be wanted, because it's there.
Very few people think McDonald's serves good food, yet a lot of people buy it. Similarly, nobody would continue to spend money to produce more and more TV shows that nobody wants to watch. Being "mindless garbage" clearly isn't an impediment to success with the right audience. Consider the pareto principle.
> Plenty of other examples in society of things that exist but nobody appreciates them.
Just because it isn't "appreciated" doesn't mean there isn't demand for it.
> Anyway, been reading Mazzucato recently.
Doesn't seem to have done you any good.
I really wish there was a 1-3 word phrase for this concept, because I keep needing one. I am hopeful that the field of AI Safety will give us one.
It's pretty simple really, if there's a way to determine value then there must be a way to say that it's correct in one way or another. If things are always just worth whatever someone will pay for them "because of the market" then you aren't saying anything at all. Plus you can't explain a number of common observations.
Why? Because you said so?
> The cognitive dissonance of not having a real theory is causing you to lash out at me, for no reason other than pointing out your missing theory.
I wasn't aware that you're a mind reader.
> It's pretty simple really, if there's a way to determine value then there must be a way to say that it's correct in one way or another.
There's a shortcut: I simply define value in terms of supply and demand, the exchange of goods and services, based on subjective disposition.
If you are clamoring for some other theory of value, feel free to do so. I think that's a waste of time.
> If things are always just worth whatever someone will pay for them "because of the market" then you aren't saying anything at all.
If you believe that, then any further discussion must be fruitless.
> Plus you can't explain a number of common observations.
I probably can, but you won't like the explanation.
Everyone complains about useless procedures, but I think the real culprit is that we have no framework for discussing value.
We do - money. That's why GDPR breaches cost lots of money - to make sure organisations don't allow breaches to happen.
My school in fact had my contact details from way before GDPR was a thing. Recently, they were forced to write to confirm that they were allowed to continue contacting me. If I had not responded, they would be compelled to delete my information.
So if I'd missed the message, I would unknowingly be disconnected from further communications.
And what is to be gained from this? All the risks of someone hacking the school system are the same, someone could still spam everyone, and so on. Why not just have an opt-out system, defaulting to stay-on-the-list? Then people who care to be removed can do so, but we can assume quite confidently that most alumni of a school want to remain in contact.
It just smells like someone had a use case they were thinking of that perhaps made sense to have a default-remove, but they ended up imposing it on everyone.
Unnecessary processes should be considered like unnecessary code - maintenance overhead - and so the usefulness of any process should with some frequency be questioned.
In some sense it would be even better for society if those who have lot of wealth risk even more into wild ideas and moonshot projects.
For example many paintings have huge price tag, and we could live without them. Many CO2 removal could devices are very useful but not financially sustainable ...
font-family: sans-serif; width: 80%; margin: auto; color: #1A202C; background-color: #F7FAFC; line-height: 1.5rem;
What about to play Go, to dispatch Uber drivers, to translate languages, to convert natural language queries into web searches?
Quite fun to read regardless!
Later he extended these ideas to an entire book [1].
The author's ideas are highly unconventional, but I won't say it's unreasonable or even uncommon in tech - Bill Joy's Why The Future Doesn't Need Us shares similar concerns, the author simply took a much stronger position.
[0] https://berglas.org/Articles/AIKillGrandchildren/AIKillGrand...
Maybe they are right. I often wonder about the point of building another CRUD form. We’ve taken paper forms and digitized the process, but haven’t really figured out how to avoid needing the form in the first place. Massive digital bureaucracies exist that are all about pushing around digital paperwork. It’s as if there was a twist of history that caused the development of mechanical horses pulling rubber tired carriages, instead of just inventing cars.
Most forms get digitized, then reworked later. Many digital agencies try to think beyond just forms and try to create smarter ways of getting data.
Naturally this is done for clients that spend ludicrous amounts of money to get there. So you'll never see them in governments.
Was totally thinking the same thing when I read this article. How could the above line be written seriously - PMs don't have guilt about destroying projects!