Everything is broken, and it’s okay
increment.com
increment.com
> Once you accept failure is inevitable and commit to avoiding the concatenation of failures that causes catastrophes, you—and your organization—need to decide what risk looks like and what to prioritize. If everything is the most important thing, then nothing is.
[...]
> The more complex the system, the more likely it is that some part of it is broken. As engineers, we’re at the apex of literally millions of hours of design and engineering time to help us, and our users, with everything from finding our phone to continuously integrating our code. We have to assume that some of that infrastructure, and some of our work, is broken.
> If we accept these imperfections, we can work toward building resilient systems that can handle a little static and deal with flawed foundations without falling over. We don’t stop working toward perfection just because it’s impossible.
The author's point isn't to throw up arms in defeat. Her point is to "courageously change the things you can change, accept the things you can't change, and find the wisdom to know the difference between the two".
There will always be tension between striving for the ideal world where everything is always written to perfection using the latest innovative technology, and the real world where legacy code and technical debt are par for the course. It's all too easy to compare software on which lives depend (avionics, healthcare,...) with video games, short-lived marketing apps or business suites. "value for your money" means very different things to different people in different contexts.
It does amuse me though that an ideal of perfection is having no legacy code, all written with the latest technologies.
I once spent some time refactoring some really old gnarly code to be beautiful and well designed. The rest of the team looked on in horror and asked why I was replacing highly reliable, battle tested code that had been in successful operation for nearly ten years!
And they were probably correct in doing so. If people are expected to be making changes to it, but it's generally working well otherwise, adding explanatory docs, notes and tests around that code would likely be far more valuable than rewriting it. You have to understand the problem before rewriting anyway - writing up that documentation and supporting evidence is often a better goal than "rewrite" when a system is working pretty well in service.
I've also seen the "don't rewrite it!" on code that is breaking/wrong constantly - like... daily. People have adjusted habits around it, and making a change is seen as 'risky' but there's a lot of unacknowledged operational debt that can be cleaned up with updating the code (investigate, doc beforehand too).
And trying to tell a client to 'accept the things they can not change'... doesn't often go over too well. Harder still in some cases because, technically, almost anything can be changed, but estimating the time/effort/cost is a difficult task in itself.
Beware the difference between being nice and being genuinely kind. The latter requires the courage to be willing to upset somebody and lose a lucrative contract.
It's important to acknowledge that and bring that to the table. Even worse, compromising for the sole sake of signing a contract is a risky proposition. Projects fail because visions and ideas are being sold without acknowledging the reality represented in the limitations of existing technological solutions or infrastructure that needs to be maintained.
The problem is more often than not, the guy who pays developer salaries wants the developers to spend less time on polishing.
The customers want the polish without knowing that's what they want. Customers want good software without knowing how to quantify good in software terms.
I run a Windows machine to play games on. I would dearly love to not run a Windows machine at all. But to play the games I want, I have to run a Windows machine. I'm not buying Windows because I love Windows and think it's amazing tech. I'm forced into buying Windows because the thing I want to do will only work on Windows.
I'm forced to consider buying a macbook again, despite hating how closed the OS is and how dependent it forces me to be on Apple Support's weird shitty lying statements. I have to consider this because I will probably need to develop an iPhone app in the near future and I can't do that on a non-Apple machine.
People buy tech to do things, not because they love the tech.
People pay for features first, and UX/UI second, if at all.
People pay for software that makes them money. When two competing software offer the same feature set, than and only then does UX/UI get any important.
UX/UI is not for your customers: it's for attracting pigeons to your CRUD website in order to sell them to your advertising customers.
I always loved the Erlang philosophy of let it crash. The idea you build a highly resilient system out of error prone parts rocked my world when I first encountered it.
None of this is to say we should accept broken things or not try to make them better. We should instead accept some parts will always fail and design accordingly.
While idea to construct everything from components overall seem to work better than to build monoliths quite often (Space X vs NASA, Linux vs many other OSes) but not always (e.g. SQLite).
You can and you must do your best, but you can't do everything - you must choose what is your sacrifice.
Verified* instances in safe Idris. Of course GIGO is still valid and failures in lower-level components (such as a processor just not behaving as specified) can cause failures in higher-level components, but it's simply impossible for there to be a failure in that component itself - any observed failure will necessarily be the result of a lower-level failure.
More likely, you're not unaware of these possibilities, you're just using a meaning of "failure" that excludes them. But regardless of whether you call these failures or call them something else, they prevent you from guaranteeing perfection.
And there is the cost/benefit calculation - the effort required to get something perfect is often greater than the benefit of having a perfect component. The purpose of commercial code is to make money, and if technically perfect code makes less money than technically imperfect code, then is it fulfilling its purpose perfectly?
Not my experience; the system may need new components or need to rearrange the existing ones, but the components themselves don't become imperfect. Consider something like a prime factorization algorithm; maybe your system will change to the point where you don't need it, or need to extend it, but it continues to be perfect.
> And there is the cost/benefit calculation - the effort required to get something perfect is often greater than the benefit of having a perfect component. The purpose of commercial code is to make money, and if technically perfect code makes less money than technically imperfect code, then is it fulfilling its purpose perfectly?
Tech companies go bankrupt all the time, so I think it's a stretch to say that commercial coding is working perfectly. Automating a sloppy human process in an equally sloppy way can certainly make you money, and maybe there's more money in doing that than in making perfect things, but we should at least acknowledge that there's an unexplored alternative here.
> Tech companies go bankrupt all the time
generally speaking they don't go bankrupt because of bad tech, though. It's usually imperfectly understood customer needs. No amount of perfect tech is going to stop that.
Currently yes; that's what I'm arguing could change. Every library has to be written somewhere, and at a lot of jobs I've found myself rewriting the same should-be-standard pieces again and again.
> The stuff that's developed in-house is to service a customer need, and customer needs definitely change over time.
Yes and no. There are definitely parts of the system that change over time, but a lot of what gets written in-house ends up being surprisingly basic/generic things like database mapping, job scheduling etc.. You'd be surprised how often there just isn't a publicly available implementation of the thing you need even when it really "should" be part of the standard library.
> generally speaking they don't go bankrupt because of bad tech, though. It's usually imperfectly understood customer needs. No amount of perfect tech is going to stop that.
I'm not convinced. The way bad tech generally manifests itself is that the company is slow to react to changing customer needs, because it takes too long to make changes to their systems. Having been on the inside, I've definitely seen things that you would think from the outside are business failures but had poor technical choices at the root of why they happened.
What does it even mean for an internal specification to be "wrong"? The specification for the system can be wrong, and maybe the component wouldn't be useful in a correctly-specified system, but even then I think it's misleading to say the component is "broken".
Yes, such as the part that judges whether something is fallible or not.
If anything, every research on complexity and correctness shows that it does not compose and does not survive in a dynamic system. I would love to see the research that support this claim. It would be a revolution in these fields.
I would like to broaden it a bit and find out if there possibly is some truth to it.
The bad news: it depends a lot of what you mean, it may not be the answer you search and i do not have a "perfect" reference, because it is considered mostly obvious in these fields...
But before i get there, a few things.
> (that we are building castles on quick sands with our leaky abstractions that try to fit into constantly changing reality)
Nearly all the references i point to will tell you that this leakiness and this constant changing reality is exactly what makes them work in the real world.
Now onto references. A lot of them can be found at http://resiliencepapers.club/
I particularly advise to start at https://www.youtube.com/watch?v=Pb_zYs8G6Co
https://www.youtube.com/watch?v=PGLYEDpNu60
For more in depth, i particularly like https://www.ida.liu.se/~729A15/mtrl/CSEnew.pdf
And this is basically a whole meta study of all the research and ideas that came to this field since WW2. Beware it is long https://www.sintef.no/globalassets/upload/teknologi_og_samfu...
For an example of how things go wrong when things are correct but interact well together, i advise these two papers on space systems. (I link to Adrian blog on them because they may be easier to read, but there is a link to the papers in there)
https://blog.acolyer.org/2017/11/30/the-role-of-software-in-... https://blog.acolyer.org/2017/12/01/analyzing-software-requi...
If these things make you curious for more, there is a lot. Or if you want to talk about/have help navigating what is in the http://resiliencepapers.club/ link, feel free to reach to me on twitter or by email. I will be happy to talk more.
The company knew the game was full of bugs still but were under immense pressure to ship out the game way before it was ready.
As an example, Android is Humpty Dumpty. Manufacturers just try to glue as many pieces together as possible.