Balancing “If it ain’t broke, don’t fix it” vs. “Release early and often”
redhat.com
redhat.com
I don't think there's much consternation over actual bugfixes. There's consternation over constantly-shifting UIs and APIs.
The only time there's any tension between those two things is when people conflate bugs with poor design. The messy but correct thing to do when you've correctly implemented a poor design is to maintain backward compatibility while also offering the option to use the newer, more correct behavior. See, e.g., strcat() → strncat() → strlcat()
OpenBSD claims it first appeared in Version 7 Unix, which was released in 1979.
https://man.openbsd.org/strncat.3 https://en.wikipedia.org/wiki/Version_7_Unix
For simple systems, yes, test all components, build in redundancies, etc. All those things that made reliable system design possible in the 1950s.
For complex systems you're quickly wasting your time on diminishing returns. The failure modes of complex systems lie in obscure interactions between components conditional on rare input combinations. You have to find that stuff out in production. You have to make users into testers.
However, there are better and worse ways to do this. Ideally, a failure should only be experienced by a user with a fallback solution, not a plain failure to accomplish the task entirely.
We landed on the moon with a reliable computer and reliable software in the 1950s. What software that you've written is "far more complex" than that?
(Personally, I'd posit that a lot of the complexity is the ten or fifty layers of framework and language and library between the programmer and the computer. But I digress...)
I worked for a corporation that was so change-averse, that we needed to buy our development systems on eBay.
They actually had a point, and it was hard to argue against, but I found it absolutely infuriating, as the solution was to plan for change, and establish a process to be constantly evaluating and refining for new developments.
Instead, it was "Wait until we can't bear it any longer, then have a huge screaming match meeting." A new budget would be approved for new machines (and, thus, new operating systems and development tools), processes would be updated, and that would sit, until it was no longer new.
As I said, it was hard to argue against, because it was a 100-year-old company that had been successfully delivering really high-end stuff, since Day One. They did that by being so conservative that they hadn't discovered fire, yet. Measure 300 times, cut once, etc.
When there was a problem, it escalated quickly (and was used as fuel to tighten things down even more), as the company was held to standards that are probably up there with NASA.
Speaking of NASA, I can't help but notice that this rather plucky little outfit, called SpaceX, seems to be running circles around them. They seem to have figured out how to "fail fast," yet also deliver insanely high Quality stuff.
Might be worth ignoring their CEO's tweets, and look at what they are doing...
I honestly think the idea of "if it ain't broke, don't fix it" is pretty silly as applied to large, engineered systems. If your system is more than 1000 lines of code, I guarantee you have a latent bug somewhere. Even if you don't have any bugs, the ecosystem around you changes so fast that you eventually will.
I mean, as vendors, part of the debt we inherit is that of our customers. Then our customers switch to something else. As vendors, how do we support both the past and the future? By making changes opt-in if possible, but still, that technical debt is there, and requires devs and other support staff.
I don't work in anything rocket-science, but I stopped asking clients if they want to upgrade. We decide when to do it for them, and we take the blame if something goes sideways.
I still prefer to have something break, than lose a client because they didn't realize they could upgrade instead of opting for another product (and they probably silently grew resentful at the lack of features of the old version).
They're already 20 years in. Wonder if there's anything left from the beginning (or if it all got blown up).
Those aren't companies that build anything lasting or environmentally friendly. It's just all throwaway toys.
Is your stable Univac mainframe application broken? Operationally, probably not. But your business cannot adapt becuase it relies on a platform for which everyone who knows it is dead.
And if SpaceX had gone to space before NASA it would probably look like NASA could run circles around SpaceX...
"Fail fast" is great when the only thing at risk is your own money but when you throw human life into the mix not so much.
How do you think their fatality rate will compare to that of NASA?
In software, I've seen repeatedly how new hires with big ideas (and youthful confidence that they know a better way of doing things) will come into a company and want to rewrite things "properly." In one management position, much of my role involved the politics of defending a well-working, well-maintained, big-revenue-driving application from a steady onslaught of such ambitious exuberance.
At the time, I thought "this must be the same phenomenon that drives shrinkflation": new MBA arrives at pickle-making company, seeks to bolster their career by demonstrably saving the company millions, convinces execs to put one less pickle in each pickle jar, consumers won't notice; step 3: profit!! In 2022, when you buy a cereal box, it's half air on the inside. In 1980, it was only 20% air.
I also shake my head when media pundits equate the success of a particular congressional session with the amount of new legislation they pass. I can imagine no simpler way to cruft-paralyze a democracy (while following the rules) than "releasing laws early and often."
They may increase profits for a short time, but as consumers realize what's happened, ultimately they see the brand as not having standards and lead to it becoming a commodity. A small group of long term invested owners would never do such a thing, because that would be 'killing the goose who laid the golden egg', but modern shareholding has no loyalty to quality.
In my experience, these new hires are right a certain percentage of the time, and wrong a certain percentage of the time. I can take no default position here.
I tell new hires that they're gonna see something that offends their sensibilities about the way code should be written. That puts them in the center of a conflict. On the one hand, they believe the code should be written differently. On the other hand, someone else wrote the code that way for a reason. Their first step is to dig in and figure out the reasons on both sides--why it was written one way, why they think it should be written another. The second step is to figure out what the consequences and risks are of making changes or not making changes.
The way I think about it, I want to simultaneously keep the software working, and protect the "flame" that new engineers have to improve things--something which can be all too easily extinguished.
Newbies may be right in that a system would be better rewritten; but often wrong that they think they can do it ‘properly’. At least, if they dive straight into it.
So making them use the Charleston’s fence approach is a good one.
Libraries should be much more conservative. I don't think "release early and often" was said about libraries.
And yes, it's tricky that it's not always a clear line between these two categories. A unix command line utility is sort of top-level software, but also likely to be use in a scripted/automated fashion.
So it's not always cut and dry, but the more you are aware of software depending on your software, the more careful you should be with releases.
As a user, I also hate "release early and often".
In both roles, releasing early and often effectively means I'm always using beta software, with all the headaches using such software brings.
However, as part of an internal development process -- that is, before customers see the release, "fail fast" is a totally legitimate and decent approach.
A new library should iterate rapidly and be very clear about that. Big warnings, obvious version numbering, and good communications. You look for an audience of actual users that can keep up with that. That's how you figure out what the right library is, both in terms of the big abstractions and the small details.
But then once you hit a 1.0 release, things change. You're shifting from an audience of fellow explorers to an audience who wants stability. You can still do exploratory work, but it has to be additive in the 1.x series, and eventually you need to start a 2.x series as you learn more about what needs to change. So the iteration still happens, it just happens away from the people who just want the basic thing you got right via the initial burst of iteration.
If I work on a test system, or a system a small number of people depend on... a year or two ago this would've hurt my pride, but why spend 4 hours planning a change, if you can just muddle through problems in 2 hours. The user impact will be none or low based on the assumption, so move quickly, break things, fix things, document the problems and fixes. Everything's good.
Yet, I also have systems a dozen teams or more rely upon. For these systems, we have to move much slower and more deliberate. We cannot touch some of our database with larger changes, unless we have a way in, a way out, and a prediction how long all of these take, as well as announcement to the customers, and so on, and so on. We'd love to update those faster, but it's rough.
1) PO wants feature X for customer P.
2) Architect in meeting with PM and PO indicates that feature X is a simple addition to Y
3) Engineer is assigned X and says no this isn't a simple addition to Y because on deeper inspection it will fail for cases 1,2,3, we would need a different architecture.
4) PM says build it the naive way, let the customers find case 1,2,3 before we fix, we are an Agile team after all.
5) I quit.
Why PM has an option to choose here?
> will fail for cases 1,2,3
a) tip QA to test these cases
b) add cases 1,2,3 to "Known issues"
After all that is the reason they call it "beta": "Cause it beta then nothing"
To play devils advocate though, maybe case 1,2,3 are low enough risk to release. Having metrics set to watch if these are actually big problems could be “good enough.”
The first time this happened was a physical product with intention to sell to a very large American company with which my company had a relationship going back decades. I did as I was told (did not get to step 5 until after release, there were steps in between 4 and 5 which caused me significant political blowback). Ultimately they did not appreciate the corner cutting and declined to purchase. My quitting moment came when I was asked to be involved in mislabeling product to indicate it was up to standard X when it did not come to our supplier that way. The pressure I was put under to do that put me in therapy. I was a recent graduate with a young child, tough times. A year after I quit, my former company was parted out.
The last time this happened was software used for planning national infrastructure. Case 1,2,3 were common cases that would give very obviously incorrect results. The naive case was if a sliding parameter was always set to one particular value, opening that value up to change was a can of worms that required a whole new architecture. Ultimately I held my ground with the PM and said I would not build it incorrectly and implied it was a dealbreaker for me. I convinced the architect and PO of the correctness (PM was former frontend and had no clue what any of the backend stuff meant). Ultimately I ended up on the PM's (my manager) shitlist and the PO (apparently with the memory of a goldfish) couldn't understand why that feature slipped several sprints. I had better things to do so again I quit. I don't really expect that company to last too much longer at least in its current form. There were already "pivots" on the horizon which gave me deja vu (I expect company owner was gearing up for a sale).
In both cases there were pretty bad shakeups (unbeknownst to me) in the year prior to my joining. In the last case I would have appreciated a friendly heads up from a buddy at the company (they no longer had a real QA dept, they were all fired for "poor communication").
I guess I joined these companies because I have a soft heart for companies with deep history. I certainly learned a lot but my beard is a bit greyer than I'd like for my age. I think the rot was too deep to point out in each case without royally pissing of some of the longest tenured, better than betraying my ethics I guess.
I've had good work in between the first and last where my raising the flag was taken quite seriously and procedures were updated to mitigate, but still I feel I have a black mark on my name.
- Every company is navigating the marketplace, and making decisions with imperfect information
- Not every decision will be perfect (or even, good)
- Not every decision-maker will be perfect (or even, good)
- Even a collection of individually smart/reasonable people, can end up collectively making pretty awful/illogical decisions
- "Good" decisions don't guarantee market success; conversely, "bad" decisions can still result in good outcomes
- Judging the quality of decisions and decision-makers based on outcomes, is an imperfect measure of the actual "quality" of those things/people
In your first example, you were presumably an entry-level engineer, but you either mistakenly took on too much burden (emotional or practical) in terms of decision-making yourself, or you misunderstood what types of expectations you should have for the actual decision-makers.
Decision-makers are allowed / expected to make such bets: "how many and which corners can we cut as a company, to get a product out to market, that clients will want to purchase, in a sensible time-frame?" This is not unusual, this happens all the time, at every single company, all around the world. The companies who do this more successfully, are the ones who find a sweet spot between cost-cutting, efficiency, time-to-market, and customer demands/satisfaction/delight. This is a very difficult thing to juggle, and really really smart business leaders consistently fail to find the right balance, or make the wrong calls. Hopefully the mistakes aren't fatal to a company, but unavoidably sometimes they will be. So yes, your company leaders made a bad call based on the outcome, but that on its own is not enough to indict the decision or the decision-makers as being fundamentally wrong.
The fact that your company made a set of decisions that ultimately led to failure, doesn't necessarily prove that they were a bad company. And to be a devil's advocate for a second, even "mislabeling standard X" might be forgivable under certain circumstances, such as launching a product with an "X pending" label, even though you didn't finish certification process for X yet, or maybe you didn't even start (but hey not starting doesn't mean it can't say "pending").
As a manager, I actually actively filter-in for what Amazon would call "Have Backbone" as a value, when interviewing engineers, and I ask them to provide examples of times where they fundamentally disagreed with the product team, disagreed with what they were asked to build, disagreed with a proposed architecture, etc. I want engineers on my team who will speak up, who are opinionated, who care enough about their work to take pride in it and put forth effort to improve beyond the status quo.
That being said, your examples seem to indicate a rigidity of black/white thinking, all-or-nothing thinking, and an inability to collaborate towards finding a solution. These were probably the most extreme examples you had, so I'm not judging every interaction or your entire personality as being so rigid, but hopefully you have by now experienced other examples in your career, where collaborative problem-solving was possible, where you did more than point out fatal flaws but also helped formulate a path to mitigate or solve them. The companies where that was more encouraged or made possible, are the ones you probably want to work for.
- Are cases 1,2,3 named that way, because they are the top priority cases (i.e. the #1, #2, and #3 most important product features that customers care about)? Even if they are, what is the cost of a new architecture? Will it take you 3 years and 20 engineers, to rebuild Y or to make Y.v2, just so you can support X "properly"? By then the market may have moved on, the feature may be worthless, so it may make perfect sense to deliver a bad version of X that relies on Y.
- Or, are cases 1,2,3 legitimately either rare, or low-impact, or do have viable manual workarounds? If so, then it's entirely reasonable to defer/punt on doing new architecture right now, because either you know these cases are unimportant, or at least you don't have positive proof that these cases are important enough to justify new architecture. With more data, or clear customer demand, you can make a better case for rebuilding Y "properly". The real problem comes later: what happens if you do get strong signals of customer demand, you can prove the current solution is not scalable or extensible, and yet the business still decides that Y is good enough to never touch... well that's a business that doesn't want to stay in business.
Agile is about practicality/pragmatism, over adherence to dogma or preconceived notions. Just because Y is the wrong architecture to deliver X, does not mean it is the wrong decision to ship partial feature X. Don't be dogmatic about "correct architecture", if you care about for-profit software engineering as a profession.
Of course, if your goal is different, if SWE is a craft or a hobby or an ivory tower pursuit for you, then feel free to make whatever decisions you want that don't fit your vision of "correctness".
I think there its a little unclear that by case, I mean testcases that would fail to pass to fulfill a single feature. E.g. "I need an addition feature for a calculator" but naive implementation will result in it working for 1+1 and fail for all others.
"If it ain't broke don't fix it" => don't try to "optimize" something that works well. Maybe it will be able to handle more load after your big refactor, but if it already handle the load nicely and there is no indication load will go up, don't bother.
"Release early and often" => release after each change, don't bundle everything into a big release that happen every 6 months.
The first one is about deciding what to do, the second one is how you release the work you did.
After years of dealing with software that does this, I've soured on it completely. It has resulted in greater difficulty using software, and the software being less solid.
My early career was mostly fin-tech and when installing into banks or other financial institutions, they _very much_ want the every year or 6 mo release cycle. And not just because of "old stodgy", but rather they want a good bit of ramp-up time to ensure things are running right. In a Wall St trading firm I worked at, a release ran in parallel with real data for MONTHS before the old version was retired.
So, I'm used to that model.
From the customer's standpoint, they could "gear up" for the next release, get their ducks in a line, get the right people to assess things. I get that some of that is because the infrequent releases are "big", but also as a human customer it's fun to see a shiny new truck toy rather than a coat of anti-rust paint on its exhaust pipe, then a replacement for the broken window crank, then a new color on the odometer numbers, then ...
As a developer I also miss the Big Project Ramp Up, then a period of a lot of work (which can be Waterfall, or Agile, or bespoke artisinal project management, which is pretty much 99% of the time, just CALLED "Agile"), then a Big Project Release (+ party, some down time, and prepare for the next one).
SAAS release early/often is like death by 1000 boring dull cuts. They cut out the spikiness _so much_ that there is so little novelty that it's just not fun to work on it.
One of my suggestions is that any component that you avoid breaking, deserves to be broken the earliest. The rigidity (the resistance to change and the inclination to break) of code is a sign of greater trouble in the future, so should be addressed the soonest. That means, I don't agree with the "don't fix it if it ain't broken" sentiment fully. If that component's so valuable, it must be covered by hell a lot of test coverage to make it impossible to break it with changes in the first place. If you can break it without knowing, that's a problem that needs to be addressed immediately. I say, break it, identify breaking points, add tests to those points, so the component becomes flexible.
My second suggestion is to make the changes, but discarding them after. I also emphasize this in the section titled "Write it from scratch". By doing that, you gain certain level of insight into that rigid design, and improve your understanding of the code base, which eventually leads you to the phase where you can finally code changes that you're comfortable sharing with your colleagues.
You may think of the writing code only to throw it away a wasteful exercise, but I argue that it's not as great a loss as you may anticipate.
server folks want nothing to change, ever.
desktop folks want the latest and greatest.
mobile folks get spurious changes forced down their neck and want... well who cares what they want, this is what they are getting.
In Kaizen, "Fail Fast, Fail Early" would never get ouf of the gate. The game here is all about making other, smaller bets which gradually ratchet the company to a better place without ever having to come off the rope.
Perhaps this organisational conflict is rooted in the realities of burning through capital whilst still searching for a sustainable revenue model.
In many ways the idea of Scrum is very Kaizen-like. It mandates the idea of experimentation and measurement. With trying stuff even when what you have appears to be working. And with documenting failed experiments and moving on. Nothing risky about any of it but you wouldn't know it when you see the typical reaction to "how about we move the standups to the afternoon instead of first thing in the morning?"
The innovation pipeline described is encouraging but it's already too dogmatic in its proposed implementation. Kaizen is much more organic and, dare I say it, cultural. One of the key principles of Kaizen is empowerment and the innovation pipeline doesn't offer that at all.
Would you fly on an airline that followed this principle?
Wait till the engine fails before fixing it.
In the air.
Short cycle times is also why I use a rolling release linux distribution (Manjaro) and browser (Firefox). Always fresh and up to date. And even though I'm on the Firefox Beta channel I never have to deal with it breaking or being unstable. It's stable because they have frequent nightly builds. By the time builds hit the beta channel they are already rock solid. I was on the nightly channel for a while and never experienced many issues there either. Great example of short cycle times. With the Beta channel I'm a few weeks separated from changes happening and me seeing the feature. With the release channel it's another few weeks.
Not updating because it aint broken is very valid until the time comes when you finally have to upgrade and all hell breaks loose because you are two years behind on dealing with breaking changes and have to do a massive project to make it happen. It was getting increasingly more broken while you were doing nothing; you just did not know about it. It's still technical debt. And now you get to deal with the non linear effort to fix it and pay the price.
So, on all projects where I'm in charge we update everything very frequently. If something doesn't work I want to know ASAP and mitigate now instead of not even knowing stuff is not going to work for another few years. If you stay on top of changes like that, the effort for this is very low. Mostly stuff just works. Occasionally some library has an issue. And then we fix it, work around it or wait for the next version (and document why we can't update). Easy stuff. Basic project hygiene. The first thing I do when working on a project I haven't touched in a while is update dependencies. If I'm working on it all dependencies have to be current. I get annoyed with being a few minor versions behind. I might wait a few dot releases with major releases. But generally, I want to get that over with ASAP. If it breaks, I'll at least know that I need to deal with that. Rolling back is always an option.
Yep. Open source is not from enough, lean and stable in time open source is required.
Why not do proper testing ? I know it's expensive. And when you have an OS as an init system is even difficult. It is sad the the UNIX philosophy is dying being replaced with the Windows philosophy.