Last week half of civilization came to a grinding halt because of the crappy engineering practices of a single company. I don't know if now is the right time to complain about perfectionism in IT.
In other words, it's not perfectionism that's the productivity killer they've identified, it's inexperience.
I'm a perfectionist in the clinical sense. Shipping something that I know is imperfect gives me a high amount of anxiety. But I also have enough experience to avoid wasting my time on things that I now understand don't matter.
I don't get anxiety about not having the perfect abstraction anymore, I get anxiety about things like whether we have enough monitoring in place, whether our release plan will cause problems in prod, or whether I considered all the edge cases in this critical piece of security-related code.
The perfectionism isn't gone, but it's not really hindering my progress in my career because it's focused on the things that actually matter. I'm working on addressing the anxiety aspects of my perfectionism, but that's a mental health issue, not a software development cycle issue.
The problem is that software engineering is still, in many ways, in a pre-scientific era [1]. For example, civil engineers know how to build bridges that are good enough, i.e. that barely stand.
In the software field, we don't really know how to proceed in the same way. We have formal methods, but they are quite costly, and therefore not used very often.
And "good enough" is a relative and changing thing. What was "good enough" thousands of years ago is nowhere near standards today. This is even true a hundred or 50 years ago with the specific example of bridges. Engineers learn this the hard way because the results are visibly catastrophic and usually through the loss of many lives. But I think one of the difficulties of software is that failures are often less apparent and can be difficult to accurately attribute. But then again, when a bridge fails there are both internal and independent investigations that are extremely thorough and result in updated policies. This does not seem to be the same in software (at least to nearly the same degree), in part due to the abstracted nature of how software can cause harm. It is actually harder to attribute error (like you said, formal methods exist, but are costly and not just in the monetary sense).
I think there is an irony in this because often someone managing a group will accuse them of perfectionism when they themselves do not have the appropriate context to understand that the arguments are really about goodenoughism. I do think this is why companies that are fairly transparent to their employees have been successful. Because it helps them understand the balance and constraints of which they are operating under. To have an environment which is closed and groups are unable to communicate (e.g. in a highly classified setting[0]), the person managing the groups needs to have intimate knowledge of all the pieces coming together, because somewhere along the line someone needs to glue all the puzzle pieces together.
So I wouldn't call what you express perfectionism. Perfectionism doesn't really exist because perfection doesn't exist. Rather it sounds like you are detail minded. It also sounds like you've learned to understand what details can (or have to) be ignored, what details can be temporarily ignored but need to be revisited, and to understand that keeping track of details is still important to understand how one needs to improve and mitigate future catastrophes. Obviously there is a balance, but I think this is very different from someone who is trying to turn a map into the territory.
[0] I should mention in many classified settings people are still encouraged to communicate and bounce ideas off of one another. Just to do so exclusively in secure areas and to ensure the other person also has the requisite clearance.
> the tendency to demand of others or of oneself an extremely high or even flawless level of performance, in excess of what is required by the situation. It is associated with depression, anxiety, eating disorders, and other mental health problems.
At my worst, every single mistake I made would give me hours to days of stress and anxiety. I've gotten it much better under control in general, but it still does cause a lot of stress from time to time.
This is why I object to people equating "perfectionism" with "being bad at prioritizing truly important work". I'm pretty darn good at prioritizing what's important. I actually perceive wasting time on unimportant work as a mistake, the opposite of perfection, and if I do it and realize my mistake I get stressed about it.
Perfectionism is holding yourself (and/or others) to excessively high standards, often met by anxiety or depression when those standards aren't met. The amount of perfectionism someone has is completely orthogonal to the question of how you measure perfection—a perfectionist with a definition of perfection that is well aligned with reality may accomplish a lot while a perfectionist with a definition that's less well aligned ends up wasting a lot of time.
And while you're right that perfect doesn't exist, that doesn't stop a perfectionist from pursuing it.
To be clear, I was trying to make the point that this is ill-defined due to what is a reasonable level of acceptance is difficult to define and how the common usage of the terminology can be ill suited due to this being dependent on understanding what one is conditioning on. I'm not saying you don't place too strong conditions, I don't know you. Despite running into you here often, I don't know that side of you and it isn't something I can reasonably conclude in either way. But I do know it is common that too lax of conditions are placed. It's a spectrum after all and either end is bad.
> every single mistake I made would give me hours to days of stress and anxiety.
I can definitely empathize with this and have been there.
I think you and I are decently aligned in our belief. Maybe we're bad at explaining and/or bad at interpreting. But there are enough similarities that I think we should not discount. But I do think we were using the word perfectionism in different ways. I was trying to use the version we'd see in normal language and how it is used in the article to agree with you in that this actually isn't perfectionism (certainly not the clinical definition). Though you're right to also point out that the clinical definition of perfectionism is not about actual perfection. But I don't think this invalidates my point that the true difficulty is in determining what is appropriate levels of standards, that this is a difficult balance to strike, and that many conflate comments of details/nuance with excessive standards when they may or may not be, including that the recognition of details is important in determining this. And I do think there is a natural bias in that it is easier to identify excessive standards than it is to identify insufficient ones.
experience without growth is just inexperience with more "years of experience".
This is pretty much what happens in more technical parts of industry when you have your best workers jumping every year, poor documentation of proprietary tech, and a lack of proper training because you expect "experienced" developers to simply grasp your entire tech stack from day one. You have people falling into the same pitfalls every year (or few months even) and building on a foundation that is not fully understood. Because there is no time to properly understand.
Maybe there should be some "perfectionism". Maybe not in shipping "things that actually matter" (as business defines it), but in the boring stuff that keeps a ship afloat. They could also simply give raises to their best workers instead of having them walk, but clearly that's not in the cards.
Assuming it is following, paradigms, design patterns and making the code "cleaner", then I would agree with the article 100%.
However if it was about actual engineering concerns like better error handling, performance, test coverage, nailing down invariants/validation/parsing, etc. Then I would say that we should probably be more diligent rather than less.
The issue is that the former gets too much attention and the latter too little.
I've seen first hand how someone writing a 500 line function struggles to test it. But splitting it up into more manageable chunks of functionality makes it easier to write a test.
Imo perfectionism is chasing that last 5% performance.
Clean code, design patterns etc are just good engineering practices... Not perfectionism ....
The reason I wrote the above comment is because I've seen this being flipped on its head too often by myself and others.
This is what I don’t like about software engineering. The dogmatism. The absolutism. The more I work in the field, the less I talk about “clean code”, “ddd”, “good practices”. I find myself saying more and more “it depends”, and more often than not what the business requires is not 5 layers of code in which the concerns are separated among dozens of files and following good design patterns, but a damn single file that get things done.
There’s place for what you call good practices and clean code, but many times those things don’t lead to good products.
Every single startup I’ve worked for has eventually come to terms with needing some best practices. In my view, there actually is a baseline: CI, partial coverage unit tests for the most easily testable code, and modularization/loose-coupling so that unwinding tech debt has linear rather than exponential cost. But even that is driven by my opinions about ROI, not theory in isolation.
Clean code for a bad product won't turn it into a good product. That's the core issue. These are engineering principles but you are looking at them through business logic.
The best metphor here is to compare clean code to insurance. You don't want to insure this? Okay, you can take that risk and maybe save if nothing happens. But if/when something happens it'll be more expensive than if you paid the insurance up front.
Meanwhile, Clean code from a good product will make it resiliant to becoming a bad product. I'm sure if you played any modern video games that you've had at least a few games that you loved but had horrible technical hiccups. Unoptimized, untested logic, crashing. The issue is some of these games still sell, so maybe the business logic prevailed over properly engineered code. But they are entertainment at the end of the day, not Crowdstrike.
Just don't do them for their own sake.
But the bulk of my experience is with small startups where the stakes are high; tensions around productivity are natural, and working through them is essential.
I've been using it for many, many years and it has saved me a countless hours by postponing abstractions, modularisation, etc. to the point where it actually emerges as a pattern in the software I'm working with. Let the code repeat itself, copy and paste, and only after some pattern repeat itself for the 3rd time I start to think on how to abstract, or modularise it.
It's always so much easier to rework repeated code/patterns into abstractions than fixing misguided abstractions done too early.
Now for a proper engineering answer, there's a few (but not limited to) factors to consider when determining software rigidity:
- Halo effect: How much code does this module affect? The more things that can break when the module fails, the more need to get proper testing/architecture.
- Frequency (in both ways): How often are you changing the code (is it cutting edge, or bog standard 50+ year old algorithms?) as well as how often the application calls the code. The more times its called or changed, the more you may appreciate ways to test new implementations with minimal impact to the current iteration.
- Understanding: pretty straightforward. The less you or the team understands the code, the more need to validate that code through tests. This often falls into frequency, because inevitable the less you understand the more you will need to tinker, possibly for unknown unknowns.
It's still subjective at the end of the day, but there are metrics you can use to make your case.
Given the choice between fixing a quality issue that can be easily measured/detected and one that cannot, a lot of people will prefer the measurable, easy-to-detect one.
So the folks on the 'code quality' committee end up introducing tools that make it impossible to merge if your markdown file has two consecutive blank lines - rather than spending the time on performance or security which are much harder to keep on top of with automated tooling.
sigh
Jordan seemed to be talking about it: prioritizing; helping, rather than burdening others. Gregor seemed to be describing a lack of diligence: not fleshing out learning, fretting over less relevant details.
In my mind arguing for dilegence is a better here. It implies that the product matters, even if it is imperfect. Arguing against perfectionism is bad. To some it will imply that the product does not matter, that sloppy work is okay if you can somehow justify it (getting the job done, doing more, etc.).
There’s an epidemic of “minimum viable” when it comes to software, and that results in flying really close to “not viable”.
And what do they all say? “We should have launched sooner.”
That’s what happens when the goal of the vast majority of startups is not to build an actual business. Their goal is to “get funded”. It just feels rotten.
Perfectionism is a disease
That results in other shipping sub optimal stuff
It’s perfectly ok to ship 50% of the features 100% well and keep shipping
And most people are wrong about which group they're in.
Crowdstrike had a sloppy process without meaningful testing, only parser validation. Assigning that task to tens of thousands of admins, most of whom have no idea what they are doing just shifts blame, not accountability.
Other engineering professions do have precedent shown for this.
I'm not convinced that either certification itself or the market represent the most effective means to achieving societal good. I think what history shows is that real consequences for shoddy work are required [1]. To date, software makers haven't really been held liable for consequences, but I expect that will change if we don't clean up our act.
[0] https://blog.codinghorror.com/do-certifications-matter/
[1] "229 If a builder builds a house for someone, and does not construct it properly, and the house which he built falls in and kills its owner, then that builder shall be put to death."
- Hammurabi's Code, ~1750 BC
Allowing individual engineers to be de-licensed creates a culture adherence to norms is the priority. If you ever worked with a civil engineering organization… it’s not exactly a hotbed of innovation. Alot of the professional associations spend more time creating make-work regulations than actual safety or engineering. For example, in New York, something as trivial as refurbishing a moderate volume street crossing in an urban area requires $750k-$1M of services to address ADA compliance, multi-modal transit accommodation and other factors.
Perfectionism doesn't necessarily mean better implementations, good practices, or even doing the right thing. Chasing and stacking needless abstractions to make things appear clean is an example of such perfectionism that actually harms the industry and erodes standards.
I would argue that perfectionism is also about an inability to prioritize, something which, in the real word with all its time limits, will bite you.
I got you! It’s relatively new, published four days ago. The relevant section is called “what went wrong and why”:
https://www.crowdstrike.com/falcon-content-update-remediatio...
It’s a hard problem - the whole point of Crowdstrike is to quickly respond to threats. (And they are very good at it) So you want time to market for marginal changes to be very fast. Normally I’d give them the benefit of the doubt, but their arrogance and poor response triggers my spidey-sense.
On the other hand, I couldn't count the hours of my career that I've wasted perfecting some piece of functionality - either at someone else's behest, or due to my own motivation/professional pride - that ultimately nobody gave a shit about.
I don't really want to do that any more, and I get pretty fed up pretty quickly when some PM (or whoever else) pushes me to do something that I know ultimately isn't going to matter. I can add a lot more value to a team than by simply churning out code, and it's frustrating being in situations where I'm not allowed to do that and am instead forced to waste time endlessly fiddling with stuff that doesn't add a lot of value for users and customers.
And I'm not talking about compromising on UX or anything like that: I mean actually spending time building functionality that doesn't get used. Or spending too long on barely used and non-key workflow aspects[0].
What's the point?
[0] A really great example of a barely used but ABSOLUTELY CRITICAL piece of functionality is the export/publish functionality in Ableton Live (it's a while since I used it so I can't remember exactly how it's labelled). This is a piece of functionality that is barely used compared with other functions, at least by many of us, but it's also - in some sense - the whole point of the application in the first place because it enables you to export or publish a finished piece of music, so it needs to work and work well.
It wouldn’t have existed lol
this was not the failure of a single company.
1. Crowdstrike 2. Microsoft 3. The companies that think it’s a good idea to run what is essentially an enterprise rootkit
If you want to build value you must ship, even if the thing isn't ready yet because otherwise it will never ship and there will never be any value realized.
Sarcasm aside, you're not wrong about people being caught up in perfectionism but that doesn't make you right about people/managers conflating perfectionism with goodenoughism. I'm sure with this error (and many others) you can find someone who was trying to solve the problem/s and someone else that accused them of perfectionism. I rarely see actual perfectionism, but I do frequently see people disagreeing what is good enough. I also frequently see problems that would have been small at the time but to fix later on require massive expenditures. The problem is that not-good-enough compounds while good-enough is neutral. You need to also dedicate time to maintenance and repair, because that's what fixes not-good-enough, which frequently happens simply because we aren't omniscient.
If you're going to measure the dollar value cost of their errors as a cost of CrowdStrike being in business, shouldn't you balance that against the dollar value of savings from the attacks they've stopped?
Just comparing against the attacks they stopped is a naive and gross overestimation because it requires an assumption that if a customer did not use CrowdStrike they did not use _any_ alternative. In reality, Crowdstrikes customers would have used some other product. So the attacks that competitors could have also defended against cannot be accurately attributed to Crowdstrike. i.e. if Edison didn't invent the light bulb someone else would have. We know this because he didn't invent it, though he did make the first practical version[0].
But since we're on topic, I'd call what I suggested a "good enough" estimation. We have a reasonable estimate under reasonable constraints. It's more complex, but the amount of accuracy gained is tremendous. Arguably the other model is so inaccurate it is less than meaningless (it isn't even "back of the envelope") because its conclusions would lead us to believe things that are not true, and far from the truth. But if we want to talk about _actual perfectionism_, well this gets much more complicated much more quickly. In fact, gets exponentially complex in exponential time.
A true analysis needs to consider the counterfactual question of what if Crowdstrike did not exist at all. This is far more complicated and contains many subltities that could dramatically change the results. Because we need to understand things like how other companies would have advanced and developer were the money/manpower/resources that were allocated to CrowdStrike were allocated to others. To understand how CrowdStrike both creates and discourages competition in the space. Systems like these are (mathematically) chaotic. Who knows, maybe a manager at Crowdstrike pissed off an employee who then turned blackhat. But maybe Crowdstrike gave a job to someone who would have turned black hat would they not have gotten hired. Maybe because they were laid off from a competitor! Realistically performing these calculations to a solution that is accurate (which would still be probabilistic in nature) is intractable. Of course we can improve our earlier model by using some of these factors, but I want to illustrate that there's a difference in perfectionism which is unachievable from being reasonably accurate ("good enough").
[0] Left unknown is how quickly someone else would have come to similar practicality but my best understanding of this specific story is that it would not have been long. This is not an uncommon story in science and technology and if you look hard enough you'll almost always find that any discovery or invention was also being pursued by another group. There are exceptions, but they are rare. You just never hear about the other groups, but you can confirm this by watching advancements that happen while we're alive and can see the competition in real time. Specifically in domains where you are close to.