Responding to “Are bugs and slow delivery ok?”
uselessdevblog.wordpress.com
uselessdevblog.wordpress.com
In the 30 years I've been coding, languages have come and gone, platforms have come and gone, frameworks have come and gone, methodologies have come and gone, but two things have never changed:
1) Software development is wildly unpredictable. I've never met anybody who could reliably predict how long something would take to develop, even if the requirements were "frozen" and even if they were the person delivering it. I have met people who could predict ahead of time how long it would take them to deliver something, but what they delivered always had "bugs" (either actual incorrect behavior, or edge cases that nobody thought of until they actually used the software) that themselves took an unpredictable amount of time to fix.
2) No organization can ever accept any amount of unpredictability. Every "advance" in software development I've ever seen has claimed to (implicitly or explicitly) remove the inherent unpredictability in software development, although none have actually delivered. They're fine with spending months in planning meetings while not actually developing anything tangible as long as those planning meetings provide the illusion of predictability.
I remember working on this one major project - an online music store - and one of the execs hired his very lovely wife to project manage a team of developers. She would create these fantastic charts in Microsoft Project and every day try to somehow - somehow - get the developers to either (a) give her timescales for each component, or (b) explain why previous component X took twice as long as they had scheduled.
I've been writing code for 40 years and I still get caught out myself. I try to give realistic deadlines and then some fucking bug in the compiler or other such insanity will come along and make me look like a fool.
Call it immaturity, or perhaps just not being a "team player", but I've been coding for 15 years and stopped giving time estimates about 5 years ago.
I simply refuse, and instead, explain that I would just be making something up, and I'm not interested in taking responsibility for an arbitrary guess, since I know from experience how it can come back to haunt me.
I try to suggest saving such speculation for a time, hopefully in the near future, where the initial research has been completed and _some_ progress has been made on the foundation of the requested work, so that my speculation can have some basis in reality.
If all else fails, I give a wild estimate that I don't expect anyone to agree to, such as 2 years for work that might not even take 2 or 3 months.
Things still go off the rails of course making an an imperfect solution, but it ends up being more right than any other method I’ve tried.
*If there are any unknown unknowns I might start multiplying by the number of big ones.
Nah, it can happen through sheer chance. I remember hitting a compiler bug when I was young (the only time I hit a compiler bug). I was certainly not particularly smart, just unlucky.
The interesting thing is that it had been drilled into me that "it's never a compiler bug" and so I banged my head against that wall for a couple of weeks before a more experienced programmer took interest and showed that it was actually a compiler bug.
Definitely not in my case -- sometimes we just stumble over these things. As an inexperienced programmer (around 1989 or 1990?) I found a bug in the beta of microsoft's first C++ compiler, shipped on a dozen 5.25" floppy discs. All the other developers around me said "Don't be stupid. It's your code, it's not the compiler!" but I distilled it down to a minimal test case and proved it. The bug?
i=i+i;
That's it. It would give random results based on prior instructions unrelated to i.
I reported it and got a nice note back from the MSFT engineering team. Something like "Wow! I've no idea how we missed this. Good catch!"
It's been a while so I don't remember all the details but I remember an ancient version of gcc refusing to accept perfectly valid zero initialisation of structs. Not terribly exciting but you don't have to be too clever to hit compiler bugs.
The planning fallacy is something like "Humans are hopelessly optimistic. Even if you think you're planning for the worst case, unexpected things will happen, and your estimate is more like the best case scenario. This is true even if you try to compensate for the planning fallacy by being more pessimistic".
So you can't fix it by padding more weeks, trying to think of all the things that might happen, etc.
The only way to compensate for this is detailed in Kahneman's "Thinking Fast and Slow" (Must-read for many reasons). That is to start with prior evidence. Rather than making your estimate, start with "How long did it ACTUALLY take the last time someone did something like this?" and only then modifying the estimate for circumstances.
There is a huge gap between these two methods:
1. "How long does it take an average team to create a SAS product? Maybe a year?" Then modify for "but our team is extra smart, so maybe I'll shorten that to 10 months"
vs
2. "How long will this take? It's not a complicated product, and our team is smart: maybe 4 weeks for feature X, 6 for feature Y, 2 to deploy. I'll even throw in an extra 4 weeks of padding for unknowns, so 16 weeks".
The first is that you can predict that there will be unforeseen problems, but not how many of them nor how long each will take to resolve. The result is high variance. Then if the average such project takes 16 months, and you predict each such project will take 16 months, then some take 7 months and some take 17 months and some take 25 months and people get mad at you for underestimating some of them. But if you predict at the upper end of the range then people will get mad at you for giving 25 month estimates for projects that only take 16 months on average.
The second is that people know what the estimate is and work expands to fill all available time. When you know you have 16 months, you set a pace that will have the project finished in 16 months. Then in month 14 you encounter some unexpected problem that takes six months to resolve. If you'd found it in month 2 you'd have been fine, because you'd have cut some less critical features and spent the time on that, but now that's in the past and your project is five months late.
Project Manager/Scheduler/Sales/Customer doing one/any/all of: not believing the input of the SME (you) when they ask for input developing the schedule, overpromising, expecting the world, or so many other problems.
Too many times in my own field (heavy construction) the scheduler has asked me how long it will take to do a given task, and it is actually reasonably quantifiable to say "It will take my crew X days to move Y volume of material at Z/shift" only to be told "Well our proposal only allocated X-5 days and budgeted a Crew of <your chosen size - 4> for that task. Figure it out."
Haha, I came to that same conclusion decades ago - I don't know if it would actually produce meaningful estimates or not, though, because every time I try to estimate that way, somebody more "important" than me comes along and throws out that estimate for one that fits the timeline he already had in mind.
This requires to have prior evidence within your team to start with as not all teams have the same velocity.
Agile, for example, suggests that decreasing the cost of change at the expense of certainty of cost will increase the total chances of delivery within cost and date. Outsources are (implicitly) arguing that the decrease of total costs are worth the other friction in the organization.
This all goes back to return on investment. In the most simplistic case, if I have ten people working on this task now, and this software will allow me to reduce that to six, then that's four people I don't have to pay for that task. If those four people are making $100,000 a year, and the software lasts 10 years before having to be replaced, that's 4 million dollars. The software plus maintenance needs to cost less than 4 million dollars to make sense. We all know the simplistic case.
But if I can trade $100,000 dollars in guaranteed cost for a reduction of $500,000 in risk in making that goal, I can trade cost for certainty and that's going to be a good trade. Now, as developers we know that these numbers are often fudged, but everyone realizes this is inexact and just the best we all can do. We need to know how to think about these in terms of arguing against bad investments and steer toward good investments by knowing how the tradeoffs works.
Yes, that's the promise. I've seen it tried as earnestly as possible over the past 20 or so years since the term became popular, but I've never seen it deliver on its promises either. I suspect that a lot of that is psychological - risk-averse management hears "on time and on budget" and consciously or unconsciously starts steering the development process back to a more predictable seeming waterfall style that negates the supposed benefits of "agile" development.
I assert, however, based on 30 years of observation, that no matter how carefully you follow an agile delivery framework, whatever that may be, actual delivery and cost will have no relationship whatsoever on your estimated cost and date.
But I also predict that you and I will continue to smile and nod when they insist that it must be finished by such and such date.
The real problem is that management think they can directly constrain those kinds of variables but in reality they are not inputs they're outputs. The real inputs are things you can actually control like what you're building and which of your team you assign to build which parts of it at which times. That might come in on time and under budget or it might not but shortening a timeline or decreasing a budget won't change how long it takes or what it costs.
Of course that sucks for managers who want to make informed decisions like whether to go ahead with a project that will require an uncertain amount of investment to complete but needs to generate more return than some threshold to be worth doing. Now the managers have to find a way to guess what the important outputs like cost and timescales will be as some sort of probability curve so they can make a rational decision based on the best but still incomplete information they have available. And even then a lot of managers insist on trying to be absolute about things like time and money when the actionable decisions they need to make are more often whether to start some project at all or which ongoing projects currently have the greatest need for any additional resources that are available.
1. When doing estimates, estimate hierarchically and break down every task until the leaf nodes are 4 hours or less. If the leaf nodes are a week of effort, that means that you don't yet understand fully what the task entails. If you don't know what the task entails, it is very difficult to get an accurate estimate.
2. Maintain a database of various projects and features, and use them for comparison purposes. If you know how long something actually took in the past, you can much more accurately how long it will take to do something similar again.
3. Include specific time for various types of "overhead": meetings, code review, writing tests, debugging, making presentations, etc.
4. Derisk ahead of time things that are truly unpredictable. Doing something in a brand new programing language or framework, training an ML model on a new type of data, working on datasets of unknown quality, etc. For those you need to estimate out a project to determine feasibility and generate an estimate.
The main thing is that this is clearly not impossible. Years ago I got really interested in the Apollo program as an example of bringing in perhaps the most complex and risk-laden project ever attempted, on time. One of the things that clearly stands out about that project was that there was huge derisking happening first. At one level you had the Mercury and Gemini programs that were figuring out human spaceflight generally, and docking with Gemini, and also things like the first stage engines of the Saturn V were already in development in the mid fifties.
How long did that huge derisking take? What was the budget for the time spent doing the derisking? I'm willing to bet the "huge derisking" ended up taking longer than the actual project (and was itself open ended and unpredictable).
AFAICT it's actually worse - the exploratory phase always seems to get bogged down by theoretical discussions and imaginary scenarios that don't end up being very important (and putting off how long it takes to find the scenarios that actually do).
Where it makes sense is from a sociocultural perspective. Under Taylorist management philosophy, management is to do the thinking, and workers are to do the doing. Structuring a project where managers take on the exploratory phase fits well within the set of assumptions taught in project management programs.
If you’re doing that kind of work, your employer will be happy to replace you with an AI system.
Or a couple of Excel macros.
You also need someone with 1) strong technical and/or product skills, 2) good communication skills and decisiveness, and 3) explicit executive authority. That's the only way to avoid scope creep and bikeshedding.
When you started working on a project without knowing what you're building, or end up delivering a very different project than what was originally planned, it's hard to even say "that project was late." No it wasn't: you just never built whatever you had promised to build by the original deadline.
For example:
If the job involves dealing with anyone from Samsung or LG, it will take 2-3 times as long as you think.
If the job involves adding in a commercial third-party component which has a website about it, and that website doesn't clearly explain what the benefit of the thing actually is, the job will take 10-20 times as long as it should.
If the job involves a new feature Amazon or Google have just launched on one of their devices, and you are being CC'd on email chains involving representatives of Amazon or Google who are claiming implementation is very easy, the job will take a minimum of 6 months and may never be completed because the feature will be dropped or replaced with V2 before V1 ever actually works.
But your CTO will think it will take 10-20 times less time than it should, because that's what the vendor promised.
This is why they are not only inevitable but unending.
Most real world problems are so large and messy that they do not have closed form solutions. In this context "bug" more often means a deficiency, a reflections of changing or competing and incompatible requirements, than abstract mistakes.
In this context, I've learned to appreciate the choice becomes "which bugs", because "fixing" such bugs is not necessarily a net positive, especially when the fix is not a natural fit for the design or architecture of the overarching solution. The mindless, incessant fixing of such bugs on a case-by-case basis can result in accumulation of so much complexity and LoC it becomes self perpetuating in bug fixing alone. When you are conscious of this, and have enough respect and autonomy, it's possible to decide whether it's worth this intangible cost, or whether to put it into the pile of "wont fix", this kind of decision requires cross domain collaboration. But that pile is not worthless, you can periodically revisit it and see opportunities to a fundamentally different approach that may tick more boxes (or more preferable boxes) without being ad-hock.
> Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law.
> A company going under.
What a wild assertion: The OP hasn’t personally seen a company fail, and therefore software quality doesn’t matter? Bugs and slow delivery are fine?
It’s trivially easy to find counterexamples of companies failing because their software products were inferior to newcomers who delivered good results, fast development, and more stable experience. Startups fail all the time because their software isn’t good enough or isn’t delivered before the runway expires. The author is deliberately choosing to ignore this hard reality.
I think the author may have been swept up in big, slow companies that have so much money that they can afford to have terrible software development practices and massive internal bloat. Stay in this environment long enough and even the worst software development practices start to feel “normal” because you look around and nothing bad has happened yet.
Once upon startup. The code inherited from Netscape was... let's say charitably difficult to maintain. Turning it into something that could actually be improved upon and tested took years without any release of the Mozilla Suite.
Once when Chrome emerged. Not because the code was particularly bad, but because its architecture didn't reflect more modern requirements in terms of responsiveness or security. See https://yoric.github.io/post/why-did-mozilla-remove-xul-addo... for a few more details.
> That’s not to say that I now think that “High quality good! Low quality better!”. > As many commenters on Cupać’s post observed, while high quality isn’t necessary for success, it improves the chances of it.
_Nearly_ killed. But Mozilla is still kicking around. Maybe not doing the kind of internet-shaping work they once did (here's hoping they get back there), but they continue to exist.
This blog post is saying “Staying healthy doesn’t matter because neither I nor anyone I know died so far.”
I'd love to come to the conclusion that it's wrong, but it's right. Most companies absolutely 100% can afford to ship bugs: the proof? They're doing it, they have been doing it, they will continue to do it. That doesn't mean that every company can, always, forever. A single really bad bug can tank a company. In the right conditions, simply being buggier to a superior competitor can tank a company. However, those are mostly just things developers fantasize about. The market is by and large not decided by how elegant and well-designed your software is, moreso in some verticals than others. In fact, this is basically true for almost anything that isn't explicitly developer oriented in the first place.
Just look at enterprise software. Jesus Christ.
Sure, healthy and happy chickens are nice, but if you are into industrial farming and your goal is to make money, you will make their life as miserable as necessary to extract profit. You don't care if they are barely alive as long as the health standards are met.
It's not about you being happy as an individual, it's about the company making money. If your lawyers and sales are good enough to build a captive market of miserable customers, your company can still make a ton of money and be very successful.
- the customer: either B2B or B2C
- the market share: minimal (< 1%... including all startups) or dominant (> 30%)
- B2C is really dynamic and few bad versions/products can make the customers fly away (except when strong dominance - like Windows - or no equivalent product) and shutdown a company. Price can be a strong factor and cost of migration/switching is usually not considered
- B2B is more conservative: hard to enter the market (so small market shares will need a lot of time to take off... if there's no competitor) but once you're in, the cost of change for a company is usually high enough to tolerate more bad versions (and more if there's few competitors, and incompatibilities between products, and legal requirements to keep records, and a lot of "configuration", and requiring a strong training for a lot of people...). Companies as customer dont see the switch of software as a technical problem (replacing on editor by another one) but as a management problem (training, cost to switch, data availability, availability of people already trained, cost of multi-year/multi-instance licences...)
Is it? What I've seen is the opposite.
Businesses can be terrible top to bottom, slow, inefficient, and painful for customers, and still keep going for years and years. It's more about income/funding than product.
> I think the author may have been swept up in big, slow companies that have so much money that they can afford to . . .
That's what I'm talking about. They are legion! They could be companies that serve a niche that no one else does or with prohibitive switching costs (training is expensive). They could also be companies that somehow got enough market share that "no one gets fired for buying IBM."
Also, you know what those "big, slow companies" have in common? They are successful businesses. Unlike most startups.
While I personally haven't seen a company going under due to bad code, one can also definitely make the argument that software that is buggy or doesn't scale will lead to lost profits, which could also eventually become a problem, or present risks to business continuity.
I still recall working on critical performance issues for a near-real-time auction system way past the end of my working day, because there was an auction scheduled the next day and a fix was needed. I also recall visiting a healthcare business which could not provide services, because a system of theirs kept crashing and there was a long queue of people just sitting around and being miserable.
Whether private or public sector, poor code has a bad impact on many different things.
However, once can also definitely make the distinction between keeping the lights on (KTLO) and everything else. If an auction system cannot do auctions for some needed amount of users, that's a KTLO issue. If an e-commerce system calculates prices/taxes/discounts wrong and this leads to direct monetary losses, that's possibly a KTLO issue. If users occasionally get errors or weird widget sizing in some CRUD app or blog site, nobody really cares that much, at least as far as existential threats go.
Outside of that, bugs and slow delivery can be an unfortunate reality, yet one that can mostly be coped with.
For example, pg said that ViaWeb was successful because they had put care into their code, which allowed them to iterate quickly and integrate new features that customers requested. Whereas competitors were held back by their cumbersome code and slow cadence of releasing features.
But maybe he was actually wrong about quality being a differentiator, and his competitors could've shipped features with shitty code but had other problems.
More generally it depends on how competitive the space the product operates in, whether quality is something the buyer values and is able to evaluate! Enterprises for example infamously don't appreciate quality as much consumers because the economic buyer does not use the product.
- take a blank piece of paper (this is “the software”)
- pick two random points on the paper roughly 3 inches apart (“a requirement”)
- draw a line between the two points; the line cannot cross any other line (“the implementation”).
Repeat the exercise multiple times.
You’ll quickly learn that unless you have a “system”, drawing a three inch line goes from taking about 2 seconds, to taking 10, 15 or 30 seconds, despite “the requirement complexity” being exactly the same every time.
Now try playing with people taking turns. :)
I like playing this game with people who think that writing bad code is fine, or that they can work by themselves and not worry about what other people do as long as “their own” code is good.
You can still solve pretty much any problem with enough time and effort; but if you don’t have a “system” for good organisation, eventually, you’ll be struggling to solve basic problems because of layered unmanageable requirements, no matter how smart or capable you are.
…it’s not about shipping bugs, it’s about fundamentally reducing the speed of delivery over time, incrementally, in an unbounded fashion that eventually makes systems impossible to make changes to without compromising on requirements.
(Obviously the game is contrived, but it works very well in explaining why past requirements some times have to go away to business people in my experience).
That's provable; with no closed loops, any pair of points on the paper can be joined by a long enough line. That's why it's a good analogy for code.
...but if you, eg. a system for this; Use straight lines only and space the lines equidistant from other lines.
I mean, it's a game. Literally play the game a few times (grab a friend and do it together) and you'll very quickly figure out things that do work (leaving space) and dont work (draw crazy squiggly lines and lines that are too close together).
This is just a contrived example; if you want to make it more realistic allow 'you can erase a line and redraw it as long as the requirement is maintained', and then you can structure lines into groups, etc.
Whether you use structured code (straight lines and spacing) or squiggles (spaghetti code) is up to you as the developer implementing the requirement.
Speed only goes to zero if you insist on bug free software.
https://news.ycombinator.com/item?id=36615325
I don’t think anyone needs to waste their time arguing with a strawman.
It still got used and was one of the common deployments for MS SQL and Exchange during that time.
The original article doesn't sell a solution, it sells some spin from the author, but the original author wasn't wrong that there is a ridiculously high tolerance for buggy, low quality software, and the reason isn't super clear. These aren't cheap services either, they're billion dollar companies that can barely keep the services running sometimes or are incredibly slow to react to missing features or features not working well/correctly.
I think this situation merits discussion.
I disagree. I’d say the reason is clear: the market decides; and the decision is that current amount of bugs is ok. Most software is fine for the end user.
We (devs) notice the bugs because we’re familiar with the process; but in the end we’re just people who know how the sausage is made.
When I wrote the reason isn't super clear, I understood that as developers, you're just the sausage makers so to speak. (but let's not think of this in a bad way! Sausages are tasty and a stable of many cuisines!) The loss of agency (or lack of it from the beginning) for developers is not unknown and it's a point I'm extremely sympathetic to. Where I say it's unclear is that:
- Developers don't like bugs
- Users don't like bugs
- When it loses them money, businesses don't like bugs
Every participant in the software chain basically doesn't like bugs and wants to make it better. A bug that does a particularly bad thing at a bad time can be fatal to any business; cosmetic bugs that require constant workarounds can be costly to the users and businesses using the software, and those issues that "only take a second to workaround" add up.
Basically, as I see it all bugs are costing us a ton and chipping away at our time and expenses, but universally we still seem to be pretty okay with even fairly large blocking bugs sitting in production systems for months or longer.
That is the part that is unclear for me; it's well documented how impactful the bugs are, we can measure the costs for everyone involved, lost renewals or contracts because bugs couldn't be fixed fast enough or were deprioritized for something else; but we're still pretty okay with having these sit around.
The only simple answer I can really come up with is that the market is okay with it only because short term profit can still be derived or there is capture, or both and possibly more.
It's just a really bad situation for everyone except a select few who still profit despite everyone hating the situation.
Although when it comes to software that market is quite heavily skewed by the VC funding effect. If "the market" is getting something free or cheap as a loss leader because it's really the VC money that is paying the bills for a few years then end users are going to be much more tolerant of junk for a while.
In the VC model it's boom or bust so quality and coding for the long term might not be business priorities. In the short term unless you're so bad that it's outweighing the sales and marketing push to drive that rapid growth you can get away with a lot. In the long term you don't care so much about maintaining the early code because by the time a big exit is a realistic outcome you have the resources to throw things out and rebuild from the ground up if you need to.
Of course none of that means you're actually providing a good experience to your users during those early years or that they'd be willing to pay enough for what you're offering or stay with you for long enough to sustain and grow your business independently. You're still shipping junk and your users still don't like it. You just don't care because your bills are really being paid by someone whose interests barely align with your users in those early years and the bill payer also doesn't care because they're only interested in the big exits and expecting to write off all their other investments anyway.
If you're a software developer who does want to build good products and have happy users benefitting from them then this kind of market is toxic. If you're a user who just wants good software to do its job so you can get on with yours then this kind of market is toxic. Eventually maybe the potentially paying customers will get wise to the startup game and start avoiding it in favour of alternatives made by smaller companies that bootstrap and aim to build steady and sustainable products and businesses. But there's been so much money splashing around in VC world for so long now that a lot of people just can't think outside the box.
Like, sure, even if we make best efforts to catch both obvious and less obvious bugs, all software we ship will still be full of bugs. But knowingly shipping software full of obvious bugs... it feels unprofessional. And makes you really feel the weight of all the inevitable bug reports that come in.
The alternative often becomes to warn about it, have the organization respond (whether explicitly or otherwise) "we don't care", and then, to preserve your sanity, say to yourself "well, then all bets are off, and I'm not responsible" which also isn't healthy, because it leaves you jaded, disengaged and robs you of your sense of professionalism.
I don't have a solution here lol other than find an org that cares about quality to an acceptable degree.
I still have a lot of professional pride, but it comes more from my personal projects and sharpening my craft than to some corporate version that could be dragged down by structural issues.
This is true... but it's also true of ≈everything else in the organization. An organization that could be broken by any single thing would not survive for long! This includes the things people tediously insist to be "more important" than software quality. It certainly includes the usual suspects that push people to cut software corners: companies don't break by missing OKRs or slipping schedules or not "focusing" or any other management pablum any more than they break due to software quality.
But, of course, all of those can contribute to a company failing—including bugs, slow delivery and poor quality code. The dynamics and the extent to which different factors matter is inherently context-specific.
Companies can fail despite doing the right thing and succeed despite doing the wrong thing. This doesn't make the wrong thing effective or reasonable, it just means we're operating in the domain of complex, stochastic systems, so our simple mental models of cause and effect become misleading.
The fact that low-quality software doesn't directly cause a company to fail does not imply that investing in software quality is not right. I am firmly convinced that maintaining high quality software will have better outcomes in expectation than cutting corners pretty much universally, if only because it simply doesn't cost much—it's a matter of setting the right culture more than anything else.
There are interesting observations to be made here. Variations on "software quality doesn't matter because it won't destroy the company" are not it.
Except for sales?
There are many blog posts like this one that deny (often deride) the value of software craftmanship, including the skillset of planning.
They almost always have this same conclusiveness to them as well ("she was right"), with no tangible backing other than personal anecdata ("everyone sucked who I worked with, so everyone must suck").
If you zoom out, the reality is that people in the software industry are spoiled AF. A majority of practitioners get paid a ridiculous amount and get away with murder.
It is true that current market dynamics give people in this industry pretty much zero incentive to get good at the craft. So people don't and then to rhetorically justify their philistinism, they write blog posts like this.
It is also true that many businesses get away with writing absolutely shitty software and stay in business because of these dynamics. But that doesn't make it "ok" and doesn't say anything about the enormous amount lost to such apathy.
But, well, this is capitalism for you - whatever is the most profitable thing to do is what gets done regardless of how not okay it really is. I think the question we should be asking is: if users hate using crappy buggy software, and devs hate writing it, but it still gets written, aren't our basic economic incentives clearly out of tune?
It’s definitely market dynamics, but I think something could change those market dynamics.
Maybe a craftsmen’s guild that unionizes skilled software practitioners.
An emotionally healthy person tends to want to maintain their environment. It doesn't do anything objective to have nice furniture and pretty paintings, but you like it in your house. You want to feel at peace, so you adjust little things...the kitchen would be nicer with the plates in a pile over here instead of just pulling them out of the dishwasher.
At work, you're the householder of a million technical details, and to feel good about it you want to maintain your environment! If that window has an ugly redraw you want to recode it, if the ribbon has big ugly icons you want to collapse it into a strip menu, if the email client freezes whenever WiFi gets spotty you get annoyed because the code is still from 1995 under the surface. The software you're making and using, in service to your own craft, has an effect on you. If it makes you feel ugly, you want to fix it. This blog post -- which probably wasn't written on company time -- is here because someone needed to reorganize their thinking to feel emotional health. I think we should insist on doing a better job than just sacrificing our work to feed the money machine, even if our self-described jobs are just to turn funding into paychecks.
Which, interestingly, wasn't something a software engineer came up with.
Maybe the two are groups are not the same. Someone getting into software to make money is different to the kid who couldn't put the computer down. There are people that want to hack on software and others that want to solve business problems with computers for profit.
To really grind someone's gears they'd need to hate something that cannot simply go away.
> Where high quality is nice to have, but is not the be-all-and-end-all.
while being mostly true in all other cases I don't think it does it justice what mediocrity actually means for everyone involved. I have seen very varying levels of quality requirements/enforcement, testing, delivery speed and eventual bugs at different costs at different companies. And I in general found that higher quality software and less bugs were associated with much happier and more productive developers. And I would make the argument that less happy developers => more turnover => more second order costs. So while it might not make or break the company it certainly has a negative impact on profits downstream. And for customers too. Vendor lock in is a great thing for a company to profit from but if you really make a shitty product it opens up a lot of venues for competitors to eventually canibalize your market or from users to jump ship the next time they can. Take MS Teams for example. Lots of lock in but trust me the second better competitors are on the table I vigorously fight to switch. It's a slow burn but a burn nonetheless.
That does not mean "high quality." You can still have an architecture that's tightly-coupled spaghetti code that resists all human efforts at debugging. It just means "we threw enough paperwork at the auditors that they're not asking for more."
A relative of mine who was an accountant for a small manufacturing plant and the systematic problems from top to bottom were shocking. They worked there for well over a decade and also managed a lot of other business aspects. Nothing got shipped, billed, ordered, or paid without them knowing. The company always had just enough income to keep going. Somehow, it was sustainable without any hidden external sources of money.
Yesterday somebody shared this with me and it made me reflect about what things could we learn from other engineering disciplines. It might be just me but I have this gnawing impression that a lot of the pains we have in software are self imposed.
As someone who works for an organization that is on the receiving end of bad software, I can assure you this is not the case. There is lost opportunity, due to decreased productivity and outright lost clients. There is a souring of the work environment, an environment where people are constantly putting out fires and there is constant churn in staff because of that.
Perhaps the organization I work for is unique in that it is public sector and cannot go out of business (which appears to be the author's definition of broken). Perhaps the author has never dealt with software where edge cases involving lost and phantom data is a daily norm. Clients, real people with real lives, are literally lost by the system.
Things are so bad that losing several years of business records (or, more likely, paying to have them maintained by a retired system for some legally defined period) is considered as a viable option.
So yes, buggy software does break organizations.
But a piece of line-of-business software that sends out the wrong email once because someone made a bug in a batch script? Sure. Not the end of the world.
Users will tolerate some level of errors. Even if that level is sometimes expected to be 0.
There is no organization. There is only people. Some of those people, like your boss or your investors, may want you to burn yourself out or do unethical things to your customers so they can make more money. It's your prerogative to tell them to fuck off, and go work somewhere else if you don't get your way. Some people don't have that capacity. Software developers generally do.
Perverse organizational incentives will pressure you to be bad. Refuse.
* software quality is (more) important
* software quality is not (very) important
* bugs are not important (aka "i don't test, but when I do, it's in production")
* the less bugs you have the better, but they're not gonna kill you
* bugs are safety-critical
* slow delivery is fine
* slow delivery is not fine
In the last 20 or so years working as software developer professionaly, in various teams and quite different projects, I've noticed that:
* people (decision makers, software buyers/clients) always say they want higher quality
* same people very rarely want to back that up with their budget
* people almost always prefer more features to more quality (even when explicitly told that more feature is not a good thing for eg. MVP stage of a startup)
* most people except dedicated QA engineers can only think of a happy path (if everything works the ideal way); this includes not only other engineers (who try to think up failure scenarios but often miss important ones), but also designers (when have you ever seen a design mockup for "DNS lookup failure connecting to the server" case?), and definitely product owners or clients (in context of a dev agency)
* I have never worked on a nontrivial software project that didn't change requirements/scope
Myself, as with many other software developers who chose this field out of passion and not stratospheric wages, was for a long time appalled by this apparent lax approach to quality? Don't you want to have the best possible product?
But working with a lot of diverse clients while doing dev agency work, and running several startups myself, has taught me that software quality is not, and (it pains me to write this) should not be, the top priority. Product-market fit, treating your customers well, having a sustainable monetization strategy, marketing and sales ... if you don't execute well on those, nobody will notice the (lack of) quality[0].
In most organizations (except the ones that are swimming in cash) it's a tough act to balance all of the above. While I wish for all the projects I work on to be the best they can be, a programming work of art if you will, I can certainly empathize for people who must prioritize otherwise.
[0] unless the quality is safety critical, in case I need to point this out explicitly.
We are too often religious about these things, and may not have visibility within the business to see why there isn't a case for it.
Having said that it can also be very hard to convince management of the business case. That may have had a bad experience of these tool and practices before and see them as a waste. Thay may also, in that specific case, be right.
I certainly see a connection (several actually) between having more bugs and following development processes that emphasise metrics like time between starting work on a job and running the code in production. The advocates of Accelerate and the annual Google DORA reports seem to have taken over the web development asylum lately but the underlying arguments and data are still dubious at best.
I've also seen companies go out of business because they tried to make perfect software and never actually delivered it to anyone.
The right approach, I think, is to do the minimum necessary and do it very well. Easier said than done though.
I've personally been involved in 2-3 software projects where the immediate ramifications related to software quality and delivery led to project failure.
The idea that bad (or no) architecture and terrible code won't have business impacts is just wrong. Perhaps he was lucky, perhaps we have collective survivorship bias, but I strongly disagree on the main premise here.
In one case, some bad decisions related to API design and customer extensibility led to a product that was simply impossible to change or evolve. That company did some custom work for a few clients on a shared framework, but after ~5-8 customers, the product was basically unchangeable. Some of the customers left, but we still couldn't change it, and the result was pretty predictable.
That you can limp along with terrible practices doesn't mean that better practices wouldn't improve the situation.
Some people hear it and think it means highly-structured abstraction towers vs low-structure imperative-heavy code. Some think about horrible spaghetti with 16 levels of indentation and 1-letter variable names (moving on to 2-letter once you exhaust the first 26) and if-statements whose predicates span twelve lines. Some people think about feature completeness and bugginess (or, rather, lack thereof).
I think they're dangerous over different time horizons and you can't just prioritize over all time horizons.
Spaghetti is dangerous in the short-term - it makes small changes difficult and unsafe. You'll end up tied by your own shoestrings and not even finish the project. Luckily if this is the only problem, it's also the easiest to clean up. (But it's not usually the only problem).
Less-structured code is good for when you don't understand your requirements well. (Which you typically never will a-priori, even if you think you do). It's easier to add more layers of abstraction than it is to change or remove them. If you prematurely build an abstraction tower, you will suddenly find yourself having to plumb information up and down the entire tower as the abstractions prove to leak or not be completely perpendicular to the kinds of features you need to add. But if you wait too long, you have too much unstructured code to refactor and you're now "chasing" people adding new features to the system while you try to refactor the system underneath them. So there's just some engineering gut-sense that's important to know when to stop and codify the patterns you've seen emerge into a useful, tight abstraction.
And then of course there's features and bugginess. It's possible for the shittiest code to have very few bugs and please users. The worst thing about the worst code I've ever seen in my career is that it worked - so it was hard to want to mess with it when it was so easy to ignore! And it doesn't help that what is intractable code to some people is only a little complicated to others. Especially if you use a big language and are combining a lot of esoteric/complex features, or have a complicated logical expression and you are a logician and others are not.
I've seen projects become unchangeable because the code was spaghetti. I've seen them become unchangeable because there was too much abstraction. I've seen them become unchangeable because there was too little abstraction. I've seen awful code make big impact and I've seen excellent code flop because it wasn't competitive with "worse" but more complete offerings.
(All of this reinforcing, of course, that it's impossible to estimate how long software will take..... all of it relies far too heavily on the intuition of your mid+senior engineering talent).
There have been companies that went out of business, not only the software publisher but companies that relied on such software.
If the assertion is that "known unknown" bugs exist in code, and the publishers are aware that there might be bugs but in their QA/QC could not identify them, I can accept it. But, if the assertion is that there are "known known" bugs exists, I find it bordering unethical. The only way I can accept this if the publishers informs the buyer, and let them know what is the impact, the plan of action to remediate or mitigate the bug, and in what time frame.
"Known known bugs" (the publisher knew about the bug for extended period of time, and never informed the users) have destroyed multitude of businesses.
But something being mandatory or not doesn't influence the ROI. Something can be optional and high ROI or mandatory and low ROI. Low quality software is quite destructive over the long term and it is a horrible mistake to let coders go wild writing buggy software slowly.
That being said, the post is probably clickbait aimed at promoting her services. I doubt many people would make the mistake of thinking it a good idea to purposefully low quality written at slow speed.
More abstractly, high productivity engineers help you to exhaust the idea space through delivering products into real world before you run out of money. It's better to fail because you discovered your supposed gold mine had no gold rather because you ran of mining resources half way through.
[1] Notably this doesn't apply to capital intensive business with high marketing or infrastructure costs
Bugs are inevitable, but that doesn't make them OK.
But that's not the point the article is making. They're talking about whether or not bugs and slow delivery will break a company. Obviously it won't, because both have been a part of what the industry releases since the start.
Technical debt / quality / boneheadedness was at least a factor in each case.
I've also worked in financial services where a bad enough bug can and will lose you your (jurisdiction-appropriate) license to to do business.
If you don't care about anything that you should be care about, of course nothing will be an issue.
"Responding to “Are bugs and slow delivery ok?”: The blog post that I’ve hated the most, ever"
But if you scroll down to the end of the post:
> So thank you, Valentina, for your thought-provoking post. I loved to hate it.
So the author does not actually hate the post they say in the title they hated.