How to think about software quality (2022) | Hacker News Reader

How to think about software quality (2022)

evalapply.org

154 pointsadityaathalye66 comments

Most software exists to save someone somewhere time. I think this is the One True Metric by which we measure Software Quality.

Software which sells well historically has done so by saving huge numbers of people huge amounts of time, for whatever other complicated business they're already involved in. Excel, ERP systems, and GPT-4 all do this to great commercial success. You only need to step back and wonder how many more man hours it would take to do what they currently do by hand to realise this.

Software can also save the time of other developers by being simple, well-architected, reliably documented, etc. This too is an aspect of software quality. Its recursive nature can often become a competitive advantage in its own right: High quality code, can lead to faster change implementations (fixes and features alike), which can filter downward to a higher quality product for the end user.

I like the sentiment, but you can have quality, well architected software that’s pleasure to work in, and it being completely useless/obsolete to customers. Value != quality.

I think software quality (and all the symptoms discussed in the post) are to do with how quickly the software can be changed: Lower tech debt results in software that’s quick to change to new requirements, or change in business domain. It’s also important to realise that just like with performance, you get diminishing returns at very high levels

I think software quality is more related fo how little it changes while remaining useful.

I see the current software environment has different stake holders all defining quality from their own perspective.

Ah I see, quality can also mean ingenuity, but in my case as a software developer ingenious solutions don’t always present themselves. I’d settle for old boring plain solution that works well enough, but is of enough quality that it doesn’t cause fires, and is quick to change with the business

Not just how quickly it can be changed but how risky it is to change.

> Most software exists to save someone somewhere time.

Gaming. Social media. The Ads industry. These are all huge and they're not about saving anybody's time.

One thing I like about this is that the author doesn’t talk in terms of absolutes. He assumes (correctly) that advice is not universally applicable and that each process needs to be traced back to fundamental axioms or requirements.

As a random example I’m right now trying to explain this to a large government enterprise customer.

They write “cold blooded” software, the type where they have a development project with a fixed budget and a fixed scope and then the code goes into hibernation for five years until they can “find the budget” again.

So of course, they picked the Angular framework, one of the fastest moving, most “hot blooded” frameworks in the world.

Angular might be the right choice for a startup writing highly interactive software. It is definitely the wrong choice for an org that thinks of maintenance as something that occurs twice a decade.

It’s these things that most fundamentally kill quality. Picking a framework, language, or even operating system because it’s “hip” or “cool” or the “buzzword of the year”. Not because it’s suitable for the project.

The thousands of cuts come later, but they barely rate in comparison to having your legs chopped off at the knees right from the beginning.

Minor nit regarding this thing that will destroy quality:

“We'll fix it if customers complain."

You can’t fix everything, and perfect is the enemy of good. While punting testing to users is a no-no, recognizing that edge case issues can be deprioritized is crucial to getting things out the door. This means shipping stuff with known defects that you can respond to if necessary.

> You can’t fix everything

Oh, you certainly can't.

But you can choose how many things needing fixing you will make. Doing some shit without thinking and waiting for people to complain about symptoms of deep issues is a very effective way to create lots and lots of need for fixing things.

And my experience has been that a quality culture like that can "resource starve" annoyances for literally years due to

  while True:
      ship_feature()
      if bugs:
          to_fix = highest_priority(bugs)
          fix = make_patch(to_fix)
          fix = introduce_new_bug(fix)
          # defer bugs[1:] until they meet the complaint criteria

and that's how meetings end up with "but, how hard can that possibly be to fix?" statements

I knew a guy who said on a previous project they had a system that raised the priority of old bugs on some interval. I guess the idea was to either get them fixed or agree that they never would.

There are morale and duplicate bug issues with keeping a backlog that grows faster than you can close them, so I guess that makes a bit of sense. And better than GitHub auto-shooting-the-messenger by clocking bugs after a month or two of inactivity.

If you accept some bugs as "features" that you will accept, not fix, you can have a bug free policy.

Doing this means you make decisions early and don't carry a ton of historical baggage that gets endlessly discussed.

> You can’t fix everything

But you can fix a lot. Some of the products I've worked at had Sentry alerts all the time that nobody paid attention to, whereas others (of very similar complexity) had them a few times a week, and every time somebody would immediately get on top of it and release a hotfix, usually in a span of the same day.

OP here. I agree and I think I imply habitual punting. No product or service can completely delight every customer. I would even say customers are forgiving of minor annoyances as long as they know the Big Important things are taken care of. Yet there is some point where the minor annoyances become too many. Finding ourselves at that point could be a "too little too late" situation, with a steep cost to recover lost trust.

(edit: note I am OP)

So much software suffers from a large class of users having no choice in its use. For example, our local school district uses Bloomz, which is pretty terrible - both on the web, and in its native app.

I can report all the bugs I want, but since I'm never the person making a purchasing decision, how highly do I think those reports are prioritized?

I wonder if the teacher/administrator side of things is better.

I found the best approach is to search LinkedIn for someone at the vendor that's closely connected to me, then compose a cheery message to that person introducing myself as a fellow developer and parent of a user, ask the best way to inject a bug report etc. This actually worked. Ymmv of course.

As someone this happened to, I was actually impressed they looked me up and reached out. Knowing they cared that much meant that I cared too and I was able to prioritise the request with the product team. When we completed the request, we celebrated, and even implemented uservoice on some sites to improve the feedback loop.

One of the reasons I never put my employer on my LinkedIn profile.

Most of life's fulfilling opportunities come from just showing up. Showing up on searches, at work, at the pub.

If I got a message like this I'd just block the sender. As a developer I have no say in what my owner tells me to work on. I type what they want and nothing more.

They're not asking you to fix anything. They are asking where to submit a bug report that someone (someone else) will look at.

My answer doesn't change. I'm not on LinkedIn during work hours.

Ohh, this is actually clever. Subversive and I'm not sure I like it, but it sounds very effective.

Sounds like a hack to make a person to work overtime.

If I get a message on linkedin that pertains to a work issue, you had better believe that I'm going to be dealing with it on work time.

... while having fun while doing that work.

Post author here. I agree; so often constructive feedback loops from end users to builders are woefully absent from the equation.

I feel the greater the distance between the purchase decision maker and the end user, the weaker the feedback gets. Riffing off the six degrees of separation idea and exponential drop off of signals though lossy media, it's like by the time someone gets to a third degree of separation, any feedback signal barely registers. Only mass action has any chance of being heard, much less heeded.

Ah. So that's (part of) why enterprise software is so universally bad. The users are further from the purchasers than they are in, say, developer tools.

I believe so.

I'll pick on Government websites and the systems that back them as an all-too-familiar, internationally reviled category of software, as an example. These tend to be incredibly painful to use. Some tendering process somewhere awards a contract to the lowest bidder who then has the unenviable job of wrangling the delightful intricacies of government machinery to put something out that maybe does what it was supposed to do for the common citizen.

In such a world, I feel gov.uk stands out as a remarkable counter-example, worthy of emulation. e.g. https://www.gov.uk/guidance/government-design-principles

This plagues most enterprise software. Decisions are made by people who will most likely never actually use the software. So the decisions are made by cost and ease of dealing with the vendor. Usability is way down the list of priorities.

There's a whole thing around the nature of who makes the purchasing decisions vs who are the real users. For key organizational systems like Bloomz for schools, the decision is made at the senior executive level, and input from the people who will use the software on a day-to-day basis is rarely, if ever, taken into consideration. Sometimes the decision is made on the golf course between business buddies.

In these situations, the vendor doesn't have to care much about user experience or feedback, all they need to do is be able to check all the boxes on the whatever list of features management cares about. It comes up on HN off and on, here's an example: https://news.ycombinator.com/item?id=21224209. Thread unrolled here: https://threadreaderapp.com/thread/1182635589604171776.html

As a former teacher, my experience of education specific software was almost universally negative. We may even get an in-person meeting with the software vendor only for the reported issues to remain unpatched indefinitely.

As a developer, I’m not at all surprised, I suspect it’s negative from almost all points of view except at the district or city or state level where they pat themselves on the back for bringing technology into classrooms via spending some money, what little money they have, on tech that they don’t use and don’t understand. I don’t even have that much experience with it, but I’ve tried making educational software, in the form of multi-user whiteboards teachers could use for their classes. The people approving purchasing decisions don’t use the software, and don’t ask teachers or students about the experience or value of the software. The insane sales cycles are a never-ending nightmare of requirements and delays, far beyond the reasonable concerns about kids & privacy. After years of run-arounds, we gave up trying to use any official channels and just let teachers use the software free.

As a parent, the experience was also very negative. My kids hated almost everything the school forced them to use, because the software was terrible compared to non-educational software. Teachers didn’t get any training on the software, and didn’t have the resources or time to use it effectively. Hardware was always outdated or supremely cheap and crappy. Possible the single worst software experience was the automated essay grading software. It was beyond broken, actively harmful, and it literally didn’t work. I came home from work one day and my oldest kid was in tears and my wife at her wits end trying to finish a homework assignment to write a short concise 2 paragraph essay on a specific topic and achieve an arbitrary 90% score or whatever within 10 tries, and they’d used up 8 of them and scores were going down each time. This software isn’t even supposed to be used unsupervised for grading, but the teachers didn’t seem to understand, and weren’t following recommendations. I figured out quickly the software doesn’t understand english and would score higher if you fed it more words, regardless of their quality or relevance - the polar opposite of the assignment’s goal. So I gave it a wikipedia article on why people shouldn’t use software grading robots, which scored 100%, and then scheduled a meeting with the teacher, who proceeded to complain about the software and told me she didn’t have a choice because it was district rules and a state software contract.

The cynic in me thinks this is 100% working as intended if you view school (especially before university) as an industrial sssembly line to produce the obedient little workers of the future.

By making kids “accept” to be arbitrarily graded by an automated black box on seemingly virtuous but absurd criteria in practice, they’re being trained for what the future of work might become if coporations get their way and put AI everywhere they can. Or if they’re lucky they learn instead to cheat the system and/pr hack it but that’s a minority who will become misfits in their professional lives.

I hope you were able to cheer your kid up, regardless but this is a scary look into the near future.

You mention not being the person making a purchasing decision, who is? Can you connect with them?

I would encourage you to put together a story that illustrates the pain being caused by this software and get with the purchasers. Not only might it help your organization figure out they need to do something different it could also show off your skills to the orgainization.

Could you collect the time you and other people are spending working on these problems? How many issues are reported to you? How many bugs you've filed and have not had fixed by the vendor? Rolling this up into a real dollar coast it's having on you and the school district users.

> who is? Can you connect with them?

For something like this, usually at the senior executive level, and organizationally. There's not really one person making the decision, it's done through the organization's decision-making system. Nominally there's a responsible person, but practically speaking that individual isn't completely autonomous.

Large organizations have been making and dealing with decisions like this for ages, and rarely does dollar cost of usability ever influence. Products SAP and MS Teams get and retain market share by making the cost of moving off that software outweigh the gains.

The person making the decision is very like the superintendent of the school system, likely informed by someone from IT and maybe someone representing the teachers somehow.

They might listen to me if I bring in a coalition of other parents, but there's two reasons why I'm probably not going to do this:

1. They're likely locked into at least a year-long contract, if not more

2. Doing this is a huge amount of work for me. I'd have to find a significant number of parents who are _also_ as peeved as me, and then get them to take action - even if it's signing something - and then I'd have to bring the petition up to the school board, etc., etc., etc.

It's much easier to just mute the notifications on the app, and go through them once a week. 99% of them are just wildly inactionable anyway - things like reminding me that school is out, or that a student interest group that's irrelevant to me is having a meeting two Thursdays from now.

If I was annoyed enough to take action, I'd rather just offer to build them a replacement app - shorten that feedback loop, ya know?

This software also tends to be waterfalled to hell and back. Theyve already planned out their deliverables for the next 12 months and will be held accountable for meeting them. Your bugfix doesnt fit into that picture unless an exec treats it as a priority.

...in which case it'll be jammed into the devs' task queue with no expectation that the pre-agreed delivery dates will shift.

The issues you describe have nothing to do with "waterfall."

>Theyve already planned out their deliverables for the next 12 months and will be held accountable for meeting them.

Not sure I understand your point. Are you saying product roadmaps are incompatible with agile?

If they include detailed plans and timelines for feature deliveries then yes.

Agile doesn't mean "not planning" or even "vague planning".

Good read (reminder) and love the reference list.

Question for the OP. I suspect we're of similar tenure, and I'd love to det your perspective...

I'm a mechanical systems engineer in an industry that is growing from industrial automation to true autonomy (robotics). I've always liked coding, but don't do it enough to be a serious contributer to "production quality" code. Nevertheless, since last year I've transitioned from a more mechanical/hardware design firm to a much more mathy/software engineering firm. I find myself in a role, bridging robotics hardware and software teams working on a ~10yr old codebase. We have some great engineers! I've tended to stay on the customer requirements and algorithm development/analysis side of things, but I struggle with the lack of benchmarking and testing our group does, and I'm continuously surprised at the simple bugs we find that cause application crashes. Moreover, when a developer shows me code, I'm stricken by the number of undocumented "magic numbers" that seem like forgotten patches popped in there at one time to make things work for a test or demo, and the number if/then statements I see that are holding things together.

I'm also faced with a growing cry for "how can we best make use of these LLM things" (for which there are a thousand and one answers).

I see/feel the suffering, and what looks to me (from my 1000ft perch) like thousands of cuts.

Bottom line, I'm increasingly thinking that I really need to take a month and try tracing through the thousands of lines of C++ (which I haven't really written since grad school) in order to truly understand the state/configuration of pur code baseline so that I can provide some more explicit leadership in terms of - do we refactor everything that looks messy or the we don't 100% understand? Do we take a T.O. to build up our automated unit and system level testing? etc.

As a manager of a SW team, from your experience is it a good idea for me to dig deep into the code to gain an intimate understanding? That feels an awful lot like micromanagement to me, but I will do it if that's what 'wisdom of experience' in this domain proves is the most value-add.

If you can't understand it yourself, you will have to find someone whose expertise you trust to answer the questions for you. But you don't have to understand the software down to bits and bytes, just enough for you to make good decisions. Having pragmatic underlings bring up problems and propose solutions is what you want. Watch for idealists for whom everything is broken and needs to be rewritten. Incremental improvement is the way.

/me raises hand. idealist here, and I do want everything rewritten. Please ensure you provide your idealists with all business goals and limitations rather than shield/protect them. You’ll find that then their suggestions take those things into account

Thanks to both replies. There's certainly a time and place for refactoring, as there is a time and place for getting an MVP out the door. I guess striking a solid balance is the art of it all.

I am very transparent with business goals and the threats we face, and I do think that spurs the most engagement. It could be my rose colored glasses, but it does seem like everyone truly feels a sense of ownership and pride in the product. As someone who is very hands-on though, it's very scary not having full awareness of what's going on under the hood. However, I'm sure I can overcome that mental hurdle.

I appreciate the feedback/advice HN.

Not the OP but I'd say no.

Fundamentally software quality issues are organization and people issues. This might sounds a little nasty but what do you think of organizations where the person managing software people is a mechanical engineer, with very little experience, who thinks he should go through all the code base to "understand the state of the code". I would say that's not the kind of organization that produces quality software.

If you want to improve the software quality, invest in your software team. Work on the people- not on the code base. By your own description you're not the kind of person who should be leading the technical aspect, if you can't grow someone in the team to that role then hire someone to help with that.

In terms of your other idea, more automated testing, the answer to that is almost always a yes. But again, get your team there.

OP here. Thank you for the kind remarks. Apologies because I can speak only in generalities; the devil is always in the details. As such this opinion, not prescription.

I read your description as working through a conundrum of priorities, goals, and outcomes.

Perhaps the way is to not get stuck in the weeds of either machines or computers.

What outcome(s) matter for the overall system? What measures make sense in service of achieving those outcomes? How will people get feedback about progress in service of the outcomes?

I am not suggesting some transplanted management framework (OKRs, I'm looking at you). Rather wondering about the small set of things you can get everyone to buy into and work towards. Decision making should become more tractable --- do the magic numbers even matter in the big picture? They may, if you desire to stay within a certain failure rate and/or error bounds, and it turns out the magic numbers are part of the problem. A thing like this won't be obvious in the beginning, you will to investigate things. Being able to say "hey, our stuff is failing at X rate and it is hurting our product/business in A, B, C ways" should help negotiate with management for the time / resource budget. Sometimes it may not be worth it, and so it may be wiser to drop that battle and focus on other outcomes. The investigation ought to be useful later.

I noted Gene Kim's initiative [1] in my post (no affiliation). I feel like that collection of resources will give you much to work with. If you had to pick one of those, I would recommend the Accelerate [2] book.

And, if you wish to talk about any of it, I'm happy to discuss more: https://www.evalapply.org/index.html#standing-invitation

[1] https://itrevolution.com/the-idealcast-podcast/ [2] https://itrevolution.com/product/accelerate/

(edit: expand on the main part of the response)

I think you should probably leverage your team’s experience with the code, and understand what they would prefer.

I don’t have management experience and I don’t know your team, but I suspect if they agree there’s a problem and propose solutions, you’ll have better buy-in, and at least as good odds of successfully addressing the problem.

i find myself in a similar sounding role, i don't have answers for you though. i'm wondering, do you use git for version control and repositories?

Yes, we use Azure DevOps (switched over from Jira). Honestly, I rarely go into it as most of my time is spent developing business and customer engagement.

What about free software? That is, how can human society achieve good quality free software if it is physically impossible for free software to develop? Because personal income for free software developers is important.

I guess since it's personal income, then it's a personal problem for free software developers then? That's fine, but income level coincidentally correlates with high quality software.

I guess free software just doesn't matter then.

Options include… Volunteer. Donate. Ask the elected officials that represent you to choose open source software for government use cases. Encourage grants to fund open source. If you’re at a company, contribute back to open source.

Government officials for republics function to serve a greatest common denominator that doesn't even exist in reality. So there's no political representation at all for free software.

Well, the greatest common denominator for political representation does exist. But it by its nature excludes free software as even a talking point. Namely that capitalism is mutually exclusive from free software. Capitalism hates free software to the point that free software cannot even literally get a word, a voice, or a conceptualization that can help refine it.

That leaves us with the only realistic options for supporting quality efforts in free software being volunteer work that isn't empowered. Everything goes downhill from there. Grants and company lip service are only superficial reactions to the free software problem.

Capitalism is perfectly fine with FOSS software especially if it can use the idealism behind it to save on labor costs. Especially since the definitions of both "free software" and "open source" explicitly reject non-commercial licenses as heretical.

Correlation between in one level and quality software? Heh, lol.

This is well written, adi. I particularly like the book references.

Thank you :)

If there is just one I would pick from the lot, it would be the Meadows book.

when you make a post about software quality but did not test it on enough browsers to realize your constant width font ascii art is not rendering correctly in firefox or chrome, so the user has to grab an invisible scrollbar to see the picture... zooming out of course doesn't work.

One of the little ironies of life. I know about this (and some other annoying problems), and have been meaning to fix the typography on my site for a long while. But that would draw me into a rabbit hole of some other improvements I want to make to my site maker. And that will be, well, there's an xkcd for it. Two actually.

Automation: https://xkcd.com/1319/

Efficiency: https://xkcd.com/1445/

P.S. Also, I did not tell them to make invisible scroll bars (nor have my CSS rules, as far as I can tell)!

I think some of the bad outcome may be due to macOS being sneaky with the scrollbar display <https://support.apple.com/guide/mac-help/change-appearance-s...>, but your style.css for sure isn't helping matters with its overflow: specifications. The tradeoff, as best I can tell, is whether macOS folks get hidden scrollbars (I don't know anything about the zoom complaint, since I'm on desktop) versus omitting the overflow: and having the asciiart clip outside of its bounding box due to some other CSS interaction that I didn't bother to track down

https://www.evalapply.org/static/css/style.css#:~:text=overf... and https://www.evalapply.org/static/css/style.css#:~:text=overf...

Thank you for taking a look. I will take a look at overflow behaviour, which I don't understand too well. I think I need to take a step back and work out responsive behaviour in general. Sadly management's hands have been tied and there is not time to make small useful improvements :)

ok well this is why quality sucks in software, you dont have to theorize, you have an example right in front of you that you personally are involved in. same with me and everyone else who writes code. i have stuff i know i should fix but it would lead me down a rabbit hole so i dont. so where is this experience in our theoretical framework?

Purpose matters, I suppose. I write for myself. My site is a public extension of that writing (there are so many notes on my computer I may never publish). I've made no promises about the quality of work to anyone, and I get by just fine with what I have. It is available on a "take what is useful, discard the rest" basis. If I were writing for an audience I might prioritise things differently.

Reply on news.ycombinator.com