Quality and Effort (2018)
seths.blog
seths.blog
I'm big on Quality. Damn near obsessive about it.
I worked for a company that was renown for Quality. They were (and will be, for a while), one of those brands that is basically synonymous for "Quality."
They got there by a combination of really heavy-duty, enforced process, combined with some awesome "tribal knowledge." They have been doing it for over a century.
But it was not a good fit for software development (they make hardware). Way too inflexible.
Since leaving that company, I've been trying out some ideas that I had, as I worked there. For the most part, these ideas have been working -quite well; but I haven't really had a chance to scale them.
But software quality, these days, sucks like a supermassive, galaxy-core, singularity. It's reached the point where our entire culture is acclimated to crap software, and we even punish efforts to improve it.
Something needs to be done, and I've found that small, commonsense habits can go a long way towards that.
If we watch a craftsman at work, we will see them do lots of little things, that we barely notice (and that they, themselves, barely notice, as they are habit). These little things are what makes their work "Quality."
When they have an apprentice, they are forced to inventory their habits, and articulate them. That's one reason why martial arts schools make their senior belts train junior belts. It's also why I think that today's engineering culture of "24 months, max" at corporations is pretty corrosive to Quality.
I've found that writing my code, as if I am going to be using it to train someone else, and doing it long enough to develop Quality habits, helps me to do a better job on it.
I capitalize “Quality,” because I consider it a Core Tenet of my personal philosophy. I picked that up from that company I mentioned.
I often wanted to strangle the inflexible, stubborn bastards, but I have always been in awe of their singleminded focus on Quality.
Seth is also breezy and appropriately shallow in the quantitative sense. He strikes a nice balance.
> It's also why I think that today's engineering culture of "24 months, max" at corporations is pretty corrosive to Quality.
This is something I think a lot about. The problem is this is the incentive gradient that is an emergent effect. Obviously, if it was generally in organizations' interest to retain people, there would be some kind of incentive sink to keep them there (i.e., everyone would stop finding more competitive compensation at other companies).
It's cultural.
The tech scene has turned into a "gimmegimmegimme" culture. People have been incentivized to live a lifestyle, and hold values, that are commensurate with significant wealth.
My secret is that I've lived like a pauper, my whole life. It has earned me many sneers, but I don't really care. That would not be the case for a lot of people. I'm kinda weird.
In my case, I have found that shallow excess is not actually very satisfying, but no one wants to hear that. I'm told that "I'm just jealous, because I don't have it."
There's also tremendous cynicism; for good reason. Corporations try to sell asceticism, so they don't need to pay their developers (so their corporate officers and shareholders can live large). I know that there are "compensation consultants" that specialize in synthesizing low salary scales, so corporations can feel justified in offering a pittance. There was that collusion thing, in SV, where all the big companies colluded to suppress salaries.
That goes a long way towards earning respect and loyalty. </s>
Until there is less of a cultural "pull" to excess, there will always be people using money to "keep score."
The recording industry figured this out, long ago. Most of the excess that you see musicians (at least, ones getting started) living, is actually owned by their recording company. If they leave the recording company, they lose the crib and the ride. Sort of a seven-figure "company store."
I think that fostering an atmosphere of simple, basic respect would go a long ways towards retaining talent.
True Respect; not "pretend" respect. Most folks that run companies have very little respect for their employees, and that makes it easy for employees and applicants to justify holding their feet to the fire, compensation-wise. It's really the only leverage that a lot of folks have. They hate me anyway, so why shouldn't I make them pay for it?
I think this is obvious in the ridiculous hazing rituals that masquerade as job interviews. Corporations go to drastic lengths to make it plain that they are the ones in charge of the relationship, and that applicants are, in reality, supplicants.
This results in cynical people that learn to manipulate the hazing process, and extract what they want; with absolutely no respect or loyalty to the people that deliberately humiliated them, on the way in. I can't say that I blame them. No pride of ownership or craftsmanship. No loyalty. No incentive to leave a legacy. Just "smash and grab." I'll be gone in a year, so why bother writing software that's maintainable? It won't be my problem.
It's a very different world, from when I had been doing the interview circuit.
Also, as long as customers (including B2B customers) are happy with dreck, then dreck, they get. Why bother putting in the considerable extra cost for Quality, when the suc- um, customers, will be happy with garbage?
> I'll be gone in a year, so why bother writing software that's maintainable? It won't be my problem.
I would say that's why it's so important to have a core of competent technical employeers. One thing that is so underrated is company culture. If your technical managers or supervisors enforce stricter code quality, naturally everyone will follow on. Culture trickles down not up. The result you're seeing is from the effects of bad management or bad technical leadership.
>Also, as long as customers (including B2B customers) are happy with dreck, then dreck, they get. Why bother putting in the considerable extra cost for Quality, when the suc- um, customers, will be happy with garbage?
That's the thing. No customer likes garbage. Garbage here defined as buggy, slow or unusable product. They make do either because there's no other alternatives and/or this product meets their needs or is the market leader. However, there is still always a tradeoff between quality and time to market. It's impossible to get perfection and sometimes good enough is well... good enough.
There's an even simpler reason: learning requires feedback. Feedback on some meaningful decisions takes years to come back in. Without it, no learning.
I agree, but I would like to give you an idea about what could be done: sell liability protection.
This is something I am working on doing, so it's not entirely coming out of thin air.
The idea is this: I am writing software to be Quality from the start. Obviously, it will be FOSS.
Now, I could go the route of selling "support" (i.e., bribes to get bug fixes), but I don't like that because it incentivizes putting off fixing bugs until a customer runs into them because that will "convince" the customer that they need support. [1]
Instead, with selling liability protection, including for outages, I have an incentive to make sure the customer does not run into bugs.
And it doesn't matter if Amazon or some other big company starts selling my software as a service because my software is not my product.
Of course, creating a quality piece of software relies on a process, as you said. My current process is at [2].
[1]: https://gavinhoward.com/2020/08/the-software-industry-is-bro...
> Obviously, it will be FOSS
That could cost you customers. I know the corporation that I worked at, would never agree to it. They were quite tinfoil about IP. They owned a ginormous patent portfolio, and were quite secretive in their processes. They would insist on keeping the source completely proprietary.
I know that you are basically selling a personal guarantee, which is an idea I could stand behind, but that is still an individual attribute, as opposed to a standard industry one. It could be a potential personal selling point, and a competitive advantage.
The real goad would be an industry-wide need for it. Basically, established precedent for suing developers (and distributors) of bad software.
That’s a pretty classic Damoclean sword. Lots of claymores, buried, there.
I feel dumb, but could you tell me what "E&O" means?
> That could cost you customers. I know the corporation that I worked at, would never agree to it. They were quite tinfoil about IP. They owned a ginormous patent portfolio, and were quite secretive in their processes. They would insist on keeping the source completely proprietary.
Yeah, I thought that might be the case. I'm okay with that. The FOSS part is because I don't believe that customers should be in be dark about anything regarding my software. It would be like an architect not giving the plans to a building to the customer. Not professional, in my opinion.
Of course, to satisfy customers like the ones you mentioned, I could just give them the source. I guess this is one point where my personal preference overrides because I believe in FOSS.
To satisfy customers like that, however, maybe it would be enough if no one knows they are a customer of mine? No idea.
Anyway, I'm okay with losing customers like that; I don't think I would need many customers to pay the bills.
> I know that you are basically selling a personal guarantee, which is an idea I could stand behind, but that is still an individual attribute, as opposed to a standard industry one. It could be a potential personal selling point, and a competitive advantage.
> The real goad would be an industry-wide need for it. Basically, established precedent for suing developers (and distributors) of bad software.
> That’s a pretty classic Damoclean sword. Lots of claymores, buried, there.
I agree that it needs to be an industry-wide thing. I hate the fact that EULA's basically disclaim all liability.
(I'm okay with FOSS software disclaiming all liability because they don't know who is using their software, and it might just be a weekend project. Actually, my FOSS licenses disclaim liability too, so customers would actually be paying for me claiming that liability again.)
But here's my hope: I hope that by doing this, and gaining a competitive advantage with it, other companies would either start to ask for it, or demand it, from their vendors, or start to do it for their software to gain a competitive advantage.
In other words, I hope that I can get the ball started rolling towards becoming an industry standard. That's partly why I wrote my process down in such detail: so the industry could start to converge on standard processes like the engineering world.
(Don't get me started on Agile or any of the other current "industry standard practices." Their foci are in entirely the wrong place, in my opinion.)
Edit: Anyway, you're right: lots of claymores there. I need to carefully define the limits of my liability with a lawyer.
Errors & Omissions business insurance. Basically goof-proofing. I keep it, even though I don't really do contract work for others.
As a contractor, I'd consider it an exception to not provide full source (and proof of provenance).
Yeah, like so many of these things, having working prototypes out there, is one way to set an example.
I'm always leery of licensing requirements. I think it depends on the application.
I do think that supply-chain dependencies should have some form of assurance, but I don't think we should expect to have a full license requirement for a brand-fluffing app.
Ah, that is good to know. I will need that for sure.
> As a contractor, I'd consider it an exception to not provide full source (and proof of provenance).
Question: does a full git repo provide proof of provenance? As in, if a repo has no dependencies, and the git history shows that you personally wrote every line, does that work as proof of provenance? I think it does, but you seem to know your stuff.
> I'm always leery of licensing requirements. I think it depends on the application.
> I do think that supply-chain dependencies should have some form of assurance, but I don't think we should expect to have a full license requirement for a brand-fluffing app.
I'm not sure what you are trying to tell me here. Could you clarify?
I would think so. You could always simply copy-paste from "unwanted" sources, but having a Git repo, with full history, goes a long way towards proof of ownership. It can also help with justifying billing (some people bill by tasks, such as bug-fixing, or adding features).
> I'm not sure what you are trying to tell me here. Could you clarify?
I think that "supply chain" dependencies (like standard SDKs and libraries that are used as a "baseline") are juicy hack targets (think SolarWinds).
I would like to think that these types of dependencies should be held to much stricter standards than maybe a dancing cat animation.
Of course, there's that whole "blurry line" thing. A device SDK might not be used by many folks, but every person that has one of those devices, would use it. If the device were some kind of forensic tool, or bespoke military gadget, then that seldom-used SDK could be a rich target. You'd want it held to the same standards (or higher), than a C compiler or encryption library.
But people that like to license, want to license everything. They would insist on holding cat animations to the same standards as nuclear launch key crypto. I've seen this in action. People like that, almost stopped all productive work at a company I worked at. I won't go into details, but it's a very real problem, when you start to have industry standards and gatekeepers.
Software needs to be flexible. It's very, very difficult to have flexibility and Quality shacking up together. In the company I worked for, the hardware people insisted that flexible software was automatically "bad quality," so we were forced to apply rigid hardware development methodologies to software development; with ghastly results.
I write about "concrete galoshes," here: https://littlegreenviper.com/miscellany/concrete-galoshes/
I'm glad I asked for clarification because when you mentioned licensing, I first thought that you meant things like software licenses. Now, I know you mean things like engineer licenses.
Up to this point, I personally would like licensing, but only for the most critical projects, such as avionics, nuclear power plants, crytography, and perhaps even foundational projects like compilers. But only those things.
So I am glad you made me aware that people who want licensing tend to go overboard with it. That means I will need to reconsider my position. I wonder how the engineering world has solved that problem, or if they even have.
> I think that "supply chain" dependencies (like standard SDKs and libraries that are used as a "baseline") are juicy hack targets (think SolarWinds).
Yeah, this is one reason I wiped all outside code from the repo where I am building my software. [1] [2] [3] [4] [5]
> Software needs to be flexible. It's very, very difficult to have flexibility and Quality shacking up together. In the company I worked for, the hardware people insisted that flexible software was automatically "bad quality," so we were forced to apply rigid hardware development methodologies to software development; with ghastly results.
I think that this can be done with good design, which is the reason why my Design phase part of process is so long.
I have many reasons to think so, but a good intro to my thoughts is "Constraints Liberate, Liberties Constrain" talk by Runar Bjarnason. [6]
Of course, this only goes so far, but I have found it works in practice.
A good example is the bc/dc I wrote. I constrained its behavior a lot with assertions and maintaining the assumptions that the assertions test. I also implement a lot of the code with respect to constant data, i.e., I use constant data to define the behavior of code. (An example of this is the list of bc keywords, or the translation between characters and dc commands.)
The end result is actually a program that is fairly easy to extend: I simply relax the assumptions or change the constant data, with some exceptions where extension is harder because of bad design, like [7].
Nevertheless, you are completely right: software needs to be flexible, and flexibility and Quality often hate each other. Maybe that's what makes software a harder thing to get Quality out of than hardware, or architecture, or engineering?
[1]: https://git.yzena.com/Yzena/Yc/commit/31b97508f
[2]: https://git.yzena.com/Yzena/Yc/commit/dd1cc3795
[3]: https://git.yzena.com/Yzena/Yc/commit/2ba7c20a9
[4]: https://git.yzena.com/Yzena/Yc/commit/4560fb115
[5]: https://git.yzena.com/Yzena/Yc/commit/3c4bce0bb
I think the jury is out on that.
The tower collapse in Miami is a prima facia example of the need for credentialed engineering. In some countries, every time they have a 4-richter temblor, buildings fall like dominoes, and people die all over. These are usually nations with lax and/or corrupt engineering culture, while Tokyo can take a 6, and people just pick up the glassware, and carry on.
I remember that story about the engineer in Oregon that was fined for claiming to be an engineer, without a license.
That was ridiculous (especially when you consider that there's a rather ... large ... corporation that has huge plants, all over Oregon, filled with uncredentialed "engineers").
Honestly, if you wrote a blog post about what you think our industry needs, I would read it. I think you have a lot to say about it.
My dance card is fairly full, right now, but I need to get around to throwing another screed onto the pile.
Will you do it now? If not, will you create a ticket? If that ticket is created, will it be prioritized?
Low quality is simply loss in that pipeline. High quality is simply being engaged enough with the project to constantly be generating these insights, and systematically running them down, until you don't have any more.
If anything, low quality is what increases cost and delivery time, by adding expensive inspection, re-work, warehousing of defective product, missed returns on development time, customer disappointment, loss of feedback.
Quality achieved by exhortation ("Try harder!") is at odds with all sorts of things. Quality achieved by straightening out a process designed to produce defects in large numbers -- not at odds with anything, except perhaps office politics.
I also stumbled over a slide deck that may or may not be an understandable summary for one not well versed in the concepts of system safety: http://sunnyday.mit.edu/workshop2019/CAST-Tutorial2019.pdf
quality and reliability almost always comes down to the system, not heroic efforts of individual employees
> We need to put care into our systems. We need to build checklists and peer review and resilience into the way we express our carefulness.
Yes – the systems help, but proper incentives are also super important. Granted, designing a system wherein the incentives are properly aligned with the goal (and not easily manipulated; shortcuts can and will be exploited by humans and ML algorithms alike) is no easy task.
https://miro.medium.com/max/1280/1*f60gJ4MELQIdOoLwTilk-w.pn...
It's a very successful life hack - system-based approaches can be less prone to errors because it relies less on willpower and individual conscientiousness, both of which can stumble at random points.
This reminds me of the book checklist manifesto [1] Even simple tools like checklist can substantially decrease errors and spare much of our cognitive capacity in doing quality work.
[0]: https://www.oreilly.com/library/view/the-field-guide/9781317...
I would really like to read more of his takes on systems design, in part because I like bite-sized stuff,
but also in part because I want to see aspects like complexity and modeling either embraced or ignored by someone like this. I want to watch that particular bout and see what happens.
Feedback. Signals. Systems theory.
Fool-proofing, in essence (though not the literal translation). For those, like me, who knew they knew the term from somewhere but couldn't place it, or who just don't know it.
Related idea in software is to make invalid states unrepresentable in the system. It's also a central theme in systems safety.