Hacker News Reader

> We have software QA departments that operate very much like manufacturing or aerospace QA.

I don’t work in this industry, but this seems fairly ridiculous on its face: software is not at all like manufacturing.

In manufacturing, there’s a design and a manufacturing process, and a critical function of QA is ensuring that the manufactured produce is manufactured to spec.

With software, the software is written, compiled, and then repeatedly copied. And something should verify that it’s copied correctly, but this is straightforward and boring.

So software QA ought to be much more like the kind of validation that happens when designing hardware, not like the kind of testing and validation that happens as products are manufactured.

Ideally there should be a solid spec written and then qa can test against the spec. Maybe there is somewhere that does write solid specs, including accounting for corner cases, but in my 25 years working professionally in the industry, I’ve never seen it.

The only complete and precise specification of software is the code itself. If some other form of specification was complete, we would be able to auto-generate the code.

This is beside the point. The code specifies what the product _is_, not what it "should" be. If you ask for a word processor and I deliver a perfectly bug-free and feature-complete calculator would you really believe it lived up to spec?

This is also beside the point. I think both of you are trying to warn against the dangers that lie on both sides of this coin: people can invest too heavily in a specification and waste an enormous amount of time, and people can immediately jump into coding and code something that does not do what it was intended to do. Like with all things in life, there’s a balance between these two extremes that’s correct.

You need some level of specification so you know what you’re building, but you have to keep in mind that the final code defines what the behavior truly is. Sometimes, that behavior unintentionally becomes part of the specification because users begin to rely on it.

I do like the fact that you both used hyperbole to succinctly illustrate the dangers of veering too far in either direction though :)

A (human language) specification is simply _enough_ information about a system that a human can figure out the intention of the author. The smarter and more context-rich the human, the simpler the specification can be. The dumber and less context-rich the human, the closer the specification needs to be to code.

It's asymptotic. By the time you reach a human who is as dumb as an actual computer, the specification _is_ the code.

I work in MedTech. We do this. A design has to be reviewed by QA, and is then tested, and the test is reviewed again. So just to counter the narrative, there are companies that do that, and it is working. In other jobs I also saw the cargo cult of QA. But in some industries it is just crucial, otherwise the pressure is too high to cut corners to implement something. It is a good mechanism to counter the need to move fast and break things.

I work in software for CPU design/verification. Even here, where in theory there should be a rock-solid spec, there's not. There's a 12,000 page architectural specification, which is very helpful for specifying all the end-user visible state. But the microarchitectural specification is scattered all over different PDFs, visio docs, excel sheets, and sometimes the only spec is the RTL code itself.

> this seems fairly ridiculous on its face

I think that's why people always tell each other to not take things at face value.

Of course there is a big difference between sw and hw QA, in the thing that they test, and how they test them.

But they are also very similar. Any QA department has to think about ways that things can go wrong, and what things to test for, how to test, which testing methods, which standards to handle, keeping certifications, etc. During testing you also need to keep reevaluating if you actually are catching each problem/bug and how to implement changes in your company that decreases the amount of problems or increase the amount that you catch.

I think in that way there's a lot of overlap in thinking about business processes and how to identify problems with them.

Of course once a specific binary gets tested and approved by QA it shouldn't matter if it gets copied or whatever as long as you make sure its the same binary (by a checksum for example).

But still making sure that errors don't reach the customer, is vital in any QA. If errors does happen, QA is the department that can make sure that it doesn't happen again. And ofc be able to proof in court that you did your due diligence if something does happen.

I’m going to be using “moral crumple zone” in every conversation I possibly can from now on.

"Hey, wanna hang out with a beer this evening?"

"Nah, I don't need to slouch anymore in this moral crumple zone!"

"We need this new feature in our program!"

"If we implement this it means the management fell in a moral crumple zone"

"What seat would you have to have for your flight?"

"Anywhere, but not in the moral crumple zone, please"

An example from my actual work life, lightly fictionalized—

Them: “Please review this design.”

Me: “Ok, sure, when do you plan to start coding?”

Them: “Oh, it’s already in beta.”

Me: “So you can’t do anything with my feedback, but you’ll say I reviewed it?”

Them: “Well…”

Me: “You’re putting me in a moral crumple zone here!”

It sounds like they are calling something QA but using it as a liability shield. It makes sense that you are upset about that, but naming something QA and having it do something else doesn't mean that QA as an effort is bad. It means that the people doing that are being deceptive.

Fair point, you are correct in your inference that there are some bad actors in my workplace. However, I’ll argue that the fundamental dynamics of bifurcating the responsibility of quality from software leads to a steady state where all QA departments end up as a liability shield given enough time.

This is driven by Pournelle's iron law of bureaucracy [1], which says that people who promote the bureaucracy rather than the mission of the bureaucracy will get promoted within the organization and come to dominate its decision making.

For example, in schools, administrators make more money than teachers. This is despite both groups having similar levels of education and intelligence. The reason for this is that administrators know the laws and regulations of the environment they’re working in and ensure the continuity of the organization. Despite not directly contributing to the organization’s stated mission of education, they are in charge of the organization and take more benefits from it.

Software QA has similar dynamics. A QA department may start out making good faith contributions to the organization. Eventually there are product failures, eventually leadership needs a scapegoat to show they’re doing something, and eventually QA takes the blame. People get moved, demoted, or fired. QA realizes its risk, and takes steps to mitigate it. They create a highly systematized workflow and process, adopt or introduce standards. Then assert that following process equates to good outcomes. When bad outcomes occur, they point to their strict adherence to following process as evidence of innocence.

If the process does not support the work or mission, that is a cost they are happy to impose on other functions to deal with. This is the final state until a system disruption happens.

[1] https://en.m.wikipedia.org/wiki/Jerry_Pournelle

But what makes 'software QA' fundamentally different than 'non-software QA' to give it the problems you foresee?

Because every QA is something new for development barring regression testing.

An equivalent software QA to building planes would be to verify a known process with existing tooling.

Fundamentally different problems.

Manufacturing process builds identical widgets using standard equipment. Widgets are inspected to confirm they are within spec. Frequentist statistics are used to determine when widgets are consistently out of spec. When this happens, equipment is inspected and repaired as a corrective action. This process is well defined and linear.

Software produces bespoke, non-standard widgets to address domain specific needs. At the end of the day, software developers are defining a process for machines to follow. If you want to control quality in the software development , aside from having perfect domain knowledge for a particular project, the only way to do it do it is to define an arbitrary process for developers to follow and track adherence to it. This may have no impact, or be a hindrance. It will never add value because it will never be abstract enough to be appropriate for every domain.

I have seen a case of Software QA taking a very different shape, so I'd like to argue that the outcome you describe is not intrinsic to software QA, but rather to company culture.

The case I'm talking about does not have a separate QA department, but QA people as part of every software team. If a product fails, that team is responsible, so software devs are in the same boat as QA. They focus on learning from these failures, so no scapegoat is needed. Process does get followed, but not as a defense mechanism, but because not doing so introduces noise that is an obstacle to improvement. In case of bad outcomes, people do point out that they followed process because then it is clear that the process is involved in the failure and should be improved.

Unfortunately, companies with that kind of culture are rare.

I can see integrated QA working for teams because QA personnel would understand project specific constraints and degrees of freedom and tailor solutions in a way that top-down QA cannot.

However, there are situations where less QA may be needed, for periods of time, such as when PRs may be low. QA may be seen as overhead by management, and something to reduce. This will lead to QA shared between teams and a push for standardization and top-down process deployment to minimize complexity for these personnel will develop. Complexity to manage the QA personnel will be shifted to development teams.

This situation absolutely is controlled by company culture. A culture that neither values QA nor development will do this. A company under financial strain will do that. Companies wax and wane constantly.

It's like saying communism isnt the problem, but that it s how every single group attempted to implement it that should be blamed.

Sure, maybe, but if nobody ever can implement the theoretical utopia, maybe we should talk of things humans can do instead and ditch the unimplementable idea.

QA cannot be done by a separate team the way you dream: it will always be a political buffer zone staffed by the cheapest half-competent people you can find, expulsing good people into dev or management. Or you merge it into dev/solution design.

The reason is simple: just like contract law, you only care about quality once you are in trouble and need to reverse back the source of the issue to give to the client a post mortem. Otherwise, you care first about velocity, or $ input/hr of effort.

Other fields do QA just fine.

About 2000, Software QA (and almost all traditional QA activities) were changed. The focus was on process over inspection. "Design in quality, do not inspect it into the product"

Suppliers (to include software) were expected to manage the quality of the product they provided; the purchaser would focus on how they managed the process, not in the compliance of every part.

This had a chance until software process was tossed in the name of "agile".

Here's a stupid question: How do you know your process is good unless you inspect it?

"Hey Bob I know you're a competent engineer, but don't worry about specifying a certain type of bolt or loctite, the untrained assembly personnel will figure it out. I'm sure they won't let 200 people die in a plane crash."

Make it impossible to use a wrong bolt, and train the assembly workers.

I recall a bug I was involved with at a telecoms equipment market in the early 2000s. The bug only showed up in our biggest base stations in high load situations. We diagnosed the bug, and there were a couple of parts to it. Sloppy software design in an optional hardware module (no state machine) was one part - and was fixed. But there was another underlying issue in the way message queues were handled.

Anyhow, the fix for this was created and written. But we never got to put it into production. The reason: the company didn't have a lab test facility that could put a sufficient load on the software to prove it. Even though we were getting field failures because of this issue that were getting a bad rep, we couldn't fix it because even though the old code was known to be buggy, we couldn't prove the new code. So the process said we couldn't ship it.

Another way of looking at that is that within the ability to test, the implementations were indistinguishable, so the process mandated that the older implementation must be used. I wonder if they would have explicitly specified age as a metric if this was considered when designing the process.

> Good software teams build quality into projects by introducing continuous integration, unit testing, creating feedback, and tightening these feedback loops.

Agreed, for good software teams.

I would content that most software teams at most companies are not good.

Which is to ask, with an average to bad software team is it better to have integrated or separate QA?

Does it make sense to degrade the performance of good software teams because bad software teams exist?

Ideally we’d always have good software teams, but in the real world sometimes you have to build software with bad teams.

Leaders have options, they can do things like reduce scope, increase budget, increase schedule, or full on abandon or cancel the project. These are all options available to leaders, but they require tradeoffs and decisions to be made on a project by project basis.

It is scalable to have a strict process that everyone has to follow, then impose a watchdog to enforce it on a wide scale. It may not be better to have separate QA, but it is easier for those in charge.

It makes the most sense to me to match the org structure to the teams you have.

If I'm trying to build something with undertrained, demoralized, underpaid engineers... it's not optimal to use methods intended for self-motivated, high-performance teams.

And nothing says there must be company-wide mandates. Maybe this area gets a formal, independent QA team, but this other area doesn't.

My experience just doesn't bear out that collapsing the QA function into development always leads to better outcomes.

I've seen the opposite happen too often, and QA be the sole bulwark between idiocy and customers.

> Ideally we’d always have good software teams, but in the real world sometimes you have to build software with bad teams.

Here's a real, perhaps unexpected counterpoint. Say you have a good software team. How do they build good software with bad management?

They quit.

Exactly!

Quit and do it for someone else

> Does it make sense to degrade the performance of good software teams because bad software teams exist?

Consider the classic statistic "most drivers think they are above average".

I posit that the same is true of software teams, almost every team will self-assess as above average, i.e. good. Those teams will then imagine that, being good, they build quality into the process and very little verification QA is done.

I have worked as a software consultant for 15 years now. I've worked with at least 40 separate software teams in that time. Every single team manager would pep talk with "this is the best team I've ever seen". Some of this is obviously blowing smoke to get people to work harder and feel good. But over the years I've had candid conversations with managers and realized that most of the time the genuinely think their team is really good, truly top 10-20%.

Here's the rub. Being a consultant, I'm almost always brought in by higher level management because something is going horribly wrong. The team can't deliver quickly. The software they deliver is bug ridden. They routinely deliver the wrong software (i.e. incorrect interpretation of requirements.)

Often times these problems are not only the fault of the development team, management has issues too. But in every single case, the development team is in dire straits. They have continuous integration sure, and unit tests, and nightly builds, and lots of green check marks. But the unit tests test that the test works. The stress tests have no reality based basis for expected load. The continuous integration system builds software but it can't be deployed in that form for x, y & z reasons, so production has a special build system, etc...

In 15 years I have never once encountered a team that would not benefit from a QA team doing boring, old school, black box manual testing. And the teams that most adamantly refuse to accept that reality are precisely those that think they are really top tier because they have 90+% unit test coverage, use agile and do nightly builds.

So, my question is, do you (I don't mean the specific "you" here, rather everyone should ask themselves this, all the time) think that most bad software teams know they are bad? Including the one you are part of? Would it really hurt to have some ye olde QA, just in case, you know, you are actually just average? :)

I'm curious: in your many years of being a consultant to these bad teams, where the manager really thought they were top 20%, did you get a chance to talk to the rank-and-file team members, and did they paint a very different picture of the team health and software quality than their manager?

Also, did you run across any orgs where they basically refused to use a process like Agile, and instead just did ad-hoc coding, insisting that this was the best way since it worked just fine for them back when they were a 5-person startup?

Not parent, but in my experience as a consultant working with bad teams, the rank and file were 'doing the job.'

You usually had a few personality archetypes:

- The most technical dev on the team, always with a chip on their shoulder and serious personality issues, who had decided to settle for this job for (reasons)

- The vastly undertrained dev who was trying to keep up with the rest of the team, but would eventually be found out and tossed, usually to blame for a major issue

- The earnest and surprisingly competent meek dev, who presumably didn't have enough confidence to apply to a better job, but easily could have made it on merit, work ethic, and skill

- The over-confident dev who read a bit of SDLC practice, and could see every tree while missing the forest

The key is that, aside from the incompetent person, they had all always been working there for awhile. Consequently, there wasn't good or bad health and quality: there was just "the system" (at that company) and dealing with it.

And none of these folks ever worked at 5-person startups. ;) I think it was definitely more an issue of SDLC "unknown unknowns" they should be doing, than willful decisions not to.

> I'm curious: in your many years of being a consultant to these bad teams, where the manager really thought they were top 20%, did you get a chance to talk to the rank-and-file team members, and did they paint a very different picture of the team health and software quality than their manager?

Yes, generally I join teams and work as an engineer or sometimes as a team lead, so I'm talking to all the team members.

Most start up teams are composed of junior developers, often pretty smart people. Usually 5 or fewer years of experience. Many times these are people who have already accomplished stuff they didn't think they could do. So that generally means that yes they think pretty highly of themselves. To a degree it is quite justifiable, they tend to be very accomplished but in a narrow domain. Unfortunately they don't realize that their technical accomplishments in a specific field does not mean that they are experts everywhere. Their managers understand that these are smart people and assume again that this is therefore a good team.

Non start ups that I join are usually just plain dysfunctional.

> Also, did you run across any orgs where they basically refused to use a process like Agile, and instead just did ad-hoc coding, insisting that this was the best way since it worked just fine for them back when they were a 5-person startup?

Usually more the opposite. In my experience I come across teams that are sure they must not need any help because they follow all the rules in Scrum and have great code coverage metrics.

It is really common to see this kind of thing. I call it "the proxy endpoint fallacy". It can crop up anywhere that there is something that can be measured. In that example, it would be confusing adherence to Scrum with having a working SDLC or perhaps confusing code coverage metrics with the objective of having bug-free releases.

This isn't a software only fallacy. In politics, GDP is often confused with societal well-being. Always be wary of your metrics and change them as required to keep you tracking your actual goals.

Depending on the shape of the distribution, most drivers could be above average. Average doesn't imply 50th percentile, that's what the median is for. A minority of tremendously poor drivers could certainly mean that most drivers are in fact better than average, in the same way that my friends on average have more friends than I do.

If your devs aren't good what are the chances of your QA team being good enough to make up for their short comings? The dynamics laid out by the parent comment will just hit even harder. Your best bet is to enforce basic practices like continuous integration, coverage goals and maybe a coverage ratchet as a merge gate. Training and education on areas were the team is weak is also a must.

I'm not going to argue with the general thrust of your comment, which I think is insightful as to how incentives can compromise objectives. But...

> To give an example of how these dynamics destroy quality in software. I once found a bug in software on a piece of test equipment where a logarithmic search function was not operating on a strictly sorted list. When I pointed this out to my leadership I was told that if we changed any part of code, it would require a new FQT, which would be too expensive to conduct and was not in the budget. Although the bug would have been trivial to solve, and was clearly wrong and would not provide any benefits by remaining in the test equipment software, the process required for changes prevented solving the issue.

I've seen this happen where it was a bad thing, but also where it was a good thing.

It's all about risk.

What risk does the software defect pose to the mission? What risk is inherent in making any change to the software? Noting that even trivial changes can be fat-fingered and thus are a source of risk. I've seen it go wrong this way: a seemingly trivial change was made, but the developer accidentally checked an extra file into source control, causing a further defect.

And then: what is the cost of mitigating these risks? Maybe the software defect is as trivial as its fix. Maybe an acceptable fix would be to write up a workaround in the documentation.

I don't think it's always wrong to say no to fixing issues. I also don't think it's always right that a separate QA department contributes nothing to the organization, even if they act as a handbrake on the software developers (sometimes, precisely because they do that). Human factors are real.

> Good software teams build quality into projects by introducing continuous integration, unit testing, creating feedback, and tightening these feedback loops.

No. Good software teams are led by competent, technical management. Managers who aren't afraid to get down into the dirty details. Managers who aren't afraid to roll up their sleeves and write code if they need to.

The process doesn't matter. The management of what is or is not important does. Agile is just one process out of many.

Imagine an accounting team led by someone who never did accounting in their life: "Just make the numbers work out! I don't care how you do it! My bonus is at stake!"

But that doesn't contradict the parent, does it? I'd say you both make good points.

Well... here's a thought experiment.

Let's say you have a bunch of school children and architects create a skyscraper. I've given both groups the process to design a skyscraper.

So in both cases, I should end up with a safe building?

Why not both? Am I missing something? You can have feedback loops and CI and all that, "good craftsmanship" or "good practices" (not "best" practices because those often suck hah), where of course opinions vary on the details of that -- and then someone who is also good at the craft who spends more or most time on helping the rest work together, i.e. manage/lead them.

I’d bet the children would come out better simply because they have parents who are likely multi-disciplined as a group. A disparite group will (almost) always come up with better results than a homogeneous one (at least in my experience)

> Good software teams are led by competent, technical management.

...or perhaps with no managers at all. I'm less and less convinced of the importance of management in engineering except to give investors an illusion of control.

I sort of agree, and I do think it’s possible depending on the team. But unfortunately developers can be too opinionated and get focused on low priority things.

Sigh... This myth that the only people who can competently manage developers are other developers has been floating round for decades.

For some reason, developers seem remarkably blind to the skills other roles and disciplines require. Only a developer can do that, everyone else is basically useless fluff. Maybe it's a form of arrogance or just deep unself-awareness.

Let's apply your reasoning to medicine. I'm sure you would be completely fine with managers telling your surgeon what parts of the surgery can be 'optimized away'.

Haha, funny strawman. My reasoning is that non developers are capable of managing developers, notably people who have good management skills.

Your contention is that the surgeon should be running the hospital.

Hahah, indeed. So have you seen a law department in a company headed by someone who doesn't come from law background? How about a finance department headed by some schmuck who doesn't know anything about finance?

I've seen plenty of departments managed by people who don't come from the background of the department. My current boss is extremely good and came from a different discipline.

Although I don't deny it can help to have the background, it is not necessary to be a good manager of something. Also seen plenty of good techies promoted to management and failing badly.

This is a lie, and you know it. Even a mere idea about lawyers being managed by a non-lawyer would be laughed at. Same with finance, nobody would be stupid enough to even try it

It's ok to disagree, we don't have to accuse each other of mendacity.

I stand by what I said, although my experience is in the technical domain, not finance or law. Maybe those departments are different, I don't know.

I disagree with you. You are stating it, but you are not giving reasons. Managers who weren’t developers tend to not be able to manage the team. They can’t help with or understand the technical decisions made. The non-technical managers tend to be project managers just focused on dates.

We may be talking about different levels of management here. A manager should not be making technical decisions, they should have team leads and architects who do that. It's their job to manage the team, interface with the business, prioritise work and give cover to the team so they can get on with it.

I guess if you have a manager who is making technical decisions, they are really a hybrid manager/contributor role. Maybe that works better in smaller organisations.

Then what exactly is the non technical manager's added value?

He has no experience to lead the team in high pressure situations. Like production being down.

He can't truly have a first person understanding of the work of the people who he manages. He has to rely upon others to tell him who's good and who's bad. That sets up a pecking order.

He can't help or mentor engineers with design decisions, or provide a historical context.

He doesn't understand the technology so there's an immediate communication and knowledge barrier that has to be overcome between him and his directs.

He doesn't feel the pain of a bad decision, because he's not coding it, and he can't emphasize with them since he doesn't code.

He tends to push feature development without fixing technical debt. Again that's pain he personally doesn't feel.

To me, the role you are describing is a principal engineer or team lead, not a manager.

Simply not true however that a good manager can't lead the team in a high pressure situation. I'd say that exactly what a good manager could do well. Obviously they won't be making overtly technical decisions, that's what you are for. They can make business decisions, provide cover, get resources, communicate to other stakeholders... All the bits that need doing but would be a huge hassle for the techies who are trying to fix the issue.

I don’t see much value added by a manager that you described. What you are describing is more of a product manager role.

Not everything of value is technical. That you don't see the value is either because you have great managers shielding from having to deal with all that, or you have always worked in a place that combines management and technical responsibilities (which I never have except for very small companies).

I wouldn't call what I'm describing as product management, although it's possible they could do general management too.

Sure, but I feel it's easier to teach management skills to technical person, than it is to teach tech to someone that's only been in management.

The head of surgery should be a surgeon, not an accounting manager.

The head of accounting should be an accountant, not a surgeon.

And even at the executive level of a hospital, you would want people who have spent their careers in healthcare, rather than, say, architecture.

How to tell everyone that you never worked inside a hospital ...

I think you're sort of misunderstanding the role of QA.

You think that QA is a liability shield, but that is only a side effect of the work that they actually do.

The task of QA is exactly that: an entity that tries to assure that the quality is up to some standard. Even in favourable conditions mistakes happen, so how do you make sure as a company that not 1 in every 100 product are faulty and tarnishes the good reputation that your company has spent so much time and money on to build? You hire a QA to make sure problems get caught before delivery.

But if all humans make mistakes, and QA is human, how do you make sure that the QA doesn't make a mistake? A never ending chain of QAs expecting each other?

No of course not. One thing that helps with reducing errors is to have a rigid protocol that is followed to the letter everytime. Pilots, for example, have a preflight checklist that they have to run every time they operate the plane.

The rigid protocol of QA teams is therefore an essential part of their jobs.

Although from your standpoint as a developer it might seem strange that QA is 'preventing' you from fixing a bug, it is actually very reasonable.

Especially since you work in the defence industry, I hope you understand that it is very important that the software that operates radars, planes, missiles, bombs, etc is working exactly as expected. Understandably there is a great deal of effort made to assure that when those things are needed they work exactly to spec.

So in your example it is probably very reasonable that any change you make needs to go through some rigorous process. The fact that it 'only' was about test equipment, doesn't matter because test equipment is just as, if not more important as the stuff it tests.

The reason why QA has the side-effect of being a 'liability shield' is that it gives companies the ability to argue (and proof) after the fact that the company did their due diligence in making sure that the product was to spec.

Especially certification is basically to get an external organisation to approve your QA. In that case if you get sued you can rightfully claim that you did everything that was legally asked of you, and if there is blame, then it is the certifying company using insufficient standards.

High quality comment