QA = Time and Money. How much should you invest?

Even if your definition is more logical, this isn't how language works.

There are things that can be measured that might be correlated to quality ("bugs", sales, support, etc) but ultimately, classical quality seems to be to be a marriage of what was expected or desired with what was actually there. As a developer, my take is that quality usually stems from "developing" those expectations as much as making the relatively concrete thing to compare them too. When anyone can quantify the first half of this equation, please let me know at once.

First, you have to understand your company from a systemic viewpoint. Break it down, and understand the inputs and outputs of every part of your company. You've got your individuals, your various teams, the people they work with, the systems they build, and the product itself. You can measure the input and output of each of these systems in meaningful ways: number of tickets opened, number of defects reported, raw performance metrics, uptime, availability, quantitative user feedback like NPS, and engagement metrics like product usage or feature usage.

But there are caveats: you can't turn these metrics into goals. You need to lead toward the improvement of the systems responsible for those metrics, with the hypothesis of improving them and ultimately the end goal of providing a high quality product of value to your customers. The metrics are valuable information about how the system operates, not the end goal.

There's another problem: you're absolutely right about quality being fundamentally qualitative. It is, after all, the root of the word "qualitative." That's significant. Quality is the humanity of the engineering game: while you can measure a lot of things toward it, ultimately, as W. Edwards Deming said, "The most important things cannot be measured." It sucks, but it is closer to reality than defining any metric or set of metrics. Instead, it requires leadership and a comfort with doubt and complexity. Here's my answer to a Quora question expressing discomfort with this paradox: http://qr.ae/f5iUg

So it's a bit of a Bermuda triangle. First, there are significant metrics you can measure, and you need to decide which ones mean quality to you and your customers—I promise they exist. Second, you can't aim directly for your metrics, since any large enough organization is highly complex and over-optimizing for a single metric or set of metrics can cause significant unintended side-effects. Third, the most important things critical to quality actually can't be measured; or measuring them would cause more harm than good. This makes for a challenge, and it's why this is such a difficult problem to solve in an organization.

What I'd LOVE to see moving forward is instead a profound comfort with complexity, and an understanding of why numeric metrics aren't always the most vital ingredient to success.

Integration tests are more brittle. Which, can be good ("Hey, there's a bug here..."). For an early product, your app is probably simple enough that you get a good enough feel while dogfooding. 100% correct that also includes some informal exploratory work :D

RE: Complexity & the relative need for testing, I really like the perspective 'Out of the Tar Pits' takes. http://shaffner.us/cs/papers/tarpit.pdf

The first is that integration tests can be (not always, but can be) too inclusive and thus fail too much, especially early on. A good test process requires tests to stay green most of the time, otherwise you learn to ignore them as "known" and distrust and eventually dismiss the results. So you turn off or xfail failing tests while you triage. But if you turn off or dismiss an integration test as a known failure, you lose a lot of coverage.

The second is isolation. Even if you don't have full unit coverage, having -something- beneath the integration tests lets you tell a lot more from the combo of failures. Integration covers ABC and fails, have unit test for B and C, must be A.

The third is that they're for different purposes. Unit tests are there to tell you what you thought you were guaranteeing with your code has been guaranteed. That's a really important step in knowing that all the other stuff you -didn't- cover with integration testing (remembering that as you add moving parts, you have the combinatorial of all the ways they move--you won't cover it all) is probably correct. And, of course, it's what will ever let you refactor while still knowing your interface is correct.

Finally, there's simple terminology differences as to what's a unit and we might be on the same side. There's debate in the community, but I'm on the side that you always unit test to a public interface, period. Don't test within something not exported or exposed. That may mean you only test a module instead of component classes if they're all private implementation classes. It's still a unit test for that module.

Also, a political reason: once you start down that route it's super-tempting (especially for your boss or your boss's boss) to say "we'll test everything full stack, that gives us all of the coverage." In reality, it only gives you some of it, and you probably have no idea what even with code coverage tools (remember that paths are ultimately what you're testing, not lines or branches). And it's incredibly indirect, since you're trying to find externally driven cases to make the internals do different things. It's usually easier just to manipulate the internals directly via the nearest interface.

One more point: YAGNI is fine and well, but we all know modules or classes usually have behavior that isn't used yet because it's part of an obviously complete API (like, D from your CRUD API when nobody's had to remove anything yet). Unit test that stuff, since there is no integration test that'll cover it. Otherwise it's landmines waiting to be found. The unit tests -are- the client code at that point.

So one or two high level acceptance/integration sanity tests, sure. But I'd never say that you should have "integration testing in place" first. That implies a level of completeness or formality that I think would probably be counterproductive in almost all cases.

It goes the other way as well. Bringing agile methods to a situation in which having a requirement changed can take three months. Situations in which your "customer" is a faceless defence ministry (or two) with the level of responsiveness you'd expect. Processes are tools; use the right tools for the job.

As for bugs being literally unknown, I'm jealous--it'd be nice to claim I ever guided a project to that state.

I certainly can't make that claim. I was only there for five years; I think I saw three staged releases (which were for the purpose of allowing integration testing with other components). Maybe two. :)

To me, writing thorough documentation and solid tests from the beginning serves a very important purpose that is often overlooked: it establishes the right type of culture. I think of it as laying the foundation of a building. If you skimp out on materials, the whole thing will be unstable and prone to collapse. And going back later to fix it is often times a lot more costly than doing it right in the first place.

You would never make the argument that a solid foundation is unnecessary if an earthquake never hits, right?

However, I think most consumer or enterprise software companies greatly undervalue quality and its impact on their competitive value proposition, and especially its impact on development efficiency and speed.

Even if the customer isn't willing to pay for higher quality, your developers are still paying for it with their time and ability to innovate; and your competitors will happily eat your lunch while you frantically hire "better developers" and use up all your time putting out fires.

There is a diminishing return on level of quality and level of QA done; you need to do what's appropriate for your market and your customer expectations. I just don't believe that most companies are anywhere near the appropriate level—not just of after-the-fact testing and best practices, but also in up-front built-in quality, design investment, and debt management.

It is a bit strong, isn't it? I felt that to caveat it might detract from the effectiveness of the point. There does come a point where the software is effectively flawless from the customer's point of view, and doing anything to improve it (including more/better QA) from there doesn't make any more money. But if you reach that point, you know :)

That said, I liked the article - thanks for sharing.

The two most useful code coverage stats I've found are 0 and "less than yesterday." The former tells me something pretty important about the culture of the team who owns that code; and having rules against the latter is about the only way to ensure a test-on-checkin policy.

For some but not all of my projects, failure was not an option. Imagine I lost a disk sector in my storage firmware.