Comment by sz4kerto | Hacker News Reader

Comment by sz4kerto | Hacker News Reader

"Before I weigh in further, I’d like to make sure you’re familiar with the testing pyramid."

The testing pyramid is a par excellance SWE kool-aid. Someone wrote a logically-sounding blogpost about it many years ago and then people started regurgitating it without any empirical evidence behind it.

Many of us have realised that you need a "testing hourglass", not a "testing pyramid". Unit tests are universally considered useful, there's not much debate about it (also they're cheap). Integration tests are expensive and, in most cases, have very limited use. UI and API tests are extremely useful because they are testing whether the system behaves as we expect it to behave.

E.g. for a specific system of ours we have ~30k unit tests and ~10k UI/API tests. UI and API tests are effectively the living, valid documentation of how the system behaves. Those tests are what prevent the system becoming 'legacy'. UI and API tests are what enable large-scale refactors without breaking stuff.

Isolated QA should not exist because anything a QA engineer can do manually can be automated.

everyone gets the pyramid wrong in my opinion.

the vertical axis is not test type. It is would you run the test. At the bottom are deterministic fast tests for something completely unrelated to what you are working on - but they are so easy/fast you run them anyway 'just in case'. As you move up you get tests that you more and more want to aviod running. Tests that take a long time, tests that randomly fail when nothing is wrong, tests that need some settup, tests that need some expensive license (i can't think of more now but I'm sure there are).

You want to drive everything down as far as possible, but there is value in tests that are higher so you won't get rid of it. Just remember as soon you get to the 'make would run this test but I'm skipping it for now because it is annoying' line you need a seperate process to ensure the test is eventually run - you are trading off speed now for the risk that the test will find something and it is 10x harder to fix when you get there - when a test is run all the time you know what caused the failure and can go right there, while later means you did several things and have forgotten details. 10x is an estimate, depending where in your process you put it it could be 100 or even 1000 times harder.

The testing pyramid comes from a time of desktop apps with no API and when UI tests were extremely expensive. I made 100% sense in that context, it never did in other contexts. Despite its omnipresence it had not made any sense for the vast majority of us in the past 25 years.

> Isolated QA should not exist because anything a QA engineer can do manually can be automated.

I almost entirely agree. You'd certainly want a majority of QA responsibilities to fall to devs since that keeps the feedback loop tight. Also with the amount of compute we have you can trivially outpace a typical QA team with a single dev working on fuzz tests and property based tests. If you really still feel the need to outsource then there are cool tools like Antithesis [1] which let you trade money for compute in their complicated fuzz testing suite.

Where that falls apart imo is when your software interacts with hardware in any non-trivial way. Fuzz testing doesn't really work when your system is spinning big and heavy things for example. Real hardware that interacts with the world will always create this gap where the devs don't know how to fully test it. Your systems engineers are best equipped to handle this and ensure certain controls follow the right curves, but they typically don't have the bandwidth or are very inexperienced with software. I think a QA organization _can_ fill this gap and deliver value. But I'd almost always prefer hiring another 1 or 2 system engineers who can work towards this problem full time instead. It's much easier to train someone in software than it is to train them in motor control or sealed systems.

[1] https://antithesis.com/

I’ve never encountered an initiative to “shift left” that wasn’t directly motivated by clunky, slow, unreliable and unmaintainable E2E tests. Failing earlier, especially pre-deployment, with targeted integration and contract testing is fabulous but it can’t replace rubber hitting road.

I’ve had quite a bit of success in helping my dev teams to own quality, devising and writing their own test cases, maintaining test pipelines, running bug hunts, etc. 90% of this can be attributed to treating developers as my customer, for whom I build software products which allow them to be more productive.

> because anything a QA engineer can do manually can be automated.

Looks like you never worked with a decent QA team and do not understand the full scope of quality management. They have plenty of creative tasks not aligned with other roles.

> Isolated QA should not exist because anything a QA engineer can do manually can be automated.

Well, sort of maybe, but it's not always economical. For a normal web app - yeah I guess. Depends on the complexity of the software and the environment / inputs it deals with.

And then there's explorative testing, where I always found a good QA invaluable. Sure, you can also automate that to some degree. But someone who knows the software well and tries to find ways to get it to behave in unexpected ways, also valuable.

I would agree that solid development practices can handle 80% of the overall QA though, mainly regression testing. But those last 20%, well I think about those differently.

> And then there's explorative testing, where I always found a good QA invaluable.

Yes, I agree. We do this too. Findings are followed by a post-mortem-like process: - fix the problem - produce an automated test - evaluate why the feature wasn't autotested properly

> it's not always economical. For a normal web app - yeah I guess

What do you define as "normal"? I can't think of anything harder to test than a web app.

Even a seemingly trivial static HTML site with some CSS on it will already have inconsistencies across every browser and device. Even if you fix all of that (unlikely), you still haven't done your WCAG compliance, SEO, etc.

The web is probably the best example case for needing a QA team.

Reply on news.ycombinator.com