Comment by tmcneal | Hacker News Reader

Comment by tmcneal | Hacker News Reader

Parent Full thread

We sell a product in the QA space and get to talk to a lot companies about this.* The pattern I see is that companies in the 1-10 employee range tend to have a QA process that comes about organically. You start out with zero customers so a bug doesn't really "cost" anything. As you get customers, the cost of a bug grows to a point where it makes sense to do manual test passes - maybe before a feature release, maybe after every deployment. Usually it's in a Google Doc or Google Sheet, and it's either one person running through the whole thing or a bunch of people divide-and-conquering it.

Eventually QA ends up being owned by either the Engineering team or a standalone QA team. I've seen both work, and both not work. Generally if it's not working for Engineering, it means they're not automating enough or the tests are flaky and they're just ignoring it. If the QA team approach is not working, they're probably overworked... Likely because the engineering org has deeper quality issues they need to fix, but they're trying to make up for it by throwing bodies at it in the QA org.

* We have a no-code test automation product that is used by both QA and Engineers: https://reflect.run

I think "manual test pass" is very seductive and very dangerous. Compared to writing good automation, manual testing once is cheaper, easier, and faster. Manual testing indefinitely is more expensive, harder, and slower. Short term thinking gets you stuck with flawed and burdensome manual testing.

In my view, the best approach is rigorous automation and manual exploratory testing which will find the kind of bugs automation misses and inspire new test cases.

There's definitely an inflection point.

I think automation is often a waste of time at the beginning of a project too. Bugs don't cost much (if anything). Building functionality that gets tossed away is routine. Tests don't provide value - getting the code in front of customers and seeing if it's what they really want provides value.

The problem is that once people get used to not having automated tests is can be difficult to build up the habit.

Unit tests are also often hard (if not impossible) to retrofit, although IMO that's a bigger problem with unit tests than it is doing testing at a later stage in the project.

The problem is not about automation or manual but creating and maintaining a good test plan (including exploratory tests). And we just simply cannot rely on engineer for good test plan. After having a good test plan we can decide to how much we should have for manual vs automation.. the simple rule of software dev is use before reuse, manual before automation.. but some are just too much obsessed with automate everything motto.

Reply on news.ycombinator.com