Signposts for “when to hire more QA testers”
functionize.com
functionize.com
It's ok to have a QA 'team' that represents a group of QA specialists and focuses on their unique interests and areas of responsibility, but the members of that team should be embedded in the development teams, working alongside them as they develop, helping to write test plans, identify problem areas before the code is written, clarifying acceptance criteria, etc. If there's a "QA manager", that person should be more like a mentor, or the kind of good PM who runs interference and supports the development effort, but isn't actively directing the development effort.
If your QA people aren't in the room (covid aside) with the developers, and are instead all gathered together in a separate "QA unit", you've already lost. By the time they identify problems, it's too late to fix them well, and they'll never change the development culture into one that learns to produce fewer bugs in the first place.
Some projects have such sensitivity that a separate team which can do full regression testing along with proper testing of deliverable is required. Too many projects short what is actually tested and this can lead to stress on the support teams which in turn will feed back to the development teams
Avoiding a QA silo doesn't automatically fix the root issue. They'll do this with their managers or teammates too if things are sufficiently FUBAR.
As a developer, a good QA team is worth their weight to me in gold. Reach out proactively and engage them about new features you've built that you want 'em to hammer on - they'll be better at finding all the edge cases you didn't think about than you will. By engaging them early, you're more likely to get bug reports for code you still remember, and less likely to have them filing bugs right before launch at the 11th hour where your manager might plead for yet more crunch and overtime. If they're thoroughly testing my code for me, that's less time I need to spend carefully hammering my own code to avoid future blame. They'll help me hammer out good repro cases for strange bugs, and test on more varied hardware with different timings and usage patterns.
> they don't work for QA so QA has no direct authority
If devs aren't held to account for shipping broken shit, embedded QA won't necessairly fare much better.
> QA comes after the fact and it's too late to change things (or at least too late to change them the right way)
You can have quick turnarounds with QA teams. They definitely need to be able to get their hands on builds quick enough to provide devs useful feedback before shipping, though, and can't be siloed to the point of blocking direct communication. Even external QA teams employed by another company can succeed here though.
> QA lacks the engineering background to understand what it is that they're testing (so they test the superficial things and don't test the showstoppers);
Hire better QA and/or educate your existing QA better. They can chase superfical things and checklists, but if you can't explain enough of the basics for them to help chase down a crash or misbehavior you might be worried about, you've got a problem. Poor documentation, poor communication, poor guidance, poor understanding, too much bureaucracy... something fixable.
Perhaps the structure of game development has some tips here - we've often got multiple QA teams:
1. Internal studio-wide QA teams, which help chase down bugs internally so you don't have to - probably in the same building, at least. Rarely gatekeepers in and of themselves, but they do have the ear of your production staff.
2. External second party QA teams - possibly contractors, possibly employed by your publisher, maybe in another city entirely. Slower turnaround, but more vast and a fresh set of eyes not ruined by being too close to the project for too long. Can also include not-quite-"QA" specialty stuff like usability testing. Brought on later into the project.
3. External third party QA teams. Employed by the console vendors, not by you or your publisher. These guys can throw a wrench into your release schedule, and then charge your company for the privilige of having another go if you've failed the certification process a few too many times (or at least, they could at one point.)
Even if you have an adverserial relationship with the console vendor's QA teams, you'll buddy up to your second party and internal QA teams to find enough of the bugs to ship on time with minimal crunch if you have the slighest lick of sense, even if you haven't the slighest bit of shame at shipping broken things. They're the ones who will help you avoid failing certification, after all.
This is probably not universal, though - what sides of the fence of org design do people fall into, here?
Eventually QA ends up being owned by either the Engineering team or a standalone QA team. I've seen both work, and both not work. Generally if it's not working for Engineering, it means they're not automating enough or the tests are flaky and they're just ignoring it. If the QA team approach is not working, they're probably overworked... Likely because the engineering org has deeper quality issues they need to fix, but they're trying to make up for it by throwing bodies at it in the QA org.
* We have a no-code test automation product that is used by both QA and Engineers: https://reflect.run
In my view, the best approach is rigorous automation and manual exploratory testing which will find the kind of bugs automation misses and inspire new test cases.
I think automation is often a waste of time at the beginning of a project too. Bugs don't cost much (if anything). Building functionality that gets tossed away is routine. Tests don't provide value - getting the code in front of customers and seeing if it's what they really want provides value.
The problem is that once people get used to not having automated tests is can be difficult to build up the habit.
Unit tests are also often hard (if not impossible) to retrofit, although IMO that's a bigger problem with unit tests than it is doing testing at a later stage in the project.
I am not sure _why_ this is but I think it has something to do with the "Not my problem" mentality where engineers say "QA will figure out if this works or not" instead of "I'm responsible for figuring out of this works".
It hasn't become a red flag for me _yet_ but it's definitely _a_ flag.
That said, an understaffed QA team can be really helpful. Not enough people means you can't ship garbage and expect QA to catch it; they'll catch many things, but not everything. Not enough people also encourages efficiency in testing --- test the most important stuff well, test the other things opportunistically.
A QA team also can really help turn customer problem reports into actionable bug reports.
When I've been on teams without formal QA teams, the team was focusing way more on having good automated test practices in place.
This also applies from the SDET/SDE separation, which I see someone else in an adjacent thread mention. I've found a lot of value from taking ownership of both the code and the automated browser testing code. I found when those roles are separate, the browser tests end up breaking too often due to developers not being aware of how coupled they are to the current implementation. I also found when a dedicated team is writing browser tests, it becomes easy to go overboard with how much you test.
Maybe it's because we were embedded in the dev team.
Sorry you didn't experience the same thing.
I don’t think this is an anomaly. Every engineering team I’ve been on since has given this responsibility to the developers writing the code, and every one has been stronger for it.
I try to submit feedback and half the time the feedback tool itself fails to work, the other half of the time my bug is closed or ignored so that the team can 'focus on issues with a broad customer impact'.
Is there any evidence that Microsoft's product quality has improved recently? The only way I can imagine justifying that point of view is if you divide by the cost to get that quality, which is a useless metric to me as a customer.
Maybe I just experience bugs that no one cares about, but I no longer recognize Microsoft Office or Windows as solid software given the massive increase in the volume of bugs I experience.
My MS friends have the feeling that it was a painful transition, yet ultimately a good thing, and that quality has gone up.
My experience on the ground, so to speak, is the exact opposite. The May(!) update still has showstopping bugs with a not-small portion of my clients. Last year, one was bitten by the wipe-the-user-folder upgrade bug. The trust in running windows updates is at an all-time low amongst the general users, and I can't blame them.
I trust my friends. I also trust my own experiences. I wonder where the disconnect is?
I'm sure that's what the execs say at the all-hands meetings but ask the former MS SDETs who still use MS products and you'd hear a much different opinion. ;)
My standard quip when hitting yet another bug in a MS product, which is an all too frequent occurrence nowadays, is "Gosh, maybe MS should consider hiring some SDETs."
If you have good engineer-written and maintained tests, then you don't need QA.
The problem comes from relying on developers to do manual tests. It works as long as the product is simple and newcomers can learn the entire product. But, developers doing manual testing won't scale once the product becomes too complicated to remember all the corner cases and gotchas. Even worse, newcomers just won't understand the product as well as the people who originally wrote it.
So, if you have a complicated product, and a lot of debt in developer-written tests, then you need QA.
With regard to the web, it's very easy to write automated tests against server-generated HTML or an HTTP API. In contrast, other platforms are all over the place in how easy / hard it is for a developer to write an automated test.
Funny, I find the opposite. Code for the web is hard to write reliable tests for. Typically I want to test that my component actually exists and is visible and does the right thing in a browser, so I need to produce the whole page instead of just the unit under test. This means that bugs - or just changes - in other parts of the page can make my test fail.
I'm not a front end expert and my colleagues rarely have been either, so maybe I'm missing something about best practices here, but I've definitely had the opposite experience to what you describe.
1) They are expensive. You can hire 3 or 4 QA people for every dev a lot of times. Why waste that money on devs?
2) They aren't that great at. They know how the program works, so most struggle to do things outside of the "happy path". I had a QA (actually my PM) paste in an entire book into a text field and break the site that way. Devs don't think of stuff like that. Another uses the browser's back button constantly, which none of the devs seem to do.
That said, it doesn't excuse crappy code. If a code change/addition fails QA, that's on the Dev. However, if that failure hits Prod, that's on QA...
Developers aren't paid to test, the majority of their time is spent making new code because that's what management/customers want. Having one person have to wear 2+ hats doesn't work when they can never take one of the hats off.
If they were legitimately given the amount of time to test as they get to code, your project could take twice as long, but it would be more thoroughly tested. And, likely, it wouldn't take twice as long because they'd actually have to think about testability and design earlier. And by "test" I do not mean just unit tests. Designing and implementing integration tests, assembling regression test suites, etc.
Devs focused on just unit tests will never catch the errors that a dedicated V&V team can find.
In my experience, QA teams are usually starved of engineering resources, so a lot of things that can be automated are done manually or the automation is janky and constantly breaks. Once these issues are brought to the forefront, it becomes a priority to fix so that your expensive engineers aren't wasting time chasing bugs or fixing broken automation.
Attitudes like this will get you...not great QA. In my experience a sufficiently strong QA person can both cost and be worth more than adding another dev to the team.
I think we have a good code-quality/speed-to-release ratio with well known tech debt that is actively being cleaned out, that is also tested after refactored out.
Having QA Tester be part of the team makes it really easy to adjust features, adjust the release if an update is needed and it is not passing quite yet quality wise. It also makes it really frictionless specifying what to expect from a ticket/feature and how to baseline test it, the tester also performs some tests of his/her own, the QA tester runs unit, automated UI and performs manual regression tests.
I've been part of teams that have QA testers outsourced as a dedicated independent team and it definitively impacted code quality and the release turnaround, mostly because of delayed communication and mis-alignment on what to test or the expectations of the test.
The process we used was the three amigos process, where the ticket dev, BA and a QA would have a 20-30 minute sit down and draw up a mind map of test cases linked to the acceptance criteria.
Once the mind map was created, anyone technical could pick up the actual writing of the tests/exploratory testing, whether that's the original dev, a QA, or any other dev. Because devs were all testing each other's tickets instead of just hoofing them over the fence to QA I found that I got a much better understanding of the different areas of the projects.
Only downside was it required a lot of meetings to really flesh out the tickets and make the acceptance criteria as detailed as possible.
Quality Assurance is (by analogy) design and code reviews, coding standards, and design for success.
Quality Control is testing -- just making sure things are as intended.
Whom do they expect to test the software? The developers themselves?
- A legit comment I have heard
If the requirement can be stated formally, in an ideal world, this should not be the case. You can write automated tests and verify systems with a level of precision that humans cannot rival and it's not expensive to do so. If your QA teams are catching errors of this variety it's a symptom of another problem: communication issues. Your QA people shouldn't be catching errors of this variety -- it's way late in the development process to realize that your developers didn't understand the requirements or didn't write tests to specify them.
There are a lot of reasons why your QA team is doing this kind of work. Common causes are:
1. The development team is pressured to prioritize budgets and deadlines
2. The requirements are not communicated well, never gathered to begin with, or no single person knows what exactly is being built or why
In the first case developers might not be given the time and space to scrutinize the requirements, write tests specifying the appropriate behaviors, and making the effort to build a system to that specification. Your QA department is basically picking up the slack: the development team pushes what they think meets the requirements and waits for the QA team to give them the thumbs up or down.
The second is also pernicious and makes the former worse.
In an ideal world the QA team should be saving their time and energy for informally-specified requirements and difficult to qualify goals such as, "It must look good in IE 11." There's no way to formally specify that requirement so it must be checked by a human who does understand what that means.
I'd caution that if your QA team is growing with your development team then you should treat it as a symptom of a larger problem. In the real world QA teams end up doing a bit of both and development teams don't always have clear requirements. Aiming for the ideal goes a long way to keeping stress down.
Probably, but I submit that nobody can do that perfectly. By all means do as much automated testing as you can! But you still need humans to sanity-check it and introduce new behaviors.
I'm a manual tester who's worked on many different projects. The larger ones have had a separate automation QA team.
On the smaller projects, I'll fetch the code base and suggest missing unit testcases to the developers.
Without writing a wall of text, I feel that dismissing the value of manual QA is like dismissing the value of a business analyst or project manager who doesn't write code.
My value comes from helping make sure the customer got what they wanted and it works - and I can do that without writing much code. But that's not to say I don't use tools and scripts to automate some of my work.
That said I don't know if that situation is unique or the norm out in industry.
I would always hire a little more QA resource than you think you need. Nothing worse than signing off new features but not having the time to think outside the box and find overlooked issues.
Also there are so many static analysis tools. FXCop etc. Been at so many companies where there are thousands of warnings in the code, many of which were genuine bugs. At one, we fixed 16,000 warnings, and managed to close off a few dozen sev 1 crashes that people had been experiencing for ages.
Ensure that you fix the bugs and write some automated test for this case if possible. Again free.
Pay someone to do a little QA of your product. I'm sure it'll hurt to give away money that you could pocket yourself... but it's the price of getting work done.