Hacker News Reader

Comment by ef4 | Hacker News Reader

Surprised to see the negativity here. I have worked in environments with traditional manual QA, and environments where all development is test-driven and nobody is allowed to merge a feature that lacks automated test coverage.

Both the productivity and the quality were higher in the places with fully automated testing. Which is not shocking at all: does anybody really think a human can run through 800 test cases better than a computer can?

It's not a magic way to save money -- the developers obviously end up spending time writing tests. But the long-term value of those tests is cumulative, whereas the effort spent on manual testing is spent anew every release.

Manual review is still good for noticing things that "feel wrong" or for helping think up new corner cases. But those bleed into product owner & design concerns, and aren't really a separate function.

View on HN

Moving from an environment with a 10:1 dev:qa to 2:1 showed me what happens when dev is not responsible for shipping working software.

No thanks. It's a bunch of deflection and diffusion of responsibility coupled with high latency flakey interactions between different teams. Everything that can slip through the cracks does slip through the cracks.

I'm sure QA can be done well, but I am convinced that giving your devs a pass to not finish their work is a dead end in several dimensions.

The best way I've seen it done is when another developer, on the same team, who is responsible for the same product, reviews your code. Not someone who writes tests for a living, someone who does the exact same job as you on the same product.

Any time I've experienced something different it's been exactly as you described.

I agree with you that that is a necessary part of the process and a great first step for small teams moving away from "just release it and see if it works". The unfortunate reality though is that people in the same role on the same team often have the same misconceptions and blind spots.

Now getting devs to see the product through the users' eyes goes a long way toward solving that, but if you have a process and team of devs that are doing that you're way ahead of the game in a lot of ways.

I'm in full agreement.

In practise, a manual QA team encourages Devs to throw shit over the wall and expect someone else to do some basic sanity checks they should have already done. By the time those are done, what the QA team theoretically could find gets shipped.

Then, when its discovered in Prod, the QA team will get in the way of a speedy fix.

That's a management problem, not a problem with having a QA team. The top-level QA manager and the top-level Dev manager should both report to the CTO, and the Dev manager should be judged by how many issues the QA team finds. To keep things fair, the QA manager should not be rewarded nor penalized based on the number of issues found pre-release, and everyone should be rewarded or penalized based on the number of issues found post-release.

This keeps the devs incentivized to make sure everything works before the code goes to QA, and it keeps everyone incentivized to eliminate as many bugs as possible before release.

Have you ever worked for a fortune 100 company? The dev manager has probably never met the CTO. Im in this right now with my currrent job. I still keep my builds and test them longer than i should. Less broken shit gets through, but i regularly delay my qa ba's because of it.

So, not the CTO, but some shared higher-level manager. The point is that the QA lead cannot report into Development, and Development cannot report into QA. Both of those arrangements can lead to a bad actor stopping the other group from doing the right thing. (Eg: if QA reports to Dev, then Dev manager can force bad code to ship, and if Dev reports to QA than QA manager can prevent shipping and force Dev to waste time on excessive/pointless test code development that's not cost-justified.)

I've met people that call something similar 'fake agile'.

Surely everyone should report into the business (end-user on the business side) lead responsible for making this functionality happen. This includes Dev, QA, business. QA should be business? This is no longer 1985, there is no reason business (product owner) shouldn't be a capable PM and BA for their own product.

In the company I used to work for, the CTO and VP of Product Management were equals, both reporting to the CEO. The CTO's reports included the QA director, Dev director, and Project Management director.

On any given project, we had a Product Manager who set the business requirements, a Project Manager who ran the project, a Dev lead in charge of the development team, and a QA lead in charge of QA. None of these people could override the others because of the reporting arrangement, so we had to collaborate instead to make sure we all met our goals.

Early on this arrangement worked really well. But it started falling apart when ownership of the company changed and politics became a factor as people who cared more about moving up through the organization than about the success of the company and quality of the products were hired and promoted. My take is that bad management will destroy any approach you might take to producing good software. That's why I eventually left.

I work for a FTSE 100 company and the dev manager and CTO in the division where I work are very well connected. I don't think the size of company is necessarily the only factor here.

> The top-level QA manager and the top-level Dev manager should both report to the CTO

For an organization with a single product (and, particularly, one small enough that the "top-level" QA and Dev manager are also first level managers), this makes some sense. Otherwise, this means that each product has no common level of technical management below the CTO, meaning it has no meaningful technical ownership.

It doesn't have to be the CTO; as I mentioned in another reply my intent was that QA and Dev need to be equals in the reporting chain, rather than one reporting to the other. That can work at any scale; even on an agile team with one QA and one Dev, they both report to the same boss who's responsible for both sides.

Of you are fine tuning the reporting chain, your org is already hopelessly broken. That is the central planning Communist model.

What will happen is that devs will ask their QA counterparts to report issues through an undocumented side-channel.

If you give an engineer a career incentive to optimize something, you'd be surprised how seriously some will take it.

Incentives often cause really nasty political wars within orgs.

When a site gives a 500 because a database went down and the web app couldn't connect to it... is that a bug, a missed test case or should ops take a hit on down time? Furthermore, if you argue that the dev team should have reasonable failsafes in code to connect to a db, in a 10 year old organization, should the current Dev team pay for something that could have been in the code for years?

If you set up a system where devs hand off to QA then they hand off to ops to deploy. Furthermore every step is incentivised somewhat against each other. Even if all teams are equal, it ends up in my opinion, to the path to CYA and Waterfall. Everyone is more concerned with problems not being 'their' fault than shipping good code.

You're absolutely right, assuming we're talking about business/consumer apps and not pacemakers or rocket ships.

An incentive/penalty system for bugs without an incentive/penalty system for features completed leads to paralysis. And a complicated incentive system leads to game playing over productivity.

It's an old saying: be careful what you measure because you'll get a lot more of it.

I'm beginning to think that the best incentive system is just a good base pay, with an emphasis on personal learning and growth. Have engineers strive to become better engineers by taking courses, attending meetings and publishing content. Quality will fall out of that as a positive side-effect.

The solution to this is to require a bug/ticket/tracking item as part of the change. Since QA knows that dev needs to create one anyway, they might as well get the credit.

Where this starts to break apart is that QA can get overzealous and file bugs for different incarnations of the same bug.

It's a bit of give and take.

The top-level QA manager and the top-level Dev manager should both report to the CTO

Having a top-level manager who is responsible for a whole discipline doesn't work, it just creates a high wall to throw shit over. Developers and QA testers really should report to the same first line manager.

That sounds like a terrible strategy for a lot of us, where time to fix is much more important than bug count.

This is because there is likely to be a tradeoff between speedy development and bug count.

If you are shipping physical CDs of software, or releases which you can't update easily, or if you are working in a critical environment where mistakes are disastrous (finance, health, space, etc.), then it is fair enough to be so concerned about bugs.

But many if not most developers work on the web in areas where the product is uncertain and evolving. Prioritising rate of change is more important here. Rather than extensive QA, monitoring and fast rollback are better choices.

A formal QA process, which goes along with a formal release process and release schedule, certainly does slow things down. I think for large-scale projects with a large feature/requirement count that's ok, because there's a lot of little details to keep track of and the formal process helps with that. But you also need a fast-track for fixing small things quickly, and for that Dev, QA, and Ops need to work together to quickly diagnose, fix, test, and release changes.

My experience with the formal process I've been talking about was in a company whose product is a large-scale web application that interacted with a complex custom back end search engine and a proprietary content database. The QA and Dev teams had a friendly adversarial relationship, but ultimately we all felt we were on the same team with the same goals: to produce the best software we could. It was a fantastic place to work.

My experience is the opposite. Having independent testers means you're going to be stuck addressing 100 defects if you don't watch what you're doing the first time around.

10:1? You do realize QA has to have a semblance of life too? Adding more QA isn't designed to screw up the process, but to alleviate the workload. If you want to blame anything or anyone, I'd usually shoot for 1) hiring incompetent QA that isn't SDETs but merely just black-box QA competency, and 2) process - or lack of process, i.e. an agile process of building a little, testing a little, and delivering consistently a working product on pretty much any basis - monthly, weekly even.... That's the ideal of course.

If this is in place, adding extra QA can be beneficial.

None of this of course is excuse for a dev not producing code that works at least in its happy path plus/minus a few of the most obvious exceptions/error paths...

How is it that in the 2:1 environment developers aren't held accountable when problems turn up in code they're responsible for?

> Which is not shocking at all: does anybody really think a human can run through 800 test cases better than a computer can?

I think this rather misses the point; it's the bug that doesn't have a test case where QA helps.

Exactly. And having the computer run through 800 checks means a skilled tester can do what they are good at - finding bugs.

This to me was always the point of automating tests. One of the best QA people I knew refused to look at the user stories. He was brilliant at finding things that fit the spec but didn't make sense and things that users might do that weren't specified.

He found bugs in the specs, gaps in the specs and just plain untested behaviour.

QA is a job that requires skill, benefits greatly from technical understanding and requires a lot of domain knowledge.

Yup, that's pretty much my job. The devs do great testing and I try and pick holes in what they miss.

This is why specification should happen collaboratively with everyone with a stake (and that includes QA just as much as BA), you can fix them before they're developed against.

This is what a great QA tester will do. Good devs don't ship code with known bugs - but they will miss things because they code to the feature/user story/known behavior. That's what their job is. A good QA person's job will be to spend a whole lot of time thinking about how to break the dev's app that the dev missed.

> does anybody really think a human can run through 800 test cases better than a computer can?

Of course not. But automated tests are just the entry ticket to the release process.

A computer can't replicate the irrationality, laziness and guile of a human end-user. That's what good QA engineers test.

It really depends on what you're making and if it's possible to automate testing AND if you remembered to test it. Chrome appeared to have broken full screen movie playback for 5% of users once. Something changed, perf got bad, people who previously could play full screen video on a low end machine suddenly got playback too slow to be useful. They just stopped going full screen. No bug reports came in. Nothing in the testing infrastructure caught it. I only notice because my father tried to show me a video on his atom based netbook.

Similarly going full screen on a 2 monitor setup broke once. Again, no automated test.

The Web Text to Speech API is broken on every browser it's in. It will work with a simple sample but start and stop it a few times and it will break. I suspect because again there is no easy way to test it.

There's lots of others.

automated testing is not a substitute for QA IMO.

Even more important is that the developers spend time writing tests and their schedules are not expanded to accomodate this, nor is their pay increased. They are simply expected to cut corners elsewhere, or work more hours, or whatever is necessary. That's the real savings.

It sounds like there is some opposition between automatic tests and QA. But those do not have to be opposed - one can have both. In fact, for many types of software I don't see how you can avoid having both. The only choice you have who is doing your QA - your employees or your end users. Maybe Yahoo can afford the latter. But for some companies it would be a disaster.

The arguments here such as "oh they just make the devs do it and don't pay them anymore" are ridiculous. Nobody's working 80 hour weeks doing two full-time jobs at once.

Please let me tip a name for that: "Volkswagening" - assigning the quality check to the production team.

Haha, I'm going to start using that term. That's a great description.

To be fair the exact meaning of Volkswagening is to fake the behavior during official tests. Example: http://qz.com/515100/samsung-is-accused-of-volkswagening-its... Note that volkswagening.com was registered 20 days after the scandal, but is available for bid, if you want to do something fun with it.

VW lawyers would probably throw a fit if somebody actually used that domain for anything commercial.

Perhaps it could display different, positive content when accessed from Volkswagen IP addresses, to ensure they wouldn't find out.

It doesn't even have to be intentional. If you write the code, you always have in mind some picture of how it should be used. That picture means you ignore the ways to use it which you not intended. So when the code is used in that way, it breaks. But you won't ever write a test for it because you would never think such usage is possible - exactly because you know too much about the right way of how it should be, so you start to think it's how it is.

This.

Devs don't think like users. They know a lot about computers and can make good guesses about how other devs think. So devs are not the best people to test code that's going to be put in front of non-dev users.

Users are likely to have different models, and a different set of expectations. You can't write tests for all those possibilities because you literally have no idea what they are - and won't find out until you put them in front of users.

I still firmly believe this is a large part of the reason for having devs carry pagers.

Maybe in better companies than I worked it that isn't the case, but devs shouldn't be your 24/7 support staff.

Pagers? Now it's just a total violation of work life balance through Slack.

My phone has never been so loud.

Isn't pager duty baked into the salary of the position? I thought that jobs where you're expected to be on call to some extent (hopefully at level 3 or 4 of support?) have an artificially high salary compared to those where it's a 9-5 gig?

Did the QA you dealt with not have any automated testing tools? That seems rather foolish to have your QA team manually test every feature every time without an automated smoke test.

QA is essential. How else do you know that you have built what you set out to built? Traditional manual QA is highly effective, so much so, that you can get huge gains by automating it. QA occurs at many levels and it makes sense to have a dedicated QA team for each level. Generally, the QA team for a level should be the exact people who requested and/or approved the specific feature subject to QA, as they are in the best position to confirm or deny that the feature meets the specification, as they created and/or approved it.

But you're talking about QA as a general process, not QA as is meant in the article:

> Software engineers at Yahoo are no longer permitted to hand off their completed code to another team for cross checking.

Dev Team A was giving a batch of code to Dev Team B to review.

I agree that the business user/product owner/whatever should be reviewing everything in test prior to approval and in production after but I'm not sure that's the article means by QA.

I think it's important to recognize that neither option is perfect by itself. Manual QA can still be an important process.

That being said, if you have to pick one or the other then you go with automated tests.

Reply on news.ycombinator.com