Hacker News Reader

Comment by alex_smart | Hacker News Reader

But testing by human is expensive

and horribly unreliable even when done by competent and motivated humans, let alone most IT workers

I've heard of good experiences using a 3rd party QA company (for frontend-heavy changes) and had pretty OK experiences doing it in-house with subject experts (though testing backend changes, so pretty much "either it works or it doesn't" in my case).

A good QA person/team will (be strongly incentivized to) make it their mission to find bugs and usability issues. Likely they will be domain experts, if only in the domain of the usage of your product. Worth their weight in gold, no machine can give you that.

Expensive isn’t unnecessary.

Developer written tests can’t tell you if your UI is intuitive for novice users.

They can however tell you whether the button/form/page/whatever is working and continues working.

Do those developer created tests include weird and random text that users will unwittingly break your form with? More than once has a form been broken because devs didn’t test for emoji or other glyphs in the text. Just because a button click calls a form with as expected doesn’t not mean the form will behave as expected with unexpected user input

Playwright is essentially a programmable headless browser. It sends a request to a page, waits for the response to load and then for things like React rendering, secondary requests etc, then once the page is ready you can do whatever you want. You can interact with the DOM, fill in forms, scroll, click things, get a div and check its size and location, anything you want. Testing special symbols and such should be no problem at all, provided that you actually think to write that test.

Which is can be used for a range of things but doesn’t remove the need for independent testing.

To use one tiny concrete example. Someone who isn’t aware of U+0022 + U+201C quotation marks will happily use U+0022 everywhere without ever considering the very slightly more aesthetically pleasing option. Independent validation is about redundancy, ideally you don’t want anything to reach that stage but people are really bad at testing for issues the’ve never considered. Usability in terms of colorblindness not just actual blindness yadda yadda good testing is really involved.

There is fuzzing, FWIW.

But the same also holds for people who are "independent testers" and unaware of the same issues.

What I found is that software developers are bad at testing edge cases while they are in the creation mode (when they focus on happy path), but that good engineers switch to breaking mode once they try to cover things with sufficient tests. TDD also encourages breaking things first, but really, this is a mindset change that is usually skipped.

Fuzzing isn’t going to find either of the issues I mentioned. It’s useful for finding the kind of issues most developers I have worked with are well aware of, though may not specifically test for.

why do you think that fuzzing won’t try inputting the alternate question mark in your input?

For what it’s worth, if fuzzing is unlikely to, then I think so is manual testing unless you get really lucky or an expert is trying to break your system because they know of frequent issues with Unicode handling.

When testing with a human: you catch it in QA runs late in the cycle that test case once it’s added to the test coverage as a known issue

When testing automated: you prevent the commit in main sequence nice the regression test is added to the test coverage.

This is the point - adding tests to your automation suite gets cheaper over time and guarantees higher quality than manual QA runs with not much more cost to the business. Not doing so is choosing to take on tech debt. But pretending like random text is something your QA will catch on their own is a pipe dream. You’re better off investing in property tests and fuzzing.

Perhaps cheaper than upsetting your customers when they are the ones doing the testing?

No machine testing can completely (or even mostly) validate a human interface.

Reply on news.ycombinator.com