Hacker News Reader

Hacker News

Comment by somewhereoutth | Hacker News Reader

Full thread

somewhereoutth

My feeling on testing:

- If it is used by a machine, then it can be tested by a machine.

- If it is used by a human, then it must be tested by a human.

View on HN

User acceptance testing is a good idea but doing it in cases where you can get away with cheaper testing is not.

Tools like playwright allow pretty nice web UI testing. You can make sure things are working properly and continue working as things change.

Doesn't replace human testing but it does ease the human load and help catch problems and regressions before they get to the human testers.

> Tools like playwright allow pretty nice web UI testing.

Can they test for color blindness and myopia?

“Doesn't replace human testing”

It’s like you stopped reading to try to score internet points or something. The answer to your question was one more sentence from where you stopped reading

The point of the question was to both see if the bar has changed and to address the overall article's point "Can we test it".

There are tools that can analyze a page for accessibility. I don't have experience with them but I wouldn't be surprised if you could include them in a CI/CD pipeline. I don't know whether they specifically control for color blindness and myopia but I wouldn't be surprised if they do. I know they check for contrast and stuff like that.

You could also check things like colors etc using Playwright but I would say it's probably the wrong tool for that job. It's more about testing functionality - make sure a page has the right content and works correctly on a technical level.

Without automated tools this type of thing can take a lot of time - in order to ensure quality you would basically have to click through the entire application for every release. Otherwise you might end up with some minor changes to one page breaking a different page and you'd never know until a tester checked it out or a user complained. With Playwright and similar tools you can catch regressions like this automatically.

But testing by human is expensive

and horribly unreliable even when done by competent and motivated humans, let alone most IT workers

I've heard of good experiences using a 3rd party QA company (for frontend-heavy changes) and had pretty OK experiences doing it in-house with subject experts (though testing backend changes, so pretty much "either it works or it doesn't" in my case).

A good QA person/team will (be strongly incentivized to) make it their mission to find bugs and usability issues. Likely they will be domain experts, if only in the domain of the usage of your product. Worth their weight in gold, no machine can give you that.

Expensive isn’t unnecessary.

Developer written tests can’t tell you if your UI is intuitive for novice users.

They can however tell you whether the button/form/page/whatever is working and continues working.

Do those developer created tests include weird and random text that users will unwittingly break your form with? More than once has a form been broken because devs didn’t test for emoji or other glyphs in the text. Just because a button click calls a form with as expected doesn’t not mean the form will behave as expected with unexpected user input

Playwright is essentially a programmable headless browser. It sends a request to a page, waits for the response to load and then for things like React rendering, secondary requests etc, then once the page is ready you can do whatever you want. You can interact with the DOM, fill in forms, scroll, click things, get a div and check its size and location, anything you want. Testing special symbols and such should be no problem at all, provided that you actually think to write that test.

Which is can be used for a range of things but doesn’t remove the need for independent testing.

To use one tiny concrete example. Someone who isn’t aware of U+0022 + U+201C quotation marks will happily use U+0022 everywhere without ever considering the very slightly more aesthetically pleasing option. Independent validation is about redundancy, ideally you don’t want anything to reach that stage but people are really bad at testing for issues the’ve never considered. Usability in terms of colorblindness not just actual blindness yadda yadda good testing is really involved.

There is fuzzing, FWIW.

But the same also holds for people who are "independent testers" and unaware of the same issues.

What I found is that software developers are bad at testing edge cases while they are in the creation mode (when they focus on happy path), but that good engineers switch to breaking mode once they try to cover things with sufficient tests. TDD also encourages breaking things first, but really, this is a mindset change that is usually skipped.

Fuzzing isn’t going to find either of the issues I mentioned. It’s useful for finding the kind of issues most developers I have worked with are well aware of, though may not specifically test for.

why do you think that fuzzing won’t try inputting the alternate question mark in your input?

For what it’s worth, if fuzzing is unlikely to, then I think so is manual testing unless you get really lucky or an expert is trying to break your system because they know of frequent issues with Unicode handling.

When testing with a human: you catch it in QA runs late in the cycle that test case once it’s added to the test coverage as a known issue

When testing automated: you prevent the commit in main sequence nice the regression test is added to the test coverage.

This is the point - adding tests to your automation suite gets cheaper over time and guarantees higher quality than manual QA runs with not much more cost to the business. Not doing so is choosing to take on tech debt. But pretending like random text is something your QA will catch on their own is a pipe dream. You’re better off investing in property tests and fuzzing.

Perhaps cheaper than upsetting your customers when they are the ones doing the testing?

No machine testing can completely (or even mostly) validate a human interface.

Reply on news.ycombinator.com