Hacker News Reader

Comment by kevsim | Hacker News Reader

Anyone have experience with successfully doing UI tests (especially screenshot tests) in a very young software product? We try to keep our "plumbing" code and backend code well tested, but the UI is changing all of the time and I'm quite worried it'd be a nightmare to maintain UI tests at this stage.

View on HN

(Selenium project founder here.) Test maintenance cost is a sane thing to be worried about! Early on, there can be lots of code thrashing and UI changes. Better to keep the UI tests to a minimum at that stage - have them act more like smoke alarms. You only need a few alarms in the right places to tell you that something is wrong. Other general advice is that in a young product, it's often more important to make sure you're building the right thing. ("Right" is subjective, but could be defined as "has users" or "makes money".) Having a well-tested product that has no users and/or makes no money is an even bigger nightmare than UI test maintenance. Once you know you've built the "right" thing (hopefully sooner than later), you can afford to invest more in all kinds of test automation (not just UI tests).

It's not quite the same as screenshot testing, but I've had success using snapshot testing in React. The idea is outlined here: https://jestjs.io/docs/en/snapshot-testing

Snapshots of the intermediate representation (e.g. JSX in React) will be less brittle than e.g. a screenshot of pixels. It also works best when your UI is a function of some easily-constructed state. Other frameworks sharing these features would also work well with snapshot testing.

In the same vein, I'm a fan of Storybook [1], which uses Jest under the hood [2], paired with Chromatic [3].

Chromatic generates screenshots from component stories for visual change detection with an easy to use and share web UI. It's neat.

[1] https://storybook.js.org [2] https://storybook.js.org/docs/testing/structural-testing/ [3] https://www.chromatic.com

I use webdriverio, which has pretty straightforward screenshot plugins. Honestly early on it's better to focus on API testing. My advice is start small for the UI and focus on a few critical components like page routing or features that would be a show stopper if they broke. I don't think screen shot tests will be of great value if your product is not very mature. When I first started I made the mistake of blindly following my predecessors user stories that weren't particularly well thought out.

> screenshot tests

> the UI is changing all of the time

Far from an expert in UI tests, but I’m not sure the former is possible given the latter.

> it'd be a nightmare to maintain UI tests at this stage.

IMO that is likely.

You can use something like Quixote. (Disclaimer: I wrote it.) Screenshot tests are always going to be brittle.

Reply on news.ycombinator.com