Hacker News Reader

Comment by mattlondon | Hacker News Reader

How do you handle "meaningless" visual changes? E.g. chrome version change introduces super-subtle blending changes to drop-shadows, and suddenly there are huge diffs because the least significant bit of a drop-shadow's grey colour changed. What is your approach to that?

So far all tools I have used like this suffer from "false positives" like this (it's a true positive technically of course, but likely false from an engineering perspective).

View on HN

We don’t at the moment differentiate, but it is definitely a feature on the roadmap. There are multiple ways to solve this and we are considering quite some of them:

1. Grayscaling browser before screenshot 2. Using custom diffing engines that enable for ignoring regions like odiff 3. Trying to be smart and analyzing images with machine learning to identify how meaningful the difference is (moonshot but worth exploring)

Our approach to this in general is that if false positive like this happens we can just batch approve all the differences and be ok with that as it was a subtle rendering change not dependent on us or our direct third party dependencies.

You could run the suite on the updated browser on the latest passed build, then automatically batch approve since you know it has to be green. Now you can run all the newest test suites in this updated browser against this approved build.

exactly this is the current workflow for many people who are using visual regression tests!

Did you look at Applitools? https://applitools.com/

Their system has a way to demarcate volatile sections.

Reply on news.ycombinator.com