Comment by jwmerrill | Hacker News Reader

Comment by jwmerrill | Hacker News Reader

Parent Full thread

Yes, we should all test our software better. But let's look at the cost/benefit here for a second.

Back in the days when "system" was a reasonable font choice, fixing a bug after shipping would take weeks, months, or years, and could cost $10^4 to $10^8. You might have had to pull actual boxes off actual shelves somewhere!

In this story as it actually happened, the bug was fixed and the fix was deployed in a single day, at the cost of a few laughs.

I'm sure they tested on Windows when they added Segoe to the list, so the problem is that they didn't test again after fiddling with other fonts that they (incorrectly) thought would be irrelevant to Windows. And remember, nothing was broken in a way that a machine would notice, so automated acceptance tests wouldn't have caught this regression.

I think it's worth considering the possibility that this is a story of Medium acting more or less in their economic interest (maybe even in the economic interest of their users?) rather than a story of a bunch of hipsters with a hilariously broken QC policy.

> And remember, nothing was broken in a way that a machine would notice, so automated acceptance tests wouldn't have caught this regression.

I'd like to nit here that visual regression test harnesses for web application development do exist. [1] [2] [3] Using something like Sauce Labs you could conceptually have these tests run on Windows machines in the cloud.

However, having implemented visual regression testing in the past in various projects, I can personally vouch that it's a huge waste of time. Your tools will break all the time, your tests will break all the time, and you'll generally have a bad time.

[1] https://github.com/chenglou/node-huxley [2] https://github.com/Huddle/PhantomCSS [3] https://github.com/bradgignac/janus

Thanks for this perspective. I've never tried this kind of automated visual regression testing except in very specialized circumstances, but I've sometimes thought that I should.

In analogy with scientific modeling, there's a risk of over-fitting your tests against the exact circumstances of your product as it exists today. You want tests to capture the important aspects of your product, while having a little bit of flexibility for the "noise" related to differences between platforms, and changes to mostly but not entirely unrelated parts of the product.

The first time I did visual regression testing, I generated a bunch of screenshots on my Mac, chucked them into version control, and then ran the tests on CI. They broke because of the font rendering differences between OS X and Ubuntu.

The tool I used, PhantomCSS, did allow for a threshold underneath which pixel differences would not get picked up, but I opted instead to run my tests locally in vagrant so that the environment would be closer to identical to CI.

So yes, platform noise is just one of the myriad of challenges you have to solve when implementing such tooling.

Reply on news.ycombinator.com