Hacker News Reader

Comment by ergothus | Hacker News Reader

Perhaps I'm more forgiving, but it seems easy for someone just thinking "It seems more clear to me, but how do I know for sure? Oh! I'll put them up on a road and see if the accident rates change!".

Only when someone points out to me that the signs they are replacing are probably not new would it become "obvious". (In fact, while I read my first article on this, my internal thought was 'so why isn't this better if it tested so well?', then they mentioned the old signs, and my thought was 'oh, right, that makes sense'.

So embarrassing mistake? No doubt. Unbelievably embarrassing mistake? No way, not with the sadly believable mistakes I've made in the past.

View on HN

Disagree. Very embarrassing mistake. Also very embarrassing that this wasn't caught by whoever signed off on replacing signs (but not surprising, I doubt those studies are read in detail often and people probably only browse the intro and conclusions).

Basically a "you failed science 101" level of embarrassment mistake in my estimation. Testing two brand new signs with the different fonts should be a fairly obvious first idea. The hard/interesting part is taking into account learning effects for the previous font. There's a pretty interesting research question in how you'd age signs in there as well (as you want to know if the gains are positive over the entire lifetime of the sign) and some interesting ethics questions about field testing.

Well presumably it's not your job to do that sort of thing, but it becomes a bit more embarrassing when it is.

> It seems more clear to me, but how do I know for sure? Oh! I'll put them up on a road and see if the accident rates change!".

Meh. The American way would be "Build a street for font, and see which one people are willing to pay more for"

But wouldn't you do A/B tests in a driver lab first before going out into the wild with the fonts?

Yes, but that could still generate in a positive result for the new font that wouldn't be seen in the real world over a long period of time. The simple fact that the new font is new could result in an improvement that wouldn't necessary last in the real world. Hence why the font is being retired after a decade of experience and data in a proper situation.

A/B testing is notoriously hard to do well, especially if you're comparing something the user already knows well with something different. Controlling for familiarity is pretty much impossible.

Reply on news.ycombinator.com