Hacker News Reader

> That's not an A/B test - you already knew a bigger button will result in more click-throughs, you don't need a test to show you that.

You'd be amazed how often "intuitive" things like that often don't actually work.

One product we sell is a specialist piece of software for managing "projects". Let me paint a picture of the interface: you have a list of "projects" which you select then hit a button at the bottom of the page to open the case. Within the project screen there is another button to go to a report page.

As this is about the most common page people use so I trialled installing an "open reports" button directly next to the "open" case button. Hardly anyone clicked on it in our test period (even clients who got an update notification telling them about it) - we actually got one piece of feedback saying it confused them.

I can list 10's of similar examples with our products where a) "intuitive" improvements turned out to be useless and b) split tests caught useless changes quickly.

Users are very fickle :) split tests can be useful.

(sure, not an exact science though).

While its not science - it is a perfectly valid application of scientific method on a small scale. He's got a variable he measures - the money he has coming in, and a single variable he changes - whether he wears a suit or a tshirt.

To make it more reliable he should have repeated it several times.

If he had done that then I don't see why this is an invalid way to prove his hyothesis.

Yeah, I mean, it's like it's more scientific to bring all of those people into a clean room where the weather is constant and then ask them how much they want to buy lemonade -- that's even further away from proving what he wants to prove.

Also completely unscientific:

If I saw a kid out in front of his grandmother's house in the summer wearing a suit selling lemon-aid, I'd purchase it and probably add a tip:

- It's the middle of summer[1] and the kid is serious enough that he's willing to sweat it out in a suit. That's an attitude I'd respect.

That would be my instant, gut reaction, walking by that stand for the first time.

[1] I'm assuming he's not selling lemon-aid while attending school

When you say: "A/B tests will tell you nothing you did not already know, they will just PROVE it if you need to convince someone else." you are wrong.

Hypothesis testing is a way to distinguish, with a given level of confidence, those events that happen due to chance from those that do not. It can very easily tell you something you did not know, because you mistook noise for signal for example, or because you failed to see a pattern that is obscured by noise. Furthermore hypothesis testing does not need a very large user base. You can get significant results with tens of samples, though obviously the more users you have the stronger the power of your tests.