Lemonade Stand Operator A/B Tests His Attire - Suits FTW | Hacker News Reader

Lemonade Stand Operator A/B Tests His Attire - Suits FTW

kens5.com

45 pointsmhb11 comments

Get over this A/B nonsense. That is not an A/B test. An A/B test is not some path to riches or some rigorous way to prove something scientifically - it's an unimportant way to prove a hypothesis when you have a very large user number.

It's the hypothesis or the idea that makes the difference, not the A/B test! A/B tests will tell you nothing you did not already know, they will just PROVE it if you need to convince someone else.

A/B tests are really the ultimate in pseudo-science. Yes, they make sense for some narrow use-cases (like googles frontpage), but making a button bigger? That's not an A/B test - you already knew a bigger button will result in more click-throughs, you don't need a test to show you that.

And this article with some dude testing out another outfit? You want to call that an A/B test? It's not - it's just some guy trying out another outfit to see if it works. How is this a test? There is nothing rigorous or scientific about this.

If you NEED to do a test, then do an actual test, and factor in everything. But coming up with a hypothesis based off experience or something you read on some blog is not an A/B test. It's just an improvement to your site.

Stop pretending that this is science. It's not. It's intuition, and calling it an A/B test will not make the process any more 'scientific'.

When you say: "A/B tests will tell you nothing you did not already know, they will just PROVE it if you need to convince someone else." you are wrong.

Hypothesis testing is a way to distinguish, with a given level of confidence, those events that happen due to chance from those that do not. It can very easily tell you something you did not know, because you mistook noise for signal for example, or because you failed to see a pattern that is obscured by noise. Furthermore hypothesis testing does not need a very large user base. You can get significant results with tens of samples, though obviously the more users you have the stronger the power of your tests.

While its not science - it is a perfectly valid application of scientific method on a small scale. He's got a variable he measures - the money he has coming in, and a single variable he changes - whether he wears a suit or a tshirt.

To make it more reliable he should have repeated it several times.

If he had done that then I don't see why this is an invalid way to prove his hyothesis.

Yeah, I mean, it's like it's more scientific to bring all of those people into a clean room where the weather is constant and then ask them how much they want to buy lemonade -- that's even further away from proving what he wants to prove.

Also completely unscientific:

If I saw a kid out in front of his grandmother's house in the summer wearing a suit selling lemon-aid, I'd purchase it and probably add a tip:

- It's the middle of summer[1] and the kid is serious enough that he's willing to sweat it out in a suit. That's an attitude I'd respect.

That would be my instant, gut reaction, walking by that stand for the first time.

[1] I'm assuming he's not selling lemon-aid while attending school

> That's not an A/B test - you already knew a bigger button will result in more click-throughs, you don't need a test to show you that.

You'd be amazed how often "intuitive" things like that often don't actually work.

One product we sell is a specialist piece of software for managing "projects". Let me paint a picture of the interface: you have a list of "projects" which you select then hit a button at the bottom of the page to open the case. Within the project screen there is another button to go to a report page.

As this is about the most common page people use so I trialled installing an "open reports" button directly next to the "open" case button. Hardly anyone clicked on it in our test period (even clients who got an update notification telling them about it) - we actually got one piece of feedback saying it confused them.

I can list 10's of similar examples with our products where a) "intuitive" improvements turned out to be useless and b) split tests caught useless changes quickly.

Users are very fickle :) split tests can be useful.

(sure, not an exact science though).

To be fair, this wasn't an A/B test. If it was actually an A/B test, he would be isolating one variable against a control. In this case, the variable he didn't control was time - maybe he made $5 on the day he wore jeans because the weather was bad or because he scowled at customers.

To be fair to the kid -

1) He's probably right about dressing the part. I wonder if he optimized; is a suit the best way to dress to increase Lemonade sales?

2) It's REALLY hard to isolate variables in this type of marketplace, even for corporations. So, in lieu of a side by side A/B test, what he did shows a lot of forethought for a teenager.

Wow, here's a dude that sells lemonade in a business suit, for seven years, making decent money. And the only comments are criticizing the title of the post? That's almost as hilarious as a dude in a suit selling lemonade in front of his grandparents house.

It's a great idea and I don't want to appear too critical but:

* Was he only wearing the t-shirt & jeans for one session? Was he sure that was what caused the effect? * $4000 in 7 years? The numbers don't really add up. He was out selling for 2 days a week for 7 years, that should be 700 sessions more or less. He really only made less than $10 a session on average?

He was likely in school for 9 of those months, so if you only have a quarter to sell (13 weeks), that's 26 times per year, or 182 times overall. 20 bucks a session at a lemonade stand is a HAUL.

My guess is that the suit made him stick out more, visually, drawing more attention. Getting a potential consumer's attention seems like one of the biggest challenges in a world of things competing for everyone's attention. The suit may have given him the edge.

Reply on news.ycombinator.com