Hacker News Reader

Comment by bemmu | Hacker News Reader

Maybe people like fresh things?

Here's an A/B test I recently ran on some notification messages. There were two messages, only difference is the word "new" in there.

    A: "You received a message from the admins about your Facebook page."
    B: "You received a new message from the admins about your Facebook page."

A clickthrough 15.64%, B clickthrough 15.95%.

About 1000 samples per group. Not sure if noise.

View on HN

https://www.mccallum-layton.co.uk/tools/statistic-calculator...

  Sample Size: 1000
  Observed Proportion: 15.64%
  Confidence Level: 95%
  Confidence Interval: ±2.25
  Range for the true population proportion: 13.39% to 17.89%

It's noise.

Hmm, I tried playing around with this tool.

When I enter 250000 in sample size I get ranges that don't overlap. Does that mean I would need that many samples to answer this?

Yes. The bigger your sample size, the bigger your ability to see small effects. A test with 1000 elements can only tell apart big effects; a test with 250000 can tell apart much smaller ones. This is the idea of https://en.wikipedia.org/wiki/Statistical_power

.3% on 1k? That's three people.. Yes, noise.

That's a great way to get a gut check on the validity of the result. Put it in real terms.

I'm often annoyed when reading reports that use percentages instead of the actual ratios. For example, "2 out of 3" is much better than "66.7%".

Reply on news.ycombinator.com