Hacker News Reader

Comment by patio11 | Hacker News Reader

Happy to answer questions, as always.

By the way, roughly 3/4 of A/B tests I participate in (my own and for clients) fail to improve business results. This compares with approximately 2% of A/B tests I've ever seen with in-depth blog posts, so I tried to redress the balance here.

View on HN

Some thing that you might want to look at are the number of free cards you are giving away. With many states having mandatory class size reduction class sizes are smaller and 15 would cover a lot of classes. My wife's class this year has been as small as 12 students and as large as 18 students. With 15 free cards she would just make the other three by hand. A lot of that is due to us already spending a large amount of money on her class due to budget deficiencies and salary freezes for all teachers for three years now.

Another issue you might be encountering is that states are adopting the common core standards http://www.corestandards.org/ and word bingo doesn't meet any of the standards for english language arts. Since everything in classrooms is documented, lesson plans are stored and in some cases handed over to the state there's not going to be a lot of use cases for activities that can't be applied to the standards. Even if those activities are just for fun due to a holiday.

I've experimented with 12 vs. 15 before. Null result. I might try pushing it down to 8 sometime. But, realistically speaking, only if it strikes my fancy some morning.

Ok, I freely admit that I know nothing about selling bingo cards.. But it "feels" to me like that number should be closer to 5. I was really surprised to see that you give away 15 cards for free. You could A/B test this too, right?

Another idea: start high and ratchet down with each subsequent word list?

How do you break that news to clients after they've paid $$$ for your services? Do you heavily caveat the potential before the engagement?

That's surprisingly less of a problem than I naively thought it would be. I'm upfront with clients prior to, during, and after engagements, and I try to set expectations accurately. That said, for the overwhelming majority of my clients, if e.g. the entire budget for my wedding was spent on a project whose final conclusion was "Experimental results: we now know one more thing that didn't work" that would not be a catastrophic loss.

Businesses routinely spend $$$, $$$$$, and $$$$$$$$$ on things that don't end up working out the way they were planned. They're pretty much OK with that, since they run "portfolio strategies" in terms of directions. Nobody is betting the company on a two-week engagement with me, any more than they bet the company on any particular man-month of engineering time.

n.b. This is partially influenced by being picky about clients and working with people who are clueful and have budget. This is one reason why if a startup with $X0,000 in the bank comes to me and asks to roll the dice on an engagement I give them a link to my blog, some suggestions, and invite them to come back when we're a better positioned to help each other out.

I suspect that people hire you not to do "a successful experiment" - but so that they can learn how to experiment in productive ways.

It's a skill that seems sadly lacking with many of the folk that I talk to :-)

The glass half empty is that there was no benefit to the redesign.

The glass half full is that you have a more visually pleasing, refreshed web site that is at least as effective as the old design. Patrick also notes that the users tend to associate visual changes with programatic improvements, even when there are no programatic improvements (placebo effect).

You just run more tests, such that 25% of tests run still turns out to make $$$$ at least.

Well, keep in mind that, with a significance threshold of 5%, one in twenty null-result tests will appear to have significance.

So just continuing to test until you get a result is not necessarily going to net you anything.

One potential upside is that, as far as the numbers are concerned, the client is free to use whichever version they want.

You recommend Stripe in a throwaway line in the post - it sounds like your experience with them has been really positive! I'm working on a project where I may end up taking people's money for services on a subscription basis. I've heard that Stripe is better for one-time charges than for re-occuring charges. Is that true in your experience, as someone who's done work with Stripe?

Stripe is a dream come true for developers. I use Spreedly (backed by Paypal) on Appointment Reminder to do subscription management, and am very happy with it, too. I hear you can use them together now, which sounds like the PB&J of money if you ask me.

P.S. I also don't hate me love Paypal, too.

As a user I love paypal because it is really, really easy to end a subscription and there is nothing, whatsoever, the merchant can do about it.

Does Paypal still have the restriction that if you use them for credit card processing, you must also offer a Paypal checkout? Offering Appointment Reminder though standard PayPal seems like a giant additional headache.

I'd never heard of Spreedly! Thank you for pointing that out: it makes this project just the tiniest bit more likely to replace my day job. :)

Maybe you covered this before and I forgot, but how did you arrive at the number you use for the max cards creatable with a free account? I get the 'more than the # of kids someone would have in a family' but 15 seems... high.

You're on the right track, but it's more than the number of kids in a classroom — not family. He mentioned it in a podcast somewhere.

Yep, most A/B tests won't give results. But the ones that do are worth it (provided they were quick and easy to setup). There's always an opportunity and design cost involved with A/B testing, but it is sort of ironical that you can't be judgmental about which A/B tests to run, because if you already know which ones are going to work, why A/B test in the first place?

Do you have any sort of logical explanation or intuition about your stated result - that 75 percent of tests do not improve business results?

It seems to suggest that design just has to be good enough. Below some set threshold for visually appeal and/or usability the design hinders money-making. However, above that threshold, further improvement isn't necessary? If this is a reasonable interpretation, that is actually quite sad and suggests that all the new design-centered startups are wasting a lot of money.

You mentioned that sales were strongly up for reasons unrelated to the post. What are those reasons?

Short version: traffic way the heck up, probably through a combination of Google algorithm changes, just natural growth of the Internet among people who matter to my business, and organic growth.

No questions here, just a small niggle: "Fewer users", not "less".

Actually, that's a widespread misconception. See here, for example: http://itre.cis.upenn.edu/~myl/languagelog/archives/003775.h...

That's a linguist's take on it - you can find many more by googling [fewer less languagelog] or similar.

That mainly refers to units of time, amounts, etc. "Less than 5 users", I can maybe accept. "Less users", no.

You sell the bingo card creator 5000 times a month? Never expected that bingo is such a big deal. Congratz! :)

Not sure where you're getting that number from but it is not accurate. http://www.bingocardcreator.com/stats/sales-by-month Units on the graph are USD.

Curiously, 26 conversions (@ $29) in 7 days seems low compared to your March figures - naively dividing out I would have expected 39 sales in 7 days...

Sorry, didn't see that the charts are in USD there. Anyway, thank you for sharing your insights with us.

Reply on news.ycombinator.com