The Most Radical A/B Test I've Ever Done
kalzumeus.com
kalzumeus.com
Desktop applications are increasingly difficult to manage on a school computer. Many teachers return each September to a computer that has been wiped clean.
At the same time, the internet is becoming more and more accessible in schools. Teachers will be able to use the web application no matter which room they are in or which computers they have access to.
Another challenge for non-tech savvy teachers is syncing work between the school and home. I think focusing on the web app will mitigate some of the anxiety that teachers experience in that area.
The last thing I need is school personnel downloading Java apps. Not that yours is malicious - but once someone learns how to install/run something, it becomes exponentially more difficult to teach them to identify what is malicious and what is not.
+1 for syncing between work/school. Often the computers at schools are inferior to ones that teachers have a home, or at least a different platform.
I hear that. Frequently, in September, from several dozen teachers asking me "How do I reinstall Bingo Card Creator on my computer which was just wiped clean? Also, do I have to pay?"
I think focusing on the web app will mitigate some of the anxiety that teachers experience in that area.
It totally solved that problem, where the feature I made in the downloadable version to do the same thing has been a catastrophic flop. (Twenty people have ever used it, and I still get emails asking to explain how to copy files onto a floppy so that they can be read on the machine at school. And then emails the next day saying that the computer doesn't know what a .bcf file is. And then emails the next day saying that they installed Bingo Card Creator and opened the file but it says it is the free trial which is odd because they know they purchased the program and they never had this issue at home [i.e. where they followed directions and put in their Registration Key].)
My Mum has been through about 8 new laptops in the last 14 months...
(the best one is when the laptop leads all went missing [never found out who did it] and their response was... ...to replace the laptops. rolls eyes)
And lets not even start to talk about the restrictions imposed on these laptops.... (I mean; I can understand it for the kids, but the teachers?)
Has anyone else tried this approach?
Anyway, interesting test and do keep us posted about the results.
This gem from the article really illustrates the added difficulty of supporting a desktop application vs a web application. For this particular request what OS she is running doesn't matter, but for many other requests it could be very important to know. Given the technical sophistication of this user, just figuring out what environment the application is running in could be an arduous process.
My initial thought (unencumbered by any actual knowledge) is that the folks who download the client version (instead of sign up for the web version) don't do so for any particular reason, so I don't think that removing the option should actually hurt.
Sounds like a good fit for Bayesian probability, using the old numbers as priors.
Now, it could be that there is a contingent of bingo-loving teachers who absolutely prefer to have the software locally installed, but I'm suspecting (without any good evidence, I grant) that this is not really a key decision-factor for most of Patrick's customers who choose to download.
But I've come to believe, after listening to customers and seeing the behavior in tests where I emphasize or deemphasize the options, that this doesn't say that users want to use downloadable applications. It was just that users want easy to use applications that don't suck. Next to the free web apps in 2006, BCC downloadable was easy to use and didn't suck. Next to BCC downloadable, BCC online is easy to use and doesn't suck.
Do you know one way or another that it isn't?
I mean, I don't think HN readers are in the bingo cards market, but still, it feels like he might be influencing the results by blogging about this beforehand.
Though as you say, the target market and HN crowd isn't exactly overlapping but there may be curious folks who can skew the results or the difference of performance between variations.
But, if people know they are being A/B tested, this allows them to make a decision about what they think they prefer.
Sometimes what people think and are willing to vote (say, by complaining they don't like their option, or by buying a product to support it) isn't what they would have actually picked.
"Several dozen HN users will go to the website, see which variation they get, clear their cookies until they get the right one, then pay me $30... and then cackle madly" is not a real worry.
But, I do think publishing the A/B test does influence the outcome if they do overlap.
You are talking about the $30 that you could get but miss the point: you might get some extra money but in the process corrupt the validity of your A/B test and make a bad decision afterwards, based on that result. Well, not that it matters anyhow, as I'm pretty sure you have actually decided already to stop doing the Java app.
I would like to have somebody that knows more about sociology and statistical surveys explain if what I am saying makes sense or not.
I think you missed his: it's pretty unlikely that anyone on HN will do that.
http://www.kalzumeus.com/2009/09/05/desktop-aps-versus-web-a...
Bingo cards Generator
Printable bingo cards
Create bingo cards
Bingo Creator
In all of them you are not one of the top 3 results. It might suggest that a better approach would be to work on your SEO than to increase your ad money.
He should spend some of his blogging energy on developing a more interesting topic/business to blog about.
Patio11 is not your dancing monkey.
Sure, his writing skills and analysis are decent/good, but the content he writes about is laughable.
Laughing all the way to the bank.
Most tech stuff posted on HN is the cutting edge of web apps. And if it wasn't, I wouldn't read this site multiple times per day.
Well, can't say we'd miss you if you stopped contributing, but I'd have to take issue with the first part. At this moment, the front page has 30 entries of course. I can't believe I'm doing this, but they are about Flash, SproutRobot, Falcon 9, bcvi, HTML5, ideas, diamonds, deflate, BigPipe, Falcon9 (again), Titan, YourWorldOfText, fivethirtyeight.com, being wrong, Toxoplasma, Soros, Readability, Google IO, Posterous, Steve Jobs, Wakemate, Amelie(), the FTC, patio11's a/b test, chrome dev tools, twitter employees leaving, Apple and HTML5, Bing cashback, a world cup calendar and John Gruber. I'll leave it to you to determine if "most tech stuff on HN is the cutting edge of web apps." I tend to think that most business stuff on HN is there to help educate someone to run their business, generally technical in nature.
Finally, there's printing bingo cards is a business any 12 year old with a computer could set up.
Why haven't you, then?
LOL. Comments like this show that you'd be better off reading and learning more about what constitutes a business on those (x - 1) re-visits to this site each day - here we have one of the (relatively) few successful businesses that we get to read in-depth about (i.e., actually making money) and you're complaining about the lack of WizzBang Widget 14.9 or TooToot2000 tech fad coverage.