What is A/B Testing?
visualwebsiteoptimizer.com
visualwebsiteoptimizer.com
If you need that much animation on a page to tell your story, isn't it a good sign that video might be more fit for purpose? Video was designed for storytelling, it will work on most devices, and it has well-established UI and conventions, such as a play button and progress bar.
If you're going to provide a play button in your scrollmation to automatically advance the action anyway, you've already recognised how irritating it is to manually page through an animated story by scrolling or swiping. So why then go to the trouble of creating a scroll-based animation that repurposes a 20+ year old convention and attempts to reinvent a tiny proportion of what video has to offer?
Is it just for the novelty value?
Is it because it makes you feel smarter?
How is it better for users trying to understand your story?
I'd love to hear from anyone who's considered both video and scrollmation and chosen the latter.
This particular animation in a video format would take 10 seconds of running time. It would also take few seconds to load and initialize whatever video playing widget it'd be using. That would make it for an oddly short video and one of those "What? That's it?" situations. So it simply wouldn't work as the only content on the page. OTOH, if you present it as an interactive animation, it will take users longer to work their way through it and so it would feel more substantial.
Secondary reason is that it makes the page more memorable and helps it stick in more people's heads than some yet another embedded video. The goal of the page is to engage people. This is the reason why you'd want to have things presented rather than just thrown on a page in black and white Times New Roman / 12px. You want your page to engage and clicking on Play Video is not engagement, it's a reflex. So by forcing visitors to do something out of their routine the page, effectively, makes them to pay more attention than usual.
Not everything that's memorable is good. I was happy to see the "play" button alternative to the scrolling interaction. I used the play button, enjoyed the video and believe I will remember it without experiencing the annoyance of having to manually play it using the scroll bar.
Try speaking it clearly and slowly as a voiceover artist would as you go through the slides. 'Meet Bob. Bob is a fisherman. He runs his own store and has an online store [boom, there's your 10 secs]'.
Memorable isn't good if they don't get to the sales ptch, which most people wouldn't.
Personally I completely agree with modernerd, this would have been a waste of time and money if it wasn't novel enough to get featured on HN or /r/programming or stuff for a few extra sales. For its stated purpose, it's useless.
Not saying I'm right and you are wrong, just that people have different irritants, as it were.
A lot of people will need something thats this simple to digest on their own speed, though. I will be sending this link out to a few past and potential future SMB clients.
The amount of people like "Bob" who are running their own website is astounding. They need simple things like this that help introduce them to the idea of A/B testing.
- It's user-friendly.
- Compatible with everything.
- I can read as much or as little as I want.
- If I want to re-read something, I know exactly how to do that and it doesn't involve clicking a play head trying to find the place it was in.
I feel like people keep trying to bring back reincarnations of flash interactivity and since it uses html5 and other buzzwords, it's ok. Well, it's not.
Every time someone "fixes" the experience of what should just be reading online, they make it much more cumbersome and annoying. One day everyone will look back at what people did with html5 and laugh the way we all look back and laugh at flash intros and playing music on load.
It also makes skipping or rewinding a matter of finely-navigated clicks into tiny target areas, whereas the best hardware scrolling controls offer giant, forgiving action-areas yet that still enable nearly pixel-level resolution and fine steps of speed/acceleration.
Depending on your device, scrolling in many directions might be very, very easy. Large trackpads, the Apple 'Magic Mouse', and most touch screens can make scrolling far more easy, adaptive, and intuitive than even 'forward' and 'back' buttons.
I tend to think people most peeved by scrolling as a major navigational technique (in both space and time) are still using tiny trackpads, mice with nubby scroll wheels, or even (shudder) ThinkPad-like joysticks or keyboard arrows. Stone knives and bearskins!
Does this apply to you?
They go too slow, they're usually painfully cheesy, and I hate that I have to listen to them. I read much faster than the narrator talks.
A video for this demo probably would have lasted 1:30 or so. I was able to scroll through in about fifteen seconds, didn't have to find headphones to do so, and got all of the same information.
That's a good point. There is a general expectation that playing a video also means playing audio, and some environments – like offices – aren't great for that.
> ...user can control the speed of information. ... Video has no way for that.
I frequently skim through video by dragging the head in the progress bar, or skip three or four minutes ahead in longer videos. With scrollmation, the default scroll speed is normally much slower than the equivalent video speed of 30fps, it results in jerky movement, and the speed of the video is directly proportional to how hard I am prepared to work to make it go faster. Good UX designers don't make users do shit work.
I can only speak for myself, but I rarely, if ever, watch website videos, especially at work.
I made it through this entire thing by scrolling.
While I understand video is sometimes the best way to convey an idea, I fricken hate it. It would need to be something very, very intriguing for me to watch. 99% of websites selling things don't fall into that category.
Personally, I liked the animation.
I've seen great results from the signup forms that http://wistia.com allows you to embed anywhere in a video. I'm not affiliated, but am impressed with the product; check out their "Call to action" and "Turnstile email capture" features in the embed builder on this page: http://wistia.com/product/superembeds
I agree. There is no way she would know to scroll properly in this interface and in most cases would get confused and give up. The objective of the presentation is to make it easy for someone (like my mom) to understand A/B testing. Make it a video, or at least provide a link to a video version.
EDIT: This akin to Amazon's usability - http://www.nngroup.com/articles/amazon-no-e-commerce-role-mo... i.e. once some behavior is establish on your site, keep it that way, but doesn't mean you need to emulate it.
Control for the users.
It's confusing first of all because you've chosen a product which people don't generally sell online - fish. If you'd chosen bikes or clothes it would have made sense.
The other thing is on the "play" version - it takes about half the animation before you even get to the online bit and when you do, the illustration of how A/B testing actually works only takes up a tiny bit of real estate.
For my money I'm not convinced that the offline analogy really adds to it. You know your market better than me (or anyone else here except Pete and Dan) but I feel like you could jump straight in with a story about an online store.
"Mike sells bikes online but he's not sure which ones to put on the front page. Will he get the most sales from showing someone bikes on the frontpage or showing them chainsets? He doesn't know and he finds it hard to read his analytics to figure out which one is best. VisualWebsiteOptimiser does all this for him..."
You CAN do A/B testing of sorts on the physical store too. This might break the attention of people that know that.
I guess the intention were good, and the drawings are of good quality, but otherwise the execution is just terrible.
First off, I was distracted by that slideshow because I kept thinking, "Who the f sells fishes online?"
Second, I felt like there were two points in the video because of the whole offline analogy. 1) Selling online is better than offline 2) A/B testing makes you more money
Within the example of the store for example, the real pain A/B testing addresses is cyclical variation leading to incorrect conclusions.
Working from that you could've shown the story of:
* the store owner rearranging his display and seeing sales go up
* but then the rest of the year sales actually going way down
* and when he changes things back to the old layout sales went waaay up again! How can that be?
* Cue image of cyclical charts and the facepalm of realizing it wasn't his redesign at all!
* Now ... on his website he can send visitors to different versions at the same time!
* So he'll never have a year of bad sales again!
Do you think it achieves that purpose?
Looking at it as sales/marketing copy though I'd say that your biggest competitor by far isn't people not understanding what it is but people thinking "Why don't I just change my site and see what happens?"
In that context I'd guess this story telling format is perfect for taking a concept like 'prevent acceptance of false hypotheses by controlling for external variation through randomized trials' and turning it into a visceral "Oh fuck. That's painful!" moment your target market can relate to.
Seriously, I think you make a great point of selling your service. I also think you gloss over things like cyclic variations -- but I'm not sure it's your job to educate people in the subtle art of fact oriented market research (or what one should call it).
For those that don't have the time to learn, changing it probably wouldn't matter. For those that do -- chances are they probably will find it out from a more comprehensive resource (like, say, a good book).
Coming to your suggestion, do you have a good book/website in mind?
Can you suggest one?
[1] https://github.com/Prinzhorn/skrollr [2] http://kushagra.me/blog/
It is the easiest star to sketch -- even my teacher in elementary school used to use the same star in my grading.
I know it isn't specifically A/B testing related, but throwing another shoutout to your app at the end like "heatmaps too!" might be nice. It seems people that would look at this slideshow might not be familiar with any web analytics services, and it seems you guys offer many services.
What about multi-variate testing? How do you explain the increase in the number of visitors needed to make a decision?
Finally, how do you reconcile giving up-to-date information without compromising on the results? (keeping an experiment going until it's statistically significant is something I can see users doing)
Those not initiated in Statistics find this a little difficult to understand. I personally go about it like this:
Me: "If you flip a coin 10 times and it shows heads 7 times, can you say with certainty that the coin is loaded?" Them: "Umm, no not really"
Me: "Ok, what if I flip it 100 times and it shows head 70 times?" Them: "Yeah maybe"
Me : "What if 7000 out of 10,000?" Them: "Of course it's loaded"
And that's how I explain the basic concept of statistical significance, but without the math. More data means it becomes more 'obvious'.
To take it a notch further, I ask "If your life depended on it, would you still bet that the coin is loaded?" People usually become unsure after this and I joke that their previous decision was based on 95% significance, but if they really want to be sure, they should wait for 99% significance.
If you've described that accurately, you're hurting some of your customers. It's not well-known (though it should be obvious, in retrospect), but a test can produce a false positive when, in fact, it's a negative.
The problem is that there is always noise in customer data and sometimes you'll randomly have enough conversions such that one of your tests will show a winner when it's not.
Matts Einersen has some simulation code to show this: http://www.einarsen.no/is-your-ab-testing-effort-just-chasin...
Evan Miller has a more in-depth (but harder to follow) discussion here: http://www.evanmiller.org/how-not-to-run-an-ab-test.html
If, in fact, you show the winner as soon as significance is achieved, it's then trivial to demonstrate that some of your customers will lose money as a result. However, if you take this knowledge and then find a good way to incorporate this into your tool, you may have another great selling point that competitors do not (that being said, the idea is complicated enough that some customers won't understand it).
Also, it makes me very happy to know that you're aware of this problem and are trying to tackle it. It makes me more likely to recommend it to clients.
It looks like you've got a good balance there :)
Deleted comment
Also doesn't render at all well...
couldn't have said it better myself
(you'll have to wait for the liquid to reach there to see the effect of your choice)
Not sure if it is correct though. Do you have a link to a better explanation?