Looks vs. Results: My ugly ad got 150% more clicks than a professional design
gkogan.co
gkogan.co
I will also add that the "professionally designed" ad is kind of busy. It has so many things going on (some text in all caps, different colors, some quoted, illustration doesn't add much value, arrows that don't mean anything) that it doesn't really catch your attention or call you to action. The "ugly" ad, however, gets to the point.
there is a bot that poorly creates phrases for targeted subcultures and digitally affixes them onto T-shirt designs and makes targeted ads for them. per usual with T-shirt sites, when people purchase, the design is created on a real t-shirt at that point in time.
there is a whole subreddit about the results.
I am humble, so humble! My humility is even written on my shirt! Also, there are some badass mock angel wings drawn on my back, in case you had been forgotten of my magnanimity.
Sure if your product is good some industries do not care how your design looks.
It's all down to the industry. If you were selling expensive water to schmucks, MS Paint simply will not suffice.
The reason it feels intuitively broken is because the conversion rate is so low. But there were about 10k impressions in his test.
Can't be 95% confident in something if natural variability drowns out the effect. That kind of conversion rate is one bot RNG away from returning to the mean (noise), it just takes some domain knowledge to recognize why p-hacking is an illusion.
It's not uncommon that people fail to notice the nuance here and it's quite important for correctly interpreting reults.
So there's about 3 standard deviations between the two, this sounds like quite a bit but really means they're both 1.5 standard deviations from the supposed mean. Which is, not great, though it might pass some of the weaker statistical tests.
Now you should actually weight the values by the total number of impressions in which case you might get a slightly higher significance since the one with 12 clicks was seen by more people.
So on the balance you should be wondering what you're paying the graphic designer for, but perhaps not start a new career designing low-budget adds.
12 vs 24 clicks is not significant, it could’ve gone either way. Also given this minuscule sample, it’s easy to conduct p-hacking to get the desired outcome
In other words, if I'm doing the math right, for him to p-hack this by rerunning the experiment in the case of no difference, he'd have to run it more than 100 times to get a 50% chance of getting as good or better significance.
There's of course plenty of other things that could be wrong outside of the simple statistical test, he could be making the numbers up, the groups might not be properly randomized etc.
Linkedin calculates an ad impression (last I checked) as 50% of the ad is on screen for at least 300ms. I can be scrolling as fast as my thumb will flick down my LinkedIn newsfeed and it would probably count.
Then the KPI being used is clicks. I don't know of any business owner who would take that as valid. It should be some kind of conversion event (newsletter signup, contact request, or purchase, etc.)
If it were up to me, I'd want 4,400 clicks and a few dozen conversion events to do my calculations on statistically significant effectiveness. Especially since the author is paying CPC (cost-per-click)... who cares about impressions at all?
It's a statistically significant difference.
That seems like a highly significant result to me...
The fact that it has a significant p-value is interesting, but the lack of information about how the author decided when to stop is suggestive of p-hacking (i.e. we don't know how many screenshots were taken, but we understand that the author posted only the most favourable one)
R> prop.test(c(12, 24), c(5563, 4403))
2-sample test for equality of proportions with continuity correction
data: c(12, 24) out of c(5563, 4403)
X-squared = 6.5211, df = 1, p-value = 0.01066
alternative hypothesis: two.sided
95 percent confidence interval:
-0.0059903650 -0.0005970741
sample estimates:
prop 1 prop 2
0.002157109 0.005450829But in this case I suspect people were just curious why the ad looked like that and clicked to find out. Those may not be the people likely to convert.
Especially if the second option is cheaper and faster, there's IMO no bayesian prior that the professional ad being better (the null hypothesis) is true.
So... seems like useful data to me.
With a prior that A is superior, you may still end up believing that A > B after updating, because there’s just so little data.
And my main point is that a 60% probability is in fact actionable in the real world, in a situation where you are forced to take action with incomplete information. Assuming you are running an ad campaign, you have to choose one of the two.
P=.95 still is an arbitrary threshold, even if it's a commonly used one.
Are you sure about the latter? How do you know that the people that click the ugly ad weren’t just curious about what it was because of how it looked rather than what it was about?
The scribble version is, at least, novel. Still probably wouldn't click through, but at least it would grab my attention for a moment.
It communicates the useful part (in particular, the "call to action") clearly and instantly, without distraction.
There's more going into this than just the concept of writing on paper.
Use a design that costs me money to produce and doesn't work, or...
Use a design that does not cost me to produce and does work, and then find something new to do when the novelty wears off?
Not a tough call for me.
A higher clickthrough rate isn't necessarily better if more of the people who came from the non-designed ad didn't do much or anything on the page and mostly clicked out of curiosity.
They don't have to worry about actually selling you a product.
Some donkeys look cute
I got to see the lesson I had to learn with our clients everyday on my commute for 3 months. I won’t soon forget it.
But, if the goal is the download the yes this works. In large part because of why 'one weird trick' took off. It's simply different. Different causes our brain to pay more attention. The professional ad, just like all ads today are relatively ignorable because we've "seen it all" and are mostly annoyed by them.
This tends to be why trends are cyclical. There are only so many things you can do with a specific medium. So, it's relatively expected that once a medium hits a certain tenure, it'll recycle on itself.
It's the guardian at the gates.
People are just more willing to give their attention to things that are novel. Which is of course based on context.
It's also one of the reasons why I've taken to using drawings within my own blog posts [1]. I've seen time on site that is MUCH higher than industry benchmarks.
https://medium.com/@paulbars/magic-internet-money-how-a-redd...
Or is this purely a ponzi thing where you buy bitcoin worth 100 bucks, spend 100 bucks on advertising and end up with the same bitcoin now worth 1000 bucks.
The self-starring commercials were an enormous success, and became a meme among Connecticut viewers through the 90s, but they also got people into Bob's locations. Today, Bob's Discount Furniture has opened stores up and down the Eastern seaboard and even on the west coast, He's retired from starring as himself in his own commercials, being replaced with a puppet likeness and becoming a sort of founder-turned-mascot like Col. Sanders. But the chain's enormous success wouldn't have happened without those initial, cheaply made, cheesy ads.
Sure Option B had around half the CPC by the looks of it but what happened next? That’s the most interesting thing of all and it’s missing from your post.
Very ugly site, but impossible to forget that branding. It's my understanding they do very well.
People are indifferent to car dealerships (They're all typically the same. Yes, luxo-brands will have free cookies and nicer waiting areas...), just like with insurance-- they are "boring" verticals so having something so out there works great. When you look at insurance companies, which companies have memorable marketing? Companies like Liberty Mutual with their vanilla "Save $200/year" commercials, or the Geico Gecko, "We are Farmers....", All State's Mayhem, etc.. The cheesier the better, since auto insurance is a boring, fungible product to consumers. Compare to another boring vertical like Men's Razors. They're essentially all the same. How are any of the marketing campaigns from P&G/Unilever actually differentiated enough to be memorable (ignoring the duopoly considerations)?
Having unfortunately worked in marketing, everyone seems to shoot for the middle-- and they end up being bland and forgettable. Chances are, you're not going to have a super memorable marketing campaign by doing something slightly different than everyone else. The companies I've worked with where their marketing is intentionally cheesy have always had better reception than the vanilla "professional" brands.
A lot of Indian websites look like this too (like paper advertising leaflets).
If the intent is to grab attention or pique curiosity perhaps we can say option B is better designed than option A :)
It became apparent that journalists tended to ignore them because they didn't look like new - quickly produced and hof the press. So many PR agencies went back to using double-spaced Courier.
That is a false assumption.
Not all clickers are equal.
Not all leads are equal.
More is not necessarily better.
Better is better. Full stop.
So what does it matter if the "ugly" ad performs better? All that means is you go bankrupt slower. Is there room to improve the performance of the ugly ad so it wanders into the realm of profitability? I'm guessing there's far less room than for the professional ad and, probably, there doesn't exist a design that will drive profitability with the current offering and more focus should be made in customer development to discover and communicate a clear value prop rather than mess around with ad copy.
Might just be because of novelty factor, but also may be tricking less sophisticated users because it looks different and those users may be less likely to convert.
It'd be interesting to see conversion results too.
the ugly ad that violates most of the ad traditions is more likely to be followed by me. (especially if it lacks the call to action part. I hate calls to action. especially when it's me who's going to pay)
As other comments have pointed out, I think it's a matter of standing out, and in this case lowering the standard of quality works, but soon the novelty will be something else.
So measuring CTR does not tell anything about if version B is better. It may be worse.
To compare we'd need to check CPA with A/B testing.
I'll click on free.
On a second look... using a ballpoint to write on that glorious paper is a crime, so I might have hesitated a bit.
The minute I see a generic ad I instantly ignore it.
I’ve worked in performance advertising field enough to see this come true so many times.