What?
If this person genuinely thinks that there is no way to measure an ad's performance while it is running, they are either too ignorant to be in business, or they hired someone who lied to them.
You can get great CTR and conversion rate running ads against your brand name. But if your SEO is sound you’re pretty much just canibalizing your organic traffic.
I’m not aware of a way to determine whether an ad just claimed credit for a customer it didn’t earn. I’m interested to learn though.
While it's not a perfect system, you should know with a reasonable degree of accuracy how many dollars you get back for every dollar you put in to ad spend.
As far as the SEO traffic issue, this can also be accounted for once you have scaled your ads beyond the levels of what SEO could yield. At low budgets this can be more of an issue, at high budgets much less so.
That's not trivial to do, both from a technical point of view (browsers - rightfully - fight these kinds of tracking attempts) but also legal (GDPR mandates that the customer opts into this kind of tracking but they have no incentive to do so).
But this is a good place to point out that, yes - there is nuance to all of this - and I didn't really mean to turn it into a thesis on online advertising (lol) so much as to say:
It's still easy to be more accurate than "Turn it off and see what happens".
Even a semi-sophisticated media buyer is going in with a plan, and some method of measurement.
Put another way, the only way to tell if a customer will still find you without the ad is to not run the ad.
If you have sufficient traffic you might be able to measure this by turning your ad spend down rather than off and measure the bottom line impact. But other than gross spend changes there’s no way of telling for sure that the ads do anything.
Sure, maybe 100 customers would eventually find you. But if you buy the traffic, you can make them all find you on the same day.
> But other than gross spend changes there’s no way of telling for sure that the ads do anything.
I'll use the most fundamental example:
If someone clicks your ad and buys the product during that session, you know the ad worked. If you keep increasing your ad budget every day, and you are consistently returning $2 for every $1 you spend, you know it's working. Turn off the ads, and the revenue goes away. You'd be surprised how many people make their living doing this.
This is the experiment from the article. It showed that ads did not affect revenue.
This is not as clear as you make it seem. Let's say there's a hypothetical product that a consumer only buys once every 5 years. If someone clicks your ad and then immediately buys the product... what if they would have bought it anyway, tomorrow, or 5 minutes from now, without the ad? How can you test that counterfactual?
If I buy cat litter online once a month every month for 5 years by going to example.org/catlitter -- but then they decide to start advertising on facebook, so now I click the facebook ad once a month to buy cat litter from the same site, are the ads "working"?
Also, definition of working: You make more than you spend, and if you stop the ads, you stop making as much.
That said, I'm not arguing for 100% accuracy either. It's certainly not. Simply that it's possible to be more accurate than not, which leads to profitability.
One more also- a comment above about how I failed to mention I'm not talking about paid search ads so much as other types like FB, YouTube, banners, etc.
That's easy to track, but answers the wrong question. It answers whether the person clicked on the ad or the organic link. What we actually want to know is whether this person would convert even without seeing an ad. The only way to answer whether a person would still convert without seeing an ad is to make sure that they don't see an ad.
On the flip side of that, most of HN seems to think purely in terms of paid search ads, when there are many other types of online advertising that exist, and that don't have this issue baked into them.
Such is the nature of internet dialogue, I guess lol.
An ad-click could have been an organic click if the ad wasn't there though, which is where the complexity is.
The ad might have great conversion, but if the customer would have clicked an organic link that navigated to your site anyway then the ad is taking credit for an organic sale.
If I get 100 organic impressions/day, and then spend $N to get 500/day, I'm speeding the process up, and nearly guaranteeing to get in front of people who would never see me organically.
*Note: I'm largely not talking about paid search ads. Those are definitely an area where you can end up competing with yourself. Sorry to anyone who I replied to earlier, and wasn't clear enough on this with.
If you can't answer that question, then you're not running a proper advertising campaign. You're just throwing money around blindly. Plenty of people do have good answers to that question, though. Unfortunately, plenty of people don't.
What is this proper advertising campaign? Does it include turning off ads for terms that rank organically high anyway?
Well, for our business, proper advertising campaigns have opened new channels for growth, allowing us to reach new people, and build our email list significantly, with a predictable ROI. It's pretty easy to tell if an ad is working when you turn it on, and your email list growth rate instantly doubles -- and all the traffic is coming from a new traffic source. (Obviously, if we were to run Google ads on search traffic, we would check whether our organic search traffic dropped when we turned the ads on, but that's a different point.)
Maybe for a real tight definition of instantly, but generally this is not true. Imagine the following scenario:
1. You run an ad for ComapnyName.
2. An enthusiastic customer promotes your business in a local bar.
3. The bar attendees search for CompanyName and click on the first result.
Because you ran an ad for CompanyName, the first result is going to be your ad. You will see very nice ROI on that ad. What you won't see in any stats, is that these people would have probably found the organic link anyway, because they were already motivated and searching for CompanyName in particular.
> Obviously, if we were to run Google ads on search traffic, we would check whether our organic search traffic dropped when we turned the ads on, but that's a different point
That wouldn't help either in this imaginary scenario, because these bar attendees are a spike in traffic. Plus if the ad has been running for a longer period, then you won't have accurate organic search traffic stats anymore either, because it's already cannibalized.
"Imagination" is the core of the problem here. Plenty of people are imagining various scenarios, whereas people with successful advertising campaigns are just raking in the money -- no need for imagination.
I've turned off plenty of ads when they stopped working. I didn't need to turn them off to discover they stopped working.
Sure, you can imagine a dozen scenarios where an ad campaign doesn't work. Fortunately, I've managed to learn how to focus on the reality of the situation -- and have reaped the rewards in the process.
> you won't have accurate organic search traffic stats anymore either, because it's already cannibalized.
That' not true for us -- as I said before, a successful campaign for us brings in new sources of traffic from new marketing channels. One of the early lessons I learned was to not cannibalize what's already working. For example, we grew our Facebook Page (and email list) from scratch, when we had no Facebook traffic, by using Facebook advertising.
Operating under different brands -- even just for testing purposes, is a one another of the way we deal with this. Of course you can continue to imagine scenarios where we might be making mistakes. I do that as well -- it's called planning. Though, none of that really matters until money is spent (and made or lost).
What I'm talking about in specific is search ads where the ad is for a term that ranks organically high anyway.
It's not just imaginary either, I've done a lot of over-the-shoulder customer observing. Just recently I saw a friend search for "dropbox" and then click on the first result in Google, which is a paid ad by dropbox for dropbox. They rank #1 anyway!
Now in dropbox's case it might be worth it, because they have enough competitors who would like to steal that ad spot. However for most businesses that's not the case for their top terms.
The word for this in the business is "incrementality", and there are several ways of measuring it. The simplest conceptually, is that you run two ads to two random groups of users: one for the product and one for something irrelevant like a charity. Then you compare conversions between the two groups.
(There are fancier ways to do it that don't require you to spend half your budget on an irrelevant ad, but that's the basic idea.)
+1 to everything you've said.
This is would be another experiment that does not test what you need to in order to demonstrate some sort of causality.
Much better is to run your control group and experiment group in parallel, like I described above.
Is your objection that showing ads for something irrelevant could affect whether the user converts?
So the experimental setup is: you create an ad for your product, you create an ad for your charity, then you compare the populations of people who click on your ad for your product with the population of people who click on your ad for the charity and see if there is any difference in conversion rates between the two populations.
How do you ensure that the people who see/click on your product ad aren't already a population more likely to convert to your product than the population of people who see/click on your charity ad? Sure, you can target the same groups of people - but that only goes so far, the ML algorithms backing the ad selection process will still preferentially show your charity ad to people likely to click on charity ads and show your product ad to people likely to click on your product ad.
I am unfamiliar with how these experiments work on the advertiser side. Incrementality is easy to measure on the platform side if advertisers report conversion metrics to you.
<script>
var treatment = readTreatmentFromCookie();
if (!treatment) {
treatment = Math.random() < 0.5 ? CONTROL : EXPERIMENT;
writeTreatmentToCookie(treatment);
}
if (treatment === EXPERIMENT) {
showAd(PRODUCT);
} else {
showAd(CHARITY);
}
</script>
Many networks offer the ability to run a fully supported incrementality study, where they effectively use one company's ads as a control for another's, but this is a version of an experiment that you can run even if you don't trust the ad network at all.
If you have offline channels directing you to a website as well, like television or print, it can get messy unless architected properly.
Those are extremely different, how does that prove anything?
If you show a neutral ad, you know exactly how much seeing your brand in ad drives revenue compared to seeing an irrelevant ad.
So in the real world I bet this is what you're talking about: "but there are clickmonkeys out there" or words to that effect.
[edit:] But to be honest, I'm not sure it directly answers the question about entanglements between ad and no ad.
The protocol for doing so would involve studying what happens when the ad is present versus when it is not. The goal is conversion, scrupulously defined as people who click one or the other and subsequently purchase. Total conversion could go up, down, or stay the same. The only way the ad "wins" is total conversion increases, and even then maybe. If organic conversions went up when the ad was present you'd have a research problem! (The effect could be time-based, i.e. "awareness", or it could be a confounding externality.)
The protocol you suggest would seem to be removing the organic placement when the ad is present, that is: one or the other. On its face this sounds more "researchy" to me. Putting feasibility aside, it would plausibly be attractive to an advertising professional, but I would espect the customer to ask "why?" and I don't see the answer to that question. What's the motivation for this approach? I can see that it makes the advertising professional's contribution crystal clear, but why should the customer pay for it?
But hey I don't have 30 years of advertising experience, nor do I consider myself a statistician or machine learning expert. I do however have over 30 years of experience as an internet plumber and (more importantly here) data sous chef, so I've tasted a lot of ingredients in a lotta stews and have a solid grasp of experiment design and causality.
You work for the customer: consider that some avuncular advice.
That is what you're talking about, how slicing up your signal before transmission impacts your ability to receive it. Here's a Jupyter Notebook which will maybe make your head explode... I mean if you like math.
If you are doing advertising right you build a funnel from awareness through to conversion and track every part of it so you then a/b test your adverts and channels.
Getting advert channel fit is the key to success and can only be done by testing and measuring your adverts. The fact that op couldn't measure the effectiveness of their adverts shows they weren't doing this and their bad outcome should be 100% expected.
If you are just buying adverts in the hope you get more conversions then you are putting those adverts out to die. Burn your money you will probably get more eyes for doing that than untested adverts.
The idea to simply turn off ads and see the effect is born of a healthy distrust of ad analytics industry bluster.
This is a great counter-point when someone mentions "use conversion tracking!" Conversion tracking is great, but not if they store a cookie for 7/15/30 days and "award" the conversion to the ad, when the customer took a different and varying path to purchase. Sure the ad contributed "some" to the conversion, but not 100%.
A story to illustrate: there was once a pizza store that had two guys go out into the city to distribute promotional coupons. The coupons had codes on them so the business could attribute sales to each coupon distributor. John went out into the city and tried his best to drum up new business. Chad stood next to the door of the pizza place and handed a coupon to anyone who was walking in. After a month, 98% of the coupons used were from Chad. Chad got a big bonus and John was let go.
There is no easy way to determine whether this is the case without just comparing when the ads are on and off and somehow dealing with the confounding variables, which isn't easy.
The most striking example is probably the experiment Tadelis convinced eBay to run on their brand-name ads, though the subtlety applies to less obvious cases of questionable ad spend too — https://faculty.haas.berkeley.edu/stadelis/Tadelis.pdf
Doing a blackout month is a legitimate technique.
I've deeply studied ad performance metrics and they are NOT as conclusive as people might think. If they were, you could just keep increasing the amount of $ spent and the conversions would go up. Ok, not exactly, there is usually a diminishing returns aspect, but you get the point.
A great example of why this requires more analysis is branded vs non-branded search terms on AdWords. Let's say you spend $10k/month on the brand "Mattel" (aka branded) and $10k/month on "toys" (aka non-branded). Mattel likely gets you like a 20x ROAS because people are probably searching for a specific Mattel toy. It's also potentially likely that if you completely turned off branded search you would net the same results.
Ad attribution is NOTORIOUSLY difficult: https://www.optimizesmart.com/what-is-attribution-problem-in... so to pretend like "this guy is an idiot, he doesn't know what he's doing" is disingenuous.
This is the case in many situations. The time between initial visit to site to conversion can be weeks, months, or years. Meanwhile, customers will visit site from various platforms. There is no way of attributing value to a single click, it needs to be approximated as an aggregate.
The Dunning Kruger effect is strong in marketing. The more you learn about marketing the more reason you have to be unsure.
Does this ad improve conversion compared to not having the ad at all?