Simple tests can overstate the impact of search-engine advertising
economist.com
economist.com
That paper concludes that a substantial proportion of conversions via search-engine ads seem to be sales to loyal users who convert via another channel even when you take away the ads, so the conversion rates themselves may be significant overestimates if what you really want to measure is the net impact of advertising.
What I took away from this is that paid links (what the author calls internet advertising or search-engine advertising) aren't necessarily more effective at bringing sales leads as “organic” search links. Maybe I am missing something, but is this "finding" a surprise?
In online advertising you want to measure your success through the rate of causal events, i.e. I advertised to someone and he converted (e.g. bought a product) because he saw the ad. Many advertisers use a correlative proxy for this by looking at how many products they are selling vs how much they are advertising (I advertised more and sold more, so my advertising is effective). The problem is there is a confounding factor that is unrelated to the effectiveness of an ad that causes people to both see more ads and buy more of your product, called the activity bias. The more actively people are surfing the internet at any given time, the more ads they will see, but also the more shopping they would have done anyway. In other words, a lot of the increase in sales are not being cause BY the ads, but for the same reason there are more ads being shown.
The solution is a control group of people who are never shown an ad. During higher-activity times both the control group and the in-group will be purchasing more products, but if an ad is truly effective the in-group should be purchasing even more than the control group. Without the control you can't measure effectiveness because you don't have a ground truth.
I'm intrigued about the 'lumpy behaviour' paper they refer to and will read it later. It'll be interesting to see how the writer interpreted it for the article.