It boggles my mind when HN commenters start claiming that measuring ad campaigns is impossible or that ads are universally ineffective. Anyone who has spent time working with large ad campaigns should know the tools and methods used to measure these things.
I suspect the HN sentiment comes from a common feeling among techies that they are somehow immune to influence from advertising, combined with a high adoption rate of ad blockers. HN commenters tend to assume that other consumers are just like themselves, which is far from the truth. In the real world, advertising (when done right) is not only very effective but not that difficult to measure using modern technology.
I don't believe this at all. The reason I am skeptical of the effectiveness of advertising is despite engaging with dozens of people over the years, I have never had a single person provide me with convincing evidence that it works. Instead, they simply repeat "but of course it works!" similar to your comment.
So, where is the evidence?
By your logic, Scientology and Gwyneth Paltrow are also selling the real deal.
We can measure customer acquisition cost and we do using normal tools like Google Analytics.
We've had days where we pause our campaigns, and we get fewer new customers that day. We've had platforms pause for a period of time because of some issue, and we get fewer new customers over that period of time.
It's one data point. I'm aware of plenty of other companies with similar results to what we see.
The graduate level stuff we run is using decioning engines with adaptive models down to EACH email, switching up not just offers but also ad copy, images, time of delivery etc. that drive VERY complex holdout matrix's which can limit the number of things we can test at once.
One of the executive decisions is when or if the drop customers out of holdout populations. Usually keep the holdout going for X period of time as a standard practice, but if a campaign is showing very strong response we might drop half the holdout if we can still make a read on campaign. We have a formal notice of decision that is published when this happens. Holdouts have alot of opportunity cost, but finance doesn't see those numbers.
Some of the digital exhaust from our primrary product can be used to solve marketing attribution issues for major brands in an area that has until now lacked attribution. Our product beta is just hitting the market.
Does the "universal holdout population" span across all the brands, or do you have a "universal holdout population" for each brand?
For example: The digital services are an ingredient brand and can only really be sold to core product owners so they'll have a differnt holdout as their target population is different. Also that area has a short measurable path and thus different metrics and measurements.
There are some B2B areas that may not have holdouts, they functionally don't do any targeted marketing.
eBay tried what you did and noticed that revenue increase didn’t warrant the ad spending costs required to do so.