Hacker News Reader

Comment by pandog | Hacker News Reader

A lot of the time you are running experiments and seeing how the data changes, not looking at the data in a vacuum.

Many of the example critiques here don't apply so much when looking at the changes in data:

- if I got 50% more hits on my site this week vs last week, that's meaningful despite 36% of people blocking ads

- if my open rate doubled when I changed my email subject, also meaningful

The other examples are hard to pick holes in as they simply say "Z is a lie", but I can be looking at multiple data sources to decide how much of a lie Z is.

View on HN

Having been at smaller companies without the data, tooling, discipline, and resourcing to conduct viable experiments, and then being at a company that is actually one of the best in the world at it and building solutions for these problems at scale for marketing teams, so many problems come back to the human element in how data is tracked, how teams collaborate or fail to which can lead to pollution and dirty data, and how decisions are made as the moment trade-offs need to be considered it becomes personal and enters messy human relationship territory.

But ultimately from a pure "did this work or not" standpoint you are right. Incrementality experiments are the gold standard.

I think the key point there is "viable experiments", most digital marketing people I've encountered are not into things like deseasoning, controlling, heck, you say "regression" and they blank. Not accounting for trends and covariates would give a misleading impressing even with high quality data.

The problem is that number optimization easily leads you down the path of optimizing the wrong thing.

Need more people to click on your email? Easy. New subject line: “you’re gonna die soon”

Need more people to click on your ad? Again, easy. Have you considered boobs and butts?

These aren’t just made-up either. We’ve all seen those weird ass mobile game ads. Vague, maybe a little bit of fetish, sexual, violent, ominous. Those ads work. 100% they settled on those ads by optimizing their numbers.

Following this kind of process blindly and optimising for it leads to a terrible product. However it can lead to more revenue in the short term.

It leads to more tracking and higher marketing budgets. And sometimes it leads to more revenue, but since the marketing department is in charge of the statistics, well, did they really?

Reply on news.ycombinator.com