Comment by gwern | Hacker News Reader

Comment by gwern | Hacker News Reader

Parent Full thread

Per-user level doesn't capture the effects I am interested in. Imagine you click on a website and the banner ad makes you subconciously less happy and so you don't submit it to HN for thousands of other users to read; how would randomizing on you and thousands of others' user IDs help? There's no way to link your user ID's randomization to the subsequent visits (or non-visits) of thousands of other people; it just averages everything out to 0 ('500 visitors were exposed to ads, 500 were not; conclusion: ???'), when in fact the real answer is something like '-1000 hits because user yellow_postit was exposed to ads and didn't like them'.

But if you did it on a per-day or per-week or per-month basis, then you could see effects like that: "on ad-free weeks, there are twice as many submissions to Reddit which drives +100 hits per day" etc. Then you don't need to try to track each individual user's activities or reshares - whatever the effects are, they'll show up at the population level.

Isn't this particular test susceptible to confounding effects? Traffic fluctuates day-to-day, week-to-week, and month-to-month, so how can you be sure it was the presence-or-absence of ads and not something else? If you randomize at the visitor level, you are sampling from both high-and-low traffic days, and control for any external fluctuations.

Because you're randomizing at the 2-day level, on average there will be just as many advertising/high-traffic days as advertising/low-traffic days, and as many no-advertising/high-traffic days as no-advertising/low-traffic days. The randomization is unaffected by traffic and uncorrelated with it. The unit of analysis is each day, not each visitor. This is why it has to be run for several months, otherwise you don't wind up with a decent n=50 pairs.

That's the tradeoff here: it lets you look at the totals, but it takes a lot longer than if you randomize per visitor in which case you could finish the test in a few days, often.

The stats went well over my head but as a web analyst I thought maybe you could have asked a simpler question. Pick a segment of your site such as visits from Google Search who landed on your homepage (most likely people who searched "gwern") which should reduce a lot of those spikes.

Subsetting will also increase the variance of each datapoint (consider the extreme case of picking a subset which was 0 or 1 visits per day), so is probably not a win. It's also hard to imagine what subset properly reflects all sources of traffic and so is informative about the total effect of advertising. Search queries definitely is not it.

Reply on news.ycombinator.com