Here is how I think you could design a more robust (but less fun) experiment:
- Come up with a bunch of topics, write them down on slips of paper, put the paper into a hat
- Each Monday, draw three topics from the hat, send some WhatsApp messages about the first, Messenger messages about the second, and don’t discuss the third. Don’t put the topics back in the hat.
- If you see any ads relating to one of the topics, screenshot them and save screenshots to eg your computer with a bit of the topic
- Separately, record which topic went to which platform
- After doing this for a while, go through the screenshots and (each of you and your wife or ideally other people) give a rating for how well the ad matches the topic. To avoid bias, you shouldn’t know which app saw the topic.
- Now work out average ratings / the distribution across the three products (WhatsApp vs Messenger vs none) and compare