Maybe if we could create a parallel universe at a specific point, and have one side try one thing and the other side do another, but there are huge ethical concerns from that as well.
Maybe if we could create a parallel universe at a specific point, and have one side try one thing and the other side do another, but there are huge ethical concerns from that as well.
Most social science experiments do not initially handle random samples.
The famous ones like the Milgram experiments and others conducted by the same researcher do. If I have my facts correct, he used to mail out letters to people randomly chosen from the phonebook to ask them to participate in his study, then travel to their locales to study them. The Milgram experiment itself was redone multiple times, using different population segments---students, random people in the city, all women, all men, etc.
As an anecdote of how poorly correlated a social experiment can be to real life---in my university, econ & social science students were required to take at least 10 experiments per semester. Often we would go there, realize the material tested was identical to what we had studied in class then behave like 'rational market actors' in order to get the best result (econ experiments used small sums of real money, so you'd be paid at the end based on performance). This isn't anything like how a random person would have behaved.
It's a specific case of the cumulative binomial distribution which is what you might look up if you want to know about the case where the probability of people liking X in general isn't 50% but something else, or the number of samples isn't 21, or the number of sampled people liking X isn't 0.