Bootstrap resampling is such a black magic thing
Sidney Siegel, {\it Nonparametric Statistics for the Behavioral Sciences,\/} McGraw-Hill, New York, 1956.\ \
Right, there was already a good book on such tests over 50 years ago.
Can also justify it with an independence, identically distributed assumption. But a weaker assumption of exchangeability can also work -- I published a paper with that.
The broad idea of such a statistical hypothesis test is to decide on the null hypothesis, null as in no effect (if looking for an effect, then want to reject the null hypothesis of no effect) and to make assumptions to permit calculating the probability of what you observe. If that probability is way too small then reject the null hypothesis and conclude that there was an effect. Right, it's fishy.
Siegel, S., & Castellan, N. J. (1988). Nonparametric statistics for the behavioral sciences (2nd ed.) New York: McGraw-Hill.
2) The fact that “substitute the data for the population distribution” both works and is sometimes provably better than other more sensible approaches is a little mind blowing.
Most things called the bootstrap feel like cheating, ie “this part seems hard, let’s do the easiest thing possible instead and hope it works.”
I agree "bootstrap" has expanded a bit in meaning but I think it's basically the same idea.
I used to think it was cheating but have realized there is a cost to it, which is replication. For many things it's just impractical in terms of computation time. So although it's great, it requires a lot.