Ok, you seem to accept that there is an assumption that the data is generated by a distribution with a mean, so start with that. This is not necessarily true: https://en.wikipedia.org/wiki/Cauchy_distribution

I use that only as an example. If you look closer you will find many other assumptions being made as well that are used to derive the actual calculation (for whatever statistical test you choose to look at).