In fact, the CLT is remarkably robust to distributional assumptions. Examples where it breaks down (e.g., non-finite variance) are comparatively rare, even if there are "many" of them.
As with all things statistical, judgment is required.
As with all things statistical, judgment is required.
I would beg to differ. They are absolutely not rare. [0] One of the most famous people in probability theory even claimed that it's the ones with finite moments that are rare in practice. (But I couldn't find this quote back, I thought it was Poisson or Laplace)
It's even worse. Many distributions where the CLT does apply, require so many samples for them to actually work, that it does not really apply in practice anymore. Any skew in your data blows up the amount of samples you need to find things like the empirical mean.
[0] Chapter 3.4 https://arxiv.org/ftp/arxiv/papers/2001/2001.10488.pdf
That sounds more like Kolmogorov than like Laplace. Someone more interested in mathematical issues than in practical questions.
Most heavy-tailed distributions make CLT convergence slow, and heavy-tailedness is relatively common.
Many methods in finance and pharmacology, just to give two examples, have had spectacular failures because they ignored that point.