So while you could imagine a society where 80% of people believe they aren't religious at all, the United States at least does not appear anywhere near that number.
https://www.pewforum.org/religious-landscape-study/importanc...
(You’re probably right anyways - there’s a reason I put ‘theoretically’ at the beginning of my first post)
It could also be that the (loose, intuitively approximated) mode on some level of granularity (the individual category perceived as most commonly seen) rather than the median is going to frequently be the intuitive yardstick for “average”, rather than median (mean of a ordinal categorical trait with no unique obvious reduction to an interval-level measure has no coherent definition, so its not really a candidate average.)
This also explains a number of political effects, especially once you start looking for it in other high time commitment social activities.