Regarding “I go to Church”, the verb to go here doesn’t actually mean movement, it’s a euphemism for “I worship”. Same with “temple”. If you say “I go to a/the church” you are actually stating movement towards a definite or indefinite place.
Again, these are challenging subtleties for language speakers where all nouns are conceptual. And most native speakers can’t explain these exceptions spontaneously