I think it's an language thing. Shoes I would wear for indoor activities like the gym I'd generally refer to as "gym shoes". Any other form of indoor shoes I'd just refer to as shoes. Indoor shoes to me immediately brings up inside the house, especially because that's how OP used "Indoor shoes" in context

Obviously people wear shoes indoors outside of the home. (The norm for a gym would presumably be a pair of sneakers.)

In the US, my experience is that people will sometimes/often take off shoes when they come into a home especially if they're going to track in snow, mud, etc. But it's certainly nothing like a universal rule and probably isn't even the norm.