You can quite easily have small rocks wedged in the tread of your shoes which will scratch floors.

Tatami is very common in Japanese houses and I was told you're not even really supposed to wear hard-soled slippers on it.

https://en.wikipedia.org/wiki/Tatami

I’m not saying the shoes would damage the floor but that’s not really true. Both sides of the contact get some wear, even if one of them is softer.

more like(ly) polyurethane, not rubber. however any sand would scratch wooden floors