I think this is one of those areas where "I'm not smart enough" because I have the exact same thoughts as you. Clearly, these rules were implemented by people "smarter than I" to the point that it seems nonsensical to us mere mortals.
The idea of being able to work remotely (especially across international borders) wasn't around when the basics of immigration law and visas were introduced in most places.