Because having to keep track of who might have noticed you on the map—then explaining your choices to them—is draining and makes you feel like you can’t make a basic personal choice without justifying it.
Because sometimes people can be unreasonable. (Bad day, drunk, generally difficult personality, etc.) The more people you add to your circle, the more you are likely to run into this.
Because it’s none of their business. You are not owed my time just because I’m nearby. That’s not a healthy boundary to have. Location sharing encourages “boundary creep” that forces you to more frequently justify and reinforce your personal boundaries, adding friction to the relationship.