I don’t think that it’s always shitty to film people in public without their consent. This is not a black and white issue.
Do you feel that consent should be obtained from everyone who is visible in any video taken outdoors, regardless of whether those people are the intended subject of the video? Do you have a consent problem with surveillance videos? How about videos captured on car dash cameras?
Let’s say you are at a tourist attraction, capturing a video of the attraction. Another visitor asks that you stop filming, because they don’t want to be in your video. Dickish to continue? I’m not so sure.
It’s easy to say “don’t be a dick,” but that statement sidesteps the reality that reasonable people can (and do) disagree about what that means.