But that's no different than a random photographer taking your picture, or a grandmother noticing you come and go through a window?
I agree "privacy" isn't the perfect word, but seems to be the best one we have to describe what we're losing as surveillance expands.
You are sitting on a park bench. It would be considered an invasion of privacy if someone opened your bag and took out a diary or note book and started reading it.
If you were meeting a friend in a public space and someone later confronted you about something you had said and produced a recording, it would probably make some portion of people uncomfortable.
But pointing it directly across the street at a particular house and leaving it there for days, weeks, months on end would probably attract some kind of negative attention from the homeowner, especially if you told them you were live-streaming to the internet (password-protected/encrypted or not).
I'm no lawyer, but, to me, that feels closer to sitting outside their house for days like a detective in an 80's movie than photographing public happenings.