Simply, the scale of observation matters. Making observations at scale is categorically different than manual observations. And yes, there is a spectrum. But the important thing is that there is a difference between the ends of that spectrum.

The solution is to recognize that ease of observation interacts with expectation of privacy and legislate what can be done at each point on the spectrum. I have no expectation that someone won't take a picture with me in the background while I'm in public, but I would find it jarring to be filmed at every public location I went, have that video indexed to my name in a database, and have all my behaviors tagged. You write the law so that the latter thing is illegal and the former thing isn't. When there's a dispute about what's illegal, you have it resolved by the courts like every other law.