People have been arrested and worse for taking video of police officers. This is not a threat I need in my daily life.
The legality of recording someone without their express consent also varies by state. I would love to record my daily public life. Laws and technology make that tricky.