That seems very excessive, though regulation so far has been positive. Not that this is you, but fear of "voyeurism" is often said by people who aren't familiar with drones. I've owned multiple drones and would never consider them a spying tool since they aren't discrete/sound like a flying lawnmower. Additionally, most drones are equipped with wide-angle lenses which means you'd have to get pretty close to spy.

My main worry about drones comes from terrorists using them where crowds of people are gathered (concerts, events, etc), as other people have mentioned.