Giving the capability of aerial surveillance to anyone who can afford a drone would mean the end of outdoor privacy, even in your own backyard, no matter how tall your fence is. Say goodbye to the sun.
It's my understanding that in the US, planes today have to maintain a certain distance from people and structures (500 ft?).
I wonder if you saw a drone 50 feet above your house if you would be in your right to, say, get out your slingshot and defend yourself. Even if the done weren't hostile, a drone helicopter could be dangerous if it runs out of energy and falls on your head.
Although, 500 feet is not far at all. I've taken a friend up with a telephoto lens and a mid-range SLR and even at 1,000 feet there's not much privacy.