Here's my line of thinking: Right now, we're essentially conditioned to the premise that when something is sailing through the air, the general gut instinct for what that UFO might be is usually limited and pretty reliable. There's a certain predictability to mere birds, baseballs, kites, balloons, et c...
When it isn't one of those ordinary things, that certain predictability pretty much goes out the window. I don't find the idea of adding remote control aircraft to my list of normative conditioned responses to flying things particularly appealing.
In my gut, I have an instinct to smash mysterious flying things that I perceive as threatening, regardless of how rational that perception is. When a bug flies too close to me (and especially my ears), my reptilian brain takes over, and demands swift, brutal action. It's that same sort of fear that Hitchcock tapped into with The Birds.
I grew up around farm fields with crop dusters, and those didn't bother me growing up, because I could see the guy inside the bubble dome cockpit, and wave to him, and he'd wave back sometimes, right before dive-bombing potato beetles with insecticide.
But take away the pilot from that equation, and I don't feel the same. The degree of investment from my conscience changes, and the air craft enters a new realm of human perception. It's private property, sure, but it's set in motion with uncertain intent.
When people make sport of these unmanned flying things (and they certainly have: http://www.theguardian.com/world/2013/jul/18/colorado-town-p...), I understand the sentiment of that stance. It's something mysterious flying over your head, and at a personal level, that taps into some specific emotions that aren't easy to part with.