Just because something is legal doesn't mean it is socially acceptable.
Drones are normally really loud and annoying. In addition, the sound tends to radiate because it's above the things that would absorb the sound. If I'm out to enjoy some quiet on a trail, and I've got a gaggle of "Drunk Go-Pro Drone Bros", I'm going to be upset, and I'm going to have words with them (been there, done that).
The threat of a physical altercation is a good check on people who don't seem to get that, yes, you are causing problems for other people.