I'm not the nicest person in the world and if someone flew a drone over my garden I would ignore it but if they would hover there long enough for me to be irritated I just might attempt to bring it down.
That would at least force the owner to identify themselves and I can afford to replace the drone if ordered to do so by a judge.