This is a tricky one; maintaining people's safety/ privacy while giving drone pilots 'freedom' is quite a balance. For instance, in the UK you must maintain a physical line of sight with the drone at all times, this makes flying winged craft virtually pointless.

Right now the geo-fencing of where you can fly is very rigid, with no provisions for altitude and distance, this could be improved to include glide-paths near airports for instance. It could also be tiered depending on the pilot's qualifications (similar to scuba diving) and if the drone is fitted with a transponder, or if they have submitted a flight plan. With these, there's no reason a drone pilot can't have similar freedom as a helicopter pilot.

I think most of these problems stem from technology being ahead of legislation, as always.