In fact that's sort of the endgame for drones. People don't control the drones, they tell them what to do/where to go. The autopilot figures out how to complete that task.
Counterpoint, which confirms your comment: I spoke with a drone designer (DoD contract, so I won't name his company) who said he won a Red Team exercise (quaint terminology, that) against the US Armed Forces by using superior fly-without-GPS algorithm to get through a GPS suppression countermeasure. So... yeah, it's a thing.
That said, trying to do dead reckoning from a series of noisy sensor measurements isn't easy. Bias drift, quantization roundoff error (which integrates to a random walk if you're sensing rate), and filter decoherence are all real issues.
This is a cat n mouse game where the drone creator always has an advantage.
http://news.utexas.edu/2013/07/29/ut-austin-researchers-succ...
I'm sure it's not trivial, but drone defence based completely on signal interception seems straightforward to defeat.