I have multiple(big,expensive) drones. I program them too.
I am not too worried about them, they are far safer than a real helicopter.
"One GPS glitch" ?? When something fails on a drone you could detect it very easily. If GPS fails you have barometric sensors, you have accelerometers, you have compass, you have gyroscopes, you have infrared sensors for horizon.
When something fails you actually reverse and disconnect the motors automatically. Now it is just 1 or 2 kilos falling down. Mine have small parachutes.
Like humans you could add redundancy to those machines, and the probability of all of them failing at the same time is almost zero.
Normal toy quadcopters weight 100 grams and you can stop the blades with your hands (or you neck) without harm.