If for example, some cloud cover shifts and we lose connection with our copter for just a few seconds, a lot can happen. Even though we built redundancies into our software (if no connection, hold position), those few seconds of lost control could cause some catastrophic problems. I think this is really the biggest issue surrounding drone flight; it's not the question of "Can we build autonomous drones?", and more of "Can we constantly guarantee control over our drones?"
Still, misuse of plenty of things can have a probability of killing someone. Automobiles are the most obvious example. Bicycles are a great example, since they can be had for similar prices and similar restrictions (i.e. basically none) as quadcopters. Even a baseball propelled from a bat can kill someone at least as easily (and I would argue much more easily) as a DJI Phantom.
http://www.wtsp.com/story/news/local/2014/06/20/students-tam...
Luckily, there are considerable safety measures so far.