It's ridiculous to compare this to cars. The size category is .5 to 55 pounds. A 55 pound car is called a 'toy' and given to five year olds. You should have just as much right to fly a tiny drone as you have to fly a kite.
That being said, cars are clearly the most dangerous of the three. But they are what we are most dependent on as well.
If you want to keep people out of the places cars are actually supposed to go, but let them be on sidewalks and such, then it's fine to have some restrictions. And likewise let drones fly under 400 feet.
But you shouldn't be banning people from the roads either, that's been a big problem making cities worse over the last century.