TBH my concern is less for the fate of people who choose to fly planes or drones - they are the ones making the choice so if they hurt themselves that sucks, but none of my business. However I am concerned for the innocent bystanders who might be under them, who did not choose to risk death or injury by an aircraft, drone or otherwise.
The template for this ofc is how we handle (or don't handle) the danger posed by people inside cars, to people outside cars. In aviation we will do a lot to avoid even one death, in the air or on the ground. But in cars we mostly don't give a crap. It took decades for drunk driving to become unacceptable, but outside of that, we are still pretty ok with death by car. The only survivor of the collision just has to claim the person outside the vehicle "darted" and we all shrug and move on.
I would just love it if we could apply lessons of the past to new technology. Instead we just hand wave it all away. Then in a few decades, if enough people die and their surviving loved ones invest enough time & energy, maybe we'll make a few tweaks to the formula.
So no, we don't need 5nm separations for 20 lb drones. But we do need some sort of structure that recognizes the people under them didn't sign up to be part of the beta test. For bonus points it should also recognize that externalities exist and should be priced but I am not holding my breath.