In what way are they potential civilization destroyers?
If you get rid of the reaction mass requirement, accelerating a superheavy object to a horrendous velocity takes time instead of a steady stream of resources.
Here's more from someone with way more citations than me: https://www.projectrho.com/public_html/rocket/reactionlessdr...
And the people in charge of not having done so are pretty stupid in other areas, so there's that.