I assume you'd also have felt this way about e.g. segregation? An equally controversial issue in its time?
You can make really good arguments from our established principles for and against most controversial issues. There is a clear method to add rights to the Constitution. I would say that happened for segregation and it took awhile to be enacted. If someone wants to change something for abortion or gun control let's amend the founding document otherwise let people live within the rules we have set.