Yes, but that doesn't stop the government from taking away rights (See: Plessy v. Ferguson/Brown v. Board of Education, Loving v. Virginia [was needed to stop abridgment of rights], DADT and DOMA, for example.)