I roughly agree with your diagnosis about why libertarianism is considered right-wing but your example elides some pertinent details, e.g. segregation was sometimes (often?) imposed on private parties by the relevant governments.