You do have that right, because you can switch jobs at will (unlike that person), in case you don't want to work with someone purely due to their past and not something they are doing at your workplace.
I don't think we should be blocking people (who have already paid their dues for the crimes they have committed in the past) from getting a job solely for their past, just because you might be uncomfortable with that.
It isn't like you are trapped, you always have an option to leave. The person who paid their dues to the prison system doesn't have any options if they become unhireable solely due to people's prejudices about their past. And at that point, what other option do they have other than turn their head towards crime again? Rent and bills aren't gonna pay themselves.
Not the best solution, but I don't think that denying someone a job based on your discomfort about their distant past is reasonable at all. Of course, as long as they weren't in-and-out of jail multiple times for the same crimes, because that's an indication that they are likely to re-commit.