To avoid the religion argument, let's switch it to a shirt with a big marijuana leaf on it.

In that case, yes. Unless the person is a voter in the act of voting.

If the colors/hat had become a signalling mechanism by the political movement and were being used to make a political statement, I would say wearing them would be in bad taste. Especially if a member of law enforcement or the military was doing it.

More directly, if the Rastafarian style hat was a prominent display in several recent political rallies for a single party, I think it should be absolutely called into equal question.

Rastafarianism is a religion. I assume religious attire is not restricted.