You can be unaffiliated (or a member of the confusingly named Independent Party) but you’re right, you then can’t vote in any primary elections.