Ah, I'm using a set instead of list so I just always add and then toss remove.