Do you know about std::unique? Generally sets are a poor tool for removing duplicates.
If so, that sounds fairly limiting to me (i.e. potential extra storage - of duplicates, sorting required, etc).
But yes, you either need to stop occasionally and remove the duplicates or use a set, if the storage is prohibitive.
Otherwise in terms of performance, sorting and unique is much faster than inserting one element at a time in a set.