Actually, if you don't need the entries to be sorted, the sparse_hash_set from https://github.com/greg7mdp/sparsepp is much faster than std::set or a sorted vector (except for iteration) and barely uses more memory than the sorted vector.
Well, you certainly could, but unordered_set will use significantly more memory than the sorted vector or sparse_hash_set, and is actually slower than sparse_hash_set.