Sets don't need to be native in languages that are both fast and have a proper type system. For example, rust and OCaml have a standard set in their stdlib, but it's not builtin.
Of course, go lacking generics makes it impossible to implement a decent set type in it.