Am I the only one who thinks writing my own Set data structure (based on a map under the hood) isn't crazy at all?
I'd be more worried about having two programmers write two implementations in the same project with subtly different interfaces, and having different parts of the code use each one. This is a cost and maintenance problem, but that's also a social and communication problem common to all projects.
Anyway, have you seen the map[T]struct{} idiom for Go? Decent enough for the basic case.