I can imagine the overhead for ints is large. But how often do you sort pure ints? A more realistic test would be some objects sorted on a field value.
[10K elements] BenchmarkLibPotato-4 10000 2166569 ns/op
BenchmarkSpecPotato-4 10000 1020842 ns/op
[10M elements] BenchmarkLibPotato-4 3 3829722562 ns/op
BenchmarkSpecPotato-4 10 1802874882 ns/op