The spread of data in the backing store usually hits similar penalties for hash tables, though. Especially in benchmark environments.
10, in particular, just feels odd. That many items almost surely fits in a single cache line.
Granted, if you are storing compound data such that each item is a reference elsewhere anyway, I can see this. Speed is likely dominated by the comparison procedures at those sizes.