Yes! You're the first to mention this.
It's not refractive index itself that's the problem, it's dispersion (roughly, the degree to which refractive index varies across the visual spectrum, described by ‘Abbe number’). We've all seen pictures of a prism splitting a beam of white light into a rainbow — for visual purposes, the less split the better.
Higher-index materials tend to have poorer dispersion, but especially in the mid-range 1.6ish, there are wide variations in quality at the same index. Glass tends to be best, if your prescription is light enough that you can handle the weight. Polycarbonate and acrylic are awful. MR-8 is in the middle, and what I've settled on for recent computer glasses.