Yeah, I was thinking about it and also lasers wouldn't work if it was 100% mini-Shapiro delays. I was thinking that maybe there's a small GR component to refraction, but the experiment would be tricky. You'd need to pin the electrons in the refractor down in a strong magentic field and then shoot some high energy photons at it, probably at least X-rays, and see if they still refract. (The high energy photons would be necessary to bring their wavelength closer to the diameter of the nucleus).