> each color of light travels at a different speed?

That is indeed correct when light is not traveling through a vacuum. Refraction occurs because of a change in velocity between mediums, and the refractive index is wavelength-dependant (dispersive) in many mediums.