Nice challenges include:
add parallellism. Traversing a tree hierarchy efficiently with SIMD or GPU code isn’t trivial. (I tend to dodge this part of the problem by using intels simd lib Embree)
Or make it spectral (trace with wavelengths instead of simple colored light, to get effects such as rainbows in refractions). This is actually reasonably simple - but then you realize it ruins your hope of parallelizing efficiently as the rays diverge when they refract!
Or implement realistic subsurface scattering. Doing it physically right means your render will never finish, so it’s back to inventing the most clever hacks - just like rasterizing!