The other parts of ray tracing like shading and so on, are usually just done on the general compute.
The problem with doing so, though, and with GPU physics in general is that it's too high latency to incorporate effectively into a typical game loop. It'll work fine for things that don't impact the world, like particle simulations or hair/cloth physics, but for anything interactive the latency cost tends to kill it. That and also the GPU is usually the visual bottleneck anyway so having it spend power on stuff the half-idle CPU could do adequately isn't a good use of resources.
The faster your fps, the more flickery this ends up looking as you're constantly pulling back the camera or player or whatever a frame after it collided. The lower the fps, the more sluggish and unresponsive it feels.
Alternatively you need to pipeline your entire world state and now you've got some pretty bad input latency