But for a game - this means dropping the frame rate, making it visually unappealing.
That's why Ray-casting would never pick up for real-time games (Unreal most definitely is not doing any ray-casting, I'm just giving it as a an extreme example).
The other problem is content creation. Current games are 7-15gb of compressed data - half of them textures, rest models, animations, etc. and fitting that in memory (and loading it in memory).
Even with the fastest drives, you end up spending a lot of time loading (or streaming).
Then if that thing looks so real, you start feeling that something ain't right if your gun can't really destroy every piece of a building... And later allowing that in the engine, makes it worse as not much pre-processed data can be reused (static lighting, bsp geometry, etc.). Or you do it (somehow) real-time.
Also this complicates the game "AI" - there is no real AI in an FPS shooter - it's simply too god damn hard. Just think about the cover system, and how laughable it would be if you have destructible surfaces, and you can see the "AI" guy trying to hide behind thinking it was a fine cover point.
Less realism, more constrained world, and better gameplay are the key components to good gaming... Not fancy graphics all the way :)