Companies like Valve and Epic have already amortized the cost of development of their engines - those are codebases that span 16 years, and both companies had multiple commercial successes. All it took was Unity to give a little nudge.
Now, IMO, the reason for giving so much for free is to keep a player out (Unity) and milk the cow one last time with the eventual royalties.
I think it's clear by now the next generation will be real-time raytracing (engines like Brigade [1]), so I wouldn't doubt they are already working on those, and to be sold in the old model.
But I'm sure ray-traced engine is the killer-app of the Oculus Rift, and companies like Valve and Epic are aware of that gold rush. Once people see ray-tracing on VR they are not going back.
We're in a perfect storm of access to development tools and interest in indie development thanks in large part to Steam and Kickstarter.
In the case of Valve, I think what butters their bread isn't Steam anyway, but the games people end up making with it that Valve get a cut from through Steam.
(Although, bonus points to Brigade for not featuring a mirrored sphere in their demo. Every other raytracing demo that I have ever seen adds these in, as if what we've all been missing from our 3d is mirrored spheres! It's a sure sign of desperation and lack of imagination when raytracers resort to those :)
Maybe we'll stop improving graphics sometime pretty soon and focus on physics, or maybe the game engine vendors will focus on licensing out assets to make game development cheaper instead of the current graphics race. But as long as the best use of GPU power we can come up with is graphics, we will eventually arrive at ray tracing.
That doesn't seem to be the case. It seems that UE4 was a pretty major rewrite, and major development is ongoing: https://trello.com/b/gHooNW9I/ue4-roadmap . Source 2 looks to have been a big effort for Valve too.
BSP, meshes + LOD, lightmaps, occlusion, particle system, terrain editor (since UT 2003)... it's still all there.