For games you have to compile release builds almost all the time anyway, otherwise performance wouldn’t be playable.
Even with C++ and heavy stdlib usage it's possible to have debug builds that are only around 3..5 times slower than release builds in C++. And you need that wiggle room anyway to account for lower end CPUs which your game should better be able to run on.
Debug builds are used sometimes, just not most of the time.
Naturally you can abstract them as middleware does it, however it is where tracking performance bugs usually lies.
I've never done it, but I just find it hard to believe the slow down would be that large. Most of the computation is on GPU, and you can set your build up such that you link to libraries built at different compilation optimizations.. and they're likely the ones doing most of the heavy lifting. You're not rebuilding all of the underlying libs b/c you're not typically debugging them.
EDIT:
if you're targeting a console.. why would you not debug using higher end hardware? If anything it's an argument in favor of running on an interpreter with a very high end computer for the majority of development..
This was building the entire game code with the same build options though, we didn't bother to build parts of the code in release mode and parts in debug mode, since debug performance was fast enough for the game to still run in realtime. We also didn't use a big integrated engine like UE, only some specialized middleware libraries that were integrated into the build as source code.
We did spend quite some effort to keep both build time and debug performance under control. The few cases were debug performance became unacceptable were usually caused by 'accidentially exponential' problems.
> Most of the computation is on GPU
Not in our case, the GPU was only used for rendering. All game logic was running on the CPU. IIRC the biggest chunk was pathfinding and visibility/collision/hit checks (e.g. all sorts of NxM situations in the lower levels of the mob AI).