I worked as a model quant for a year a couple of years ago in a bank in the team responsible for developing and implementing the derivatives pricing models. Everything was written in (a lot of) C++, but I suspected and still suspect that you could get the same performance by leveraging one of the off-the-shelf deep learning libraries with a lot less code. Maybe you would have to write a few central components in C++ and assembly to get the exact same performance. I am glad to see someone actually following through on that idea.
Backtesting a regular single symbol model on CPU is often not cheap, especially at higher frequencies. Now multiply that by 50 correlated instruments, each having say, 1500-3000 strikes+sides+expiries in their respective option markets, and the value of GPU offload should become obvious. Backtesting dispersion trades on the S&P 500 using just 5 strikes per underlying, or any other scenario where the evolution of the volatility surface is important would already potentially involve >25,000 individual option markets
All professionals backtest, often numerous times per day
Yes.
Counterparty credit risk calculations are done as Monte Carlo simulations far into the future and on multiple scenarios on top of the baseline. It easily grows to 10^8 times of the number of valuations in the baseline scenario at present time (which itself can be millions of positions).