I think there's a pretty high chance that tricks could be brought to bear to greatly reduce the costs of what's being done here (by OOMs). There's a lot of handles for tweaking.

In particular the basic crux of this approach is a Monte-Carlo annealing of the score function. In every field of science, Monte-Carlo sampling entails a huge pre-factor cost, and in every case it only reaches it's greatest potential with importance sampling, which hasn't been applied here.

This is basically the 'brute-force' version of some future approach which would replace the diffusion kernel with another process that allows one to avoid sampling huge volumes of function space which are irrelevant for your desired P(X). This would introduce dependence of the training process on the sampling pre-conditioner, but ultimately be required for highest performance.

I suspect that these authors are already thinking of how to use invertible flows to this effect.