We use on-prem GPU nodes for training deep learning models. Our group estimated the cost vs. cloud and it was significantly cheaper to go on-prem. I can't speak to security-wise and whatever-wise though :)