Still a bit to expensive for my sideproject ; ) To be honest it seems only big corp can do that kind of stuff. By the way if try to do hyper parameter tuning or some exploration in the architecture it becomes guess 10x or 100x more expensive.
https://aws.amazon.com/ec2/instance-types/p4/
p4d.24xlarge
An alternative is the p3.16xlarge for 8 V100s with 256GB of GPU RAM but you might as well get the A100s since it's only $0.50/hr cheaper