If you like playing with ridiculous amounts of hardware, definitely try out TPU VMs with TFRC: https://blog.gpt4.org/jaxtpu

You can create up to 5 TPU v3-8's in europe-west4-a. Each core is roughly equivalent to a V100, so it's sort of like having 5 p3.16xlarge's. (The TPU has 96 CPUs and 330GB of system RAM, whereas a p3.16xlarge has 64 CPUs and 488GB of system RAM according to https://aws.amazon.com/ec2/instance-types/p3/).

The best part IMO is that you don't have to figure out how to link the cores together. JAX does that for you automatically. You get 100GigE from TPU to TPU too, which you can also leverage: https://twitter.com/theshawwn/status/1406171487988498433