You don't have to have physical access to do this. Just reconfigure your network infrastructure.
But yeah, they're probably using The Cloud, because that's web scale.
Even in The Cloud, I'm struggling to see the problem. If it was running on, say, Digital Ocean, I'd just stop the instance, create a snapshot and then launch a new instance from that snapshot (and with a new IP). I'm sure other clouds have similar tools.
Not sure where they host, but they might host it in some shady datacentre which is located in a completely different region than they are with no OOB access to it other than the public IP interfaces which is bad to begin with, but not that uncommon.
... running a financial institution on servers you don't have physical access to, What is the worse that could happen?
Isn't physical access security like OpSec 101?
Maybe it's all in the Cloud?
Yeah, but mostly focused on keeping bad guys out, not making sure the good guys can get in. (E.g. hving your servers in someone elses data center is probably more secure than trying to secure them physically in a startups office, but means getting to them is harder for you as well)
I could see an argument for colocating this type of enterprise in a secure data centre. There are places with 24/7 surveillance and 24/7 armed staff on site and they are going to do the security better than your average group of startup guys.