First, two important assumption I'm making when I say this (and I feel they are reasonable assumptions). I'm not just talking about bringing a production environment back up in the same or adjacent AZ; I'm talking about true DR, where you're moving regions. I'm also not limiting my discussion to AWS' infrastructure - not with Google, Rackspace, Cloudflare and others in the space as well.
> Lots of data -> use S3 with de-normalized data, or something similar
S3's use case does not match up with many different computing models (hadoop clusters, database tables, state overflowing memory), and moving data within S3 between regions is painful. Also, not all cloud providers have S3.
> Running a VM has 3% overhead in 2016, scalability is much more important than a single node performance
Not when you have a requirement to respond to _all_ requests in under 50ms (such as with an ad broker).
> High security environments are usually payment processing systems
Or HIPPA, or government.
> delaying transactions is ok
Not really. When I worked for Amazon, they were still valuing one second of downtime at around $13k in lost sales. I can't imagine this has gone down.
> Amazon uses DNS for everything, even for datacenter moves. It is usually done within 5 minutes
Amazon also implements their own DNS servers, with some dynamic lookup logic; they are an outlier. Fighting against TTL across the world is a real problem for DR type scenarios.
> EIP (where the public facing IP can be attached to different nodes) makes this a non-issue
EIPs are not only AWS specific, but they can not traverse across regions, and rely on AWS' api being up. This is not historically always the case.
> they extremely rarely have a full region outage, so you can juggle capacity around
Not always. Sometimes, you can. But not always. Some good examples from the past - anytime EBS had issues in us-east-1, the AWS API would be unavailable. When an AZ in us-east-1 went down, the API was overwhelmed and unresponsive for hours afterwards.
> Spinning up instances should not DDOS anything, it is with constant load on the supporting infrastructure.
See above. There's nothing constant about the load when there is an AWS outage; everyone is scrambling to use the APIs to get their sites backup. There's even advice to not depend on ASGs for DR, for the very same reason.
AWS is constantly getting better about this, but they are not the only VPS provider, nor are they themselves immune to outages and downtime which requires DR plans.