If you expect to grow your start-up quickly, it makes sense to use a provider that allows you to scale. It is very costly to maintain two separate infrastructures for different providers.
Maybe you're talking about ordering servers by the hundreds, but are successful startups like foursquare doing that? Seems a few a week is what they'd add, and you could do that with most other hosts, without the added complexity of Amazon.
Dedicated hosting and rackspace cloud may be cheaper, but if you expect to be in a situation where failure becomes frequent, you'd better be on EC2.
EBS is not a unique feature, as there are multiple equivalents of EBS, along with higher level methods of replicating data, namely, couchDB, riak, cassandra, etc.
But I think you've answered my question- people use AWS because they don't realize this.
Of course the kicker is that this is in a thread about how a bug in Amazon means that the EBS volumes that were allegedly being backed up, actually weren't.