* Animoto was running several thousands of machines in 2008 (http://bit.ly/EDLtt)
* Litmus runs 400 servers (http://bit.ly/d7Hc7y)
* 99Designs runs entirely on EC2 (http://bit.ly/aotKgg)
I've personally had long-running EC2 instances with uptimes in years. You could do it cheaper in terms of hardware, but at the cost of wasting time at the colo while you could be building cool shit.
We [at 99 Designs] actually don't have a huge number of instances, although we are steadily adding more and more.
We have at present:
2 x Caching Proxies (Large)
8 x App Servers (Large)
3 x DB Servers (High-CPU X-Large)
2 x Workers (Large)
We also use RightScale for management and are at present looking at using server arrays to scale up and down the App Server layer to deal with load spikes.
The biggest thing that reduced our need for lots of servers was moving as much functionality to asynchronous tasks as possible, which are queued on each appserver via Beanstalk queues and then consumed by the worker servers. Additionally we have as many pages http cache-able as we can, which are served directly from the squid cache at the front of the app.
Hope that helps!
Cheers, Lachlan Donald CTO, 99designs
To add to the large-environment roll call, here are the persistent server counts of some 100% cloud-hosted companies:
- Bizo: 100+ instances (I work here)
- Reddit: 100(?) instances ("256 Virtual CPUs" http://us.pycon.org/2010/conference/schedule/event/148/) - HNer jedberg runs this
- ShareThis: 250 instances (I worked here)
Longest-running instance I've had is at two years and still going strong.
25 c1.xlarge
26 m1.large
22 m1.xlarge