Really Heroku is a fine product but your paying a substantial cost for the service to manage some of the systems work. I say some because everyone using services like this should have a strategy to exit should some major event happen (prices raise, service becomes unreliable, company is acquired / folds)

Most of the arguments against running a dedicated server was a straw man argument. YES High Availability is expensive if your talking an app that fits on a single server, but once you can get most services running with a N+1 reliability making it substantially cheaper.

>His harddrive / memory / NIC / etc fails?

Most dedicated hosting providers have an SLA to replace. Rackspace is 1 hour. If this SLA isn't suitable you setup a cluster.

>He accidentally runs OS updates, and breaks versions of his packages?

Welcome to backups... you should have them. Yes its expensive, yes its a pain in the arse... but you should have them anyway even if your on heroku. Lets be professionals here.

> He wants to instantly rollback to a previous release of his codebase?

Welcome to code deployment tools... Just because your not on heroku doesn't mean you can't have them

>He wants to add another server to handle incoming HTTP requests?

Its called resource planning... You know what those pretty little graphs are for. Again you should be doing this regardless of the service you are on.

> He needs to spin up a database read slave to handle a high amount of read requests?

See capacity planning.

> His load balancer fails?

Fail to wire modules, Clustered load balancers, virtual load balancers on a cluster... many options

> His Fabric script stops working because he renamed his project or reorganizes his application directory tree?

Testing... read about it... It really helps. You can also mess up any codebase by messing with the directory structure and not testing.

> Or one of an infinite amount of other possible problems?

Ok... and Heroku has a bunch of other problems.

edit: fixed typos and Load balancer quote