When would you use this kind of setup over Heroku?
1) My scale is significant enough to require at least 2 servers (unless one of them is RDS).
2) I have some background processing.
3) I can commit to reserving an instance for at least a year.
A small ec2 instance running 24x7 will cost me around $40 / month (+ bandwidth) if I reserve up at least a year's worth of usage. That's 1.7GB of RAM and 1 EC2 compute unit. For $163.43 you can get a pretty honking large instance (way cheaper than any competing VPS offering). Add in ELB and RDS and you've got a very scalable setup for very little coin.
I'm not sure how the Heroku dyno or worker performance measures up to EC2 Compute units though, but I'd love to hear from someone who does know.
For me, Chef is a no brainer for bootstrapping and managing my instances.
Heroku is also a no brainer, but as rapind pointed, it's for simpler setups or for getting an app up quickly without a lot of ceremony.
We've deployed to Heroku and eventually moved to EC2, but have kept several apps on Heroku.
For example the Heroku addon for 100MB of memcached is $20 / month... which far exceeds the cost on an EC2 box.
We've been using EC2 for a while now, before most of these plugins were available for Heroku. We're extremely comfortable with the platform and our tools for managing it.
We have most of our customer-facing nodes on m1.large instances since we have a lot of caching (mostly loading all of the data in memory) up front and then the rest is CPU bound.
Scaling Heroku is nice since it's just some knobs you turn, but Chef isn't that much more difficult and it's just a different type of knob.