I'll admit that it will definitely end up being more expensive over a more 'Hacker Focused' option like Slicehost. I think the reason that I like Heroku is that it makes my deployment/configuration easier in the same way that Rails makes app development easier. I know that there are guys over there who are implementing the Varnish cache correctly, setting up load balancing, implementing full text indexing, etc. It allows me to focus on my application and not on the thousands of configuration options that I have to think about if I'm the admin.
Now, with that said, once you start to grow your application the costs would probably start to add up rapidly and it may be in your best interest to roll your own, but in the beginning at least I think it's a great option for getting off the ground quickly and correctly.