I'm sure Heoku is a great deal if you don't have the sysadmin skills. My skills are just good enough and I manage my own box. I bought it 5 years ago..4 cores, 8 GBs, 73GB RAID-1...for $5000. I pay $105/month to host it.
This is at least the equivalent in horsepower of what you get from Heroku for $500+ a month (that's being generous). So in a few years, I'm ahead of the curve: $11,300 vs $30,000. Add in the fact that I paid down $5000 in advance as a capital investment and I'm still ahead. You could do this today for under $3000 capital investment and have an even more capable box.
Sure there's risk I'll have a hardware failure (I did have 3 years 24 hours on site warranty, but that's still potential downtime and headaches). But there is also risk that over the course of that time you will want to run things on your server you did not anticipate when you started. It certainly was the case for me. e.g. I started my app in rails then switched to merb. Now I run a second site on the same server with mongodb and solr.
Bottom line, if you have moderate linux skills and know your going to be in "some business" that requires you have a server, buy your own. If not, Heorku seems a fine choice for some.