From their "How It Works" section:

    "A dyno is roughly equivalent to a Mongrel, except that 
    dynos are spread across multiple servers, so 
    performance in most cases is greater.

    4 dynos are equivalent to the compute power of one 
    CPU-core on other systems."

A dyno is different from the common RAM+Bandwidth VPS slice, so it's difficult to make a comparison. (I'm going to do it anyway...) I would say that a 4 dyno machine, with everything else dialed down to the lowest value (so, free), you're roughly equivalent to Slicehost's 1024MB slice. (That's roughly $108/mo at Heroku, $70 at Slicehost.) You've got more storage space (so, the db can grow) on Slicehost, but you have to setup everything yourself, and adding power will take longer than a couple seconds.

It should be stressed that Heroku makes everything really simple. You can save a lot of time and energy if you've got the money to get started with Heroku up front.

I like setting up and tweaking a VPS, though. I haven't seen any benchmarks to show that Heroku with 4 dynos performs as well or better than 4 mongrels, or Apache + mod_rails on the 1024 slice, etc.

I think it's a bit expensive, but, man, it's really cool, too.