Regarding the second though, I truly believe that the cost + time savings are tremendous from using their platform vs administering your own early on in a startup product lifecycle.
Except... reality is harsh and ugly.
You want the guy with the sysadmin foo on your team from as early as possible. Because if your thinking goes along the lines you just described then clearly you don't have him yet, nor someone who told you what slippery slope you're about to tie yourself to.
If you have the admin guy then he will find you a cost effective platform to start out with quite effortlessly. Which might be heroku in some cases, but is usually just a bunch of rented or virtual servers that he sets up over a weekend. In the latter case it might actually cost a few bucks more initially than the "heroku free plan" - but he'll explain to you in kind words why he thinks it's worth that in the midterm. And he'll probably be right.
If you don't have him, then heroku can't save you. Heroku will run your stuff for a while just until things get interesting and worthwhile load starts to build. That's the point where the sum of your mistakes brings it to its knees. Due to basic mistakes usually. Related to simple things like file descriptor limits, stuff about TCP connections, or a basic understanding of how disk i/o works and how to craft the SQL to make it not hurt so much.
At that point all you can hope for is that you're already profitable enough to afford not only that guy you skipped on initially, but rather his bigger, hairy brother, which is the only kind generally willing [and able] to take on "search & rescue" gigs. For an adequate sum.
At this point your friendly "pay-later" route has turned into a nasty "pay 10-20x now" roadblock. In an "Insert coin to continue" sort of way.
This is not about what heroku can or can not do. This is about what kind of skills you need on the team for a web-startup. Don't skip on the admin. Or rather, don't skip on at least one guy who has been doing the admin thing on a real live app for a bit and also during some not-so-good times.
What if you are not a making these mistakes, how is Heroku (or another service like it) going to bring you to a harsh and ugly reality?
It seems Heroku is fine if I don't want to monitor my own sever all the time and deal with the maintenance that goes along with that.
Well, if you are running a startup from scratch - you ARE making these "mistakes" because you certainly aren't focused on the minutia of performance tuning and significant scalability concerns.
Most startups at this point are still trying to get the aircraft off the ground - much less thinking about switching on the auto-pilot for cruise.
With that, this is really, really good advice.
In sysadmin-land, all my pain comes from software that was never written with management in mind. Assumptions like 'all TCP ports are open, all the time, between all servers', 'we can put things wherever we want in the filesystem', and 'the server should be configured just like my local workstation' make upgrades and deployment a nightmare, and are nominally difficult to change in a codebase with more than a few iterations underneath it.
Oh, and my other favorite pet peeves: Applications that provide no easy way to verify whether or not they are, in fact, up or down, and having debugging information dumped into the logs marked as errors, rather than as debug messages.
Keep in mind that these won't bite you in the ass initially, when you're only on a single server, or on a small, tightly controlled cluster of boxes. They'll torpedo you when you need to scale up, and you'll get a second shot in the boilers if you ever need meet any number of regulatory standards for various industries (PCI, HIPAA, etc.).
Fixing these early-on is easy, and you can enforce scaling-and-deployment friendly coding practices through automated unit testing and CI. Won't even really cost your coding team any extra time. Fixing them down the road, after you've got a few thousand live customers and a big pile of codebase to dig around in, is... difficult, at best.
One good thing about Heroku, though, is that you essentially have to do a lot of these things properly from the beginning because they enforce a shared nothing, tiered architecture. For instance, you can't write to the filesystem, you can't set up some poorly secured e-mail server, you can't have arbitrary daemons running. In general, every process that isn't core to the web stack must be executed in a completely different tier.
Obviously you can still shoot yourself in the foot with poorly written SQL and other things, but a lot of the "sys-admin" type mistakes are avoided by enforcing best practices when you start with Heroku.
I seem to mention this frequently, but I'll mention it again: there are many business models for which you can reach any level of "worthwhile" business results without ever taxing your setup. The whole notion of "Make a service on a shoestring, get popular, race against time to keep up with the volumes of traffic possibly crushing you before you can raise money to rearchitect the service" is the exception, not the law of nature.