$135 / month buys you two 1.5GB Linode instances or one 2GB Slicehost instance (I use Slicehost but they seem to be much more expensive compared to Linode).
The reason I see renting VPSs, especially at the beginning, as a smart thing is that you are postponing the actual hardware costs towards a later time, when you are (presumably) profitable. And if you decide to cancel the whole thing, it's just a matter of shutting down the instances and that's it. It just seems much more convenient.
Now, if you already have the hardware and you trust the electricity/bandwidth provider you could self-host. But investing time and money in hardware and self-hosting might just be the one thing that stops you from doing the right job software-wise. (Of course, unless your service depends precisely on the fine-tuned hardware that you might manage yourself, but I don't think you are in this situation -- it's just a cost discussion).