I recently bought a server (xeon 5606 + 24GB Ram + 2 TB storage with hardware raid) for $2k and I am hosting it in a colo for $75. I find this ideal for small services and gives you a lot of performance allowing you to go a long way with a couple of machines before you need to scale. Adding more CPU/Ram/Storage will not add more to your monthly payment since you own the machine. Now of course it's not as convenient as AWS and if a machine goes down you are responsible to get it up again.

Scaling in AWS is a piece of cake and gives you a lot of flexibility but when it comes to performance, especially RDBMS, I find AWS to be far behind. Some will say that you can scale the service to provide good RDBMS performance but that will not avoid the per instance disk IO slowness.