You can get a single beefy box and run apache/nginx/lighttpd(what???), mysql, memcached and anything else all on one system and there are certainly people that make it work... but given the number of times I have been called in to fix these kinds of scenarios and make them work properly, I can say from experience that if you expect any amount of traffic you invest in the ability to abstract these various services away from a single machine with ease (or just have them separate to start with).
Perhaps if you do not have a lot of operating capital to begin with, virtualization/cloud options would be a good place to start. You can have with virtualization that single beefy machine and still separate out the various pieces of the infrastructure and then allow to add additional virtualization hosts to the cluster usually with a click or two.
I am happy to answer other questions to scaling (within reason) if you post in response or mail me. I'd rather you start things in a good direction and give away free advice than see a company fail because of architecture decisions which hobble expansion in the future!
Not sure that is the best way to go, but that's how we'll roll.
Also, how are 3 Heroku dynos less than $1/month? 2 non-free dynos are $40 right off the bat.
Andrew is right, we set up three different apps and so each is free.
This is definitely not a one-size-fits-all sort of question!
At that kind of server core count, it turns out that we can keep 8-12 disks busy, when crawling and indexing. For serving results, it turns out that we can keep 2 ssds busy, and, 96 gigs of ram seems to be the sweet spot.
The one thing we don't stress so much is the network. We're very conscious of locality, and we can get away with having 1 gigabit to all the nodes. We use beefy 10 gig switches in the middle to give high bisection.
Having been featured on a few large sites, it was enough. We spun up 3 dynos on Heroku for the front end, and in the first week of launch we had gone through a few hundred GBs on bandwidth (not including static assets hosted on Cloudfront).
Each app and service is different, more info about what yours does and it's known bottlenecks would greatly influence answers.
Naturally, if your startup is a backup service or something else where having as close to 100% uptime as possible is a truly key feature, then it's probably not in your best interests to rely on Heroku/AWS.
If you're running a heart-transplant registry, you probably need 10 nines, if you're running a cat dating site, you probably need less. Sure everyone wants massive uptime, but few are willing (or even need) to take the steps needed.
Even Gmail goes down sometimes.
When they first launched, their only server was a Mac Mini in a second-tier colo that was run by a friend of theirs. After about six months, they did their first major server upgrade - buying an external USB hard drive for more storage. They bought another two (and a USB hub) a month or so later. This lasted them another four months, until the got to the point where the USB bus was being constantly saturated. Only then did they move to some more enterprisey hardware.
Moral of their story was to use only as much hardware as you need now (while having plans on how to scale up in the short- and medium-term which don't cost you money now), and to worry more about getting users and building a product. Worry about scaling after you've got customers (and data that will show you actual use patterns), as opposed to paying for a complex scaling infrastructure before you've got any users.
Are you saving HD video files? Then you will probably need a data store connected to your web server.
Are you just allowing people to post text, small images and such? In that case, 1 server could easily cut it.
With proper software configuration, and a decent hardware config, you can go quite a long way with 1 machine.
Can you provide any details? Maybe someone can help out with your specific question, since the general question you've asked is quite difficult to give a proper answer to.
I've been looking for alternatives, preferably hybrid dedicated/cloud with good RAM/IO for app/cache/db and cloud based storage. Rackspace seems to offer something like this, but at a higher price. Any other suggestions?
Full disclosure: crazy CloudSigma customer running FreeBSD with ZFS in the cloud :)
I migrated to Rackspace cloud, due to the Slicehost acquisition and the first month bill was hard to swallow.