Ask yourself where your bottleneck will be, then ask yourself if the OS is going to make any difference there. The answer is very likely no.
No modern OS is going to give you any significant performance/cost advantage over another in serving regular web apps. Most of your trouble will be in setting it up and maintaining it, so go ahead and pick whatever you're most comfortable, or, if you're not comfortable, ask someone who are (and that you trust).
Also, don't pick CentOS, unless you need RH compatibility.
I saw that oracle are really emphasizing how their OS has much higher performance than linux I wondered if its correct or if their benchmarks are good: http://www.oracle.com/us/solutions/performance-scalability/b... http://jp.sun.com/solutions/infra/eda/pdf/solaris-linux.pdf
So I wondered if i'm missing some tens of percents in performance by not choosing Sun Solaris...
If you're running Oracle on Sun hardware, go for Solaris, sure, if for no other reason the support. If you're running "regular" open source software on generic hardware, just use Linux/BSD. There's nothing in Solaris that'd make Postgres faster than Linux or BSD on the same hardware.
OpenSolaris is officially dead. Solaris isn't low cost for sure. FreeBSD gives you a high performing io/network stack plus extras from Solaris world, like DTrace & ZFS, plus a focus of maintainers on just these features you seem to need most. This is contrary to many Linux distros, even commercial ones like RedHat, where the focus seems to drift to different directions over time, as they are trying to please different groups of users/donators/customers.
If you decide Linux anyway, I'd recommend Debian instead of CentOS. Contrary to common belief, RedHat/CentOS distros are "enterprise-oriented" maybe in the aspect of supporting typical Java development, but when it comes to production operations their enterpriseness does not necessarily shine, esp. when things break. YMMV.
These recommendations change a lot if you must deploy Oracle databases on the machines.
Well many people want to give their 2 cents in this thread; while this is excellent, I'd like to point I'm a sysadmin of various Unices since 15 years so I claim to know what I mean and not just promote personal prefs when saying something about OSes (if it was my personal bias I'd use Slackware forever). If you had further questions and wished to ask them privately, you'll find contact links on my profile.
Some questions that I (as a former SysAd) would ask:
1) How many is "many servers"? 10, 100, 1000, 10000?
2) What exactly is "performance for ... multi-cores multi-sockets, high IO & CPU & Memory & Networking"? This basically covers everything ever created. The high-performance for TeraGrid (sci. apps) is very different than E-Trade (financials) and very different from Amazon (e-commerce).
3) Is "low-cost support" $0 and 1 week of searching online forums? Or $100 and a phone call?
4) Do you like "port", "apt", "yum", "emerge", or "make"?
5) Do you need to have multiple versions of the same app available at the same time? E.g., gcc-2.97, gcc-3.2, 3.3.2, and 4.4.1?
6) Did you not know the answer to any of these questions? If so, I highly recommend you hire someone like @davidw says.
On what do you base this? Larry is spending more on SPARC and Solaris than McNealy ever did! It's possible that Solaris will become less suited to (for example) HPC over time but for anything related to commercial data processing, including Java and web serving, expect for it to continue to occupy a leading position among the Unixes.
I don't see it "running into the ground" - people will be using it for many years in the future. However, the supply of people picking it for new projects probably just dried up quite a bit. People will 'grow up' knowing how to work with Linux, because that's what they tried at home, at school, what they learned on the job, and what their friends know. Those network effects mean that it'll mostly be sensible to keep switching to Linux over time, gradually, unless Solaris suddenly sprouts something that makes it a must-have, which I don't see.
Especially since it's easier to find people to work with it, more open, and not driven by a billionaire who wants you over a barrel.
Also, I think that Oracle is quickly putting paid to the sort of environment where innovations like dtrace happen, although I could be wrong.
Oracle's strategy with Solaris is going to be to squeeze every last penny from it, and when it dies it dies. Killing OpenSolaris and making it clear that open source releases of the code base will be done only "at some point" after the commercial release are just the beginning of that.
They will push Solaris in the same places where they currently push Oracle DB. Those places tend to be of very little interest to 99% of the HN croud. Java and web serving are not the kinds of things that justify stratospherically overpriced software licenses...
As for SPARC, I seriously doubt they will pour significant money into it. Trying to compete with x86 is akin to setting $100 bills on fire at this point. They will no doubt keep the Sun brand alive and use it to sell x86 based machines (which is the strategy Sun itself started of course).
I believe they will. And they will also tailor it for Oracle DB and change the licensing plans to make SPARC a "wiser" choice, locking the customer both on SW and HW. The competition does it, so why shouldn't they do it also?
As far as I can figure out the only reason IBM has to keep developing POWER is their mainframe business, which is the king of platform lock-in :)
Pick what you know, though from what you have posted I'm not sure you know either of the systems you mentioned very well. Actually, unless you are a professional systems administrator then even picking what you know may not be good enough. You can setup your own servers if you are running on a low budget, but ideally you would be paying someone to take care of all your administration needs.
So, I assume from your budget you won't be buying your own servers and even if you were interested in buying your own servers you don't have the expertise to set them up properly. You will then be looking at hosting.
You should probably be looking at a hosting provider which offers managed hosting. This will be your support. Don't worry so much about the distribution, just choose whatever that provider has available.
Arguably Solaris is superior to Linux in many ways. If you want to go that route I would suggest Joyent as a provider. Linux is wide open for hosting choices. Find one you like and is within your price range and go with them.
You probably need to start out with a VPS as opposed to a dedicated server. Done right, a dedicated server is generally much more expensive than a VPS. Also, a dedicated server requires a lot of knowledge just to know what to look for.
Your widest choice of offerings for a VPS is linux. Pick one which offers a managed plan and go with whatever that plan ships with. If you have a choice, just go with CentOS. Personally, I prefer Ubuntu, but that's not on your list.
If you are using FibreChannel and SAN attached storage, then Sun might still be a good option. But seriously I'd look at HPUX and AIX as well.
My experiences with OpenSolaris on x86 (non-Sun) hardware has been disappointing. Maybe I had high expectations due to the SPARC experiences.
These days, I only use Ubuntu Server LTS editions. APT and the Debian infrastructure meets my needs well. Also use PostgreSQL rather than Oracle RDBMS.
My two cents, from pretty extensive linux experience and one short Solaris experience: do what aligns more with your IT admin's philosophy.
My feel for Solaris is that everything is managed -- there are daemons to watch daemons whose job it is to make sure certain apps don't go down.
Linux is a bit less monitored -- "Oh, your apache process shouldn't have gone down? Shit you shoulda had something watching it."
Personally, I like the more ad hoc feel to linux. But pick whichever your sysadmin prefers. Solaris, to me, felt like I was being handcuffed.
For me, that would mean OpenSuse, but that seems to be a minority choice these days.
Anyway, I would advise against OpenSolaris after Oracle (paid for Solaris might be OK) - see the recent leaked emails etc. When I use CentOs I am always frustrated that they are using such old software versions (Python 2.4!), but that may just be the clients I have. For stability and security a BSD is not a bad idea, but again check for versions of the software you need.
If you're unfamiliar with both, Linux systems are easier to get used to, in my experience.
Upgrades to the kind of systems Solaris is being used (core systems and databases) tend to stay in the same course, not only because there is a lower risk in staying on the same OS/architecture combo, but also because most of those systems are old and tied to Solaris/SPARC. Unless there is a sufficiently strong reason to switch, most "big" companies prefer to pay the premium (in SW and HW) than assume the risk.
Solaris on commodity funcions is dying rapidly, if one can't already consider it dead.
The notion that Linux is less advanced is bogus. In some aspects Solaris is more advanced, in others Linux has the lead. What matters is what's the best choice for a each task given constraints of cost, reliability and availability of skilled technicians.
Why a year ago? Oracle bought Sun. Sun was very good at creating open, useful platforms, but was very bad at making money off of them. Sun was bought by Oracle. Oracle isn't very good at building software, but is extremely good at selling it and maximizing profit from it. Oracle needs to make back the 7+ billion dollars it made on Sun somehow. How? We have part of the answer in the Google suit: by milking Sun's customer base.
Java has 10 million developers. To make 10 billion (there's a fair bit of risk in buying Sun -- if Oracle wants a 3x return, that's 10 billion each for Java, hardware, and Solaris), it needs to make $1000 per developer. How? I don't know how, but I'll run one possible scenario.
Step 1: Cut off competing JVM implementations with patent and copyright suits. This is already starting to happen.
Step 2: Release new versions of Java in two versions. The first is a free Java Express, which is okay for student use, but has no JVM, and omits many of the more enterprise-centric libraries. The second is a full-fledged version, and costs $1000 a pop. Developers would, of course, free to use the old, bit rotting version as well.
Most Java developers cannot switch platforms -- most are working on enterprise systems with hundreds of thousands or millions of dollars of sunk costs. Java immediately becomes a legacy platform, but even if only 50% of developers buy into it (even if only during the transition period), Oracle makes a cool $5 billion, almost paying for the rest of Sun.
Where does this leave Solaris? I don't know, but I suspect similar things are afoot. Oracle will want to make more money, and Oracle's customers will end up paying more. Oracle is already beginning to move in this direction with the death of OpenSolaris.
Aside from that, Oracle is incapable of attracting good developers. Former Sun employees are leaving as quickly as they can find new jobs. Will Solaris continue to have a technological advantage in the long term without Sun's brainpower? I seriously doubt it.
I know this is a lot of FUD, but with good reason. I am afraid, uncertain, and doubtful about the future of Java and Solaris. While I'm certain that Oracle will continue to develop Solaris so long as it continues to make them money, I don't know what they'll charge for it, or how well they'll develop it. I really wouldn't want to base my business around it.
(A secondary, technological answer is that Solaris advantages are much better suited to the climate of the nineties than of the teens. In the nineties, we tend to rely more on big computers with many CPUs, massive amount of memory, and many hard drives. In the teens, the trend is much more towards big arrays of cheap, unreliable, redundant boxes. Solaris' advantages are much more pronounced for the former than the latter.)