I've decided to use erlang for my next project. I'm in need of a solution that has massive parallelism (thousands of threads) and works over a geographically distributed system. From what I've read, erlang is the way to go.
The syntax is kind of wonky, but it only took me about a day to get used to. Erlang is my first functional language though, so I'm still trying to get into the swing of functional thinking.
On any given node there were between 1,000 and 3,000 processes (threads) handling requests and internal jobs. I had supervisor trees for ensuring that everything kept running. All that worked like a charm.
Most of the issues I ran into were with getting the host infrastructure (Linux in my case) configured correctly. Things like number of available socket handles etc. Erlang dies unceremoniously when it runs out of system resources.
Mnesia is really nice. My database isn't very big (~500K objects), but still needed to be redundant and highly available. Setting all that was a breeze compared to most SQL dbs.
Yaws is nice, in that it's a full featured web server, but I'd probably also look at Mochiweb (http://bob.pythonmac.org/archives/2007/11/07/mochiweb-anothe...).
Given that you have a requirement for "massive parallelism" -- high concurrency, Erlang is a very good option. I also believe Scala (http://debasishg.blogspot.com/2006/11/threadless-concurrency...) and Clojure (http://clojure.org/concurrent_programming) offer reasonable alternate approaches.
Good luck.
Lots of people use erlang for mission-critical systems. Implemented correctly, it is likely to be a good choice.