The first question to ask may well be "why does my problem need computation on a commodity cluster?". A massive multi-player spaceship battle may be best served by a HPC environment, using C++ and MPI. There's probably only a limited class of distributed problems that map nicely to a commodity cluster. Several worthy commentators have pointed out that even if the systems programming environment offered the same verbs for both, commodity network latency makes "distributed concurrency" very different from "in-box concurrency". To loosely borrow a term from physics, time delays can break the scaling symmetry of a renormalization group.

A systems architect with a sound background in mathematics may be able to reason out the cluster performance of a social algorithm operating on a Erdos-Renyi random graph. Plodders such as myself will try to prototype. I'll insist that a prototype has business value if only because otherwise I have no way of appearing to be busy.

For prototyping a distributed system on a commodity cluster, my personal preference would be Erlang. As rvirding commented with great insight in another post, Erlang has an OS feel to it. From my limited perspective, Erlang spares me the trouble of knowing Unix and networking (for example: I don't need to know what a TCP port is). Erlang gives me a minimal & consistent set of verbs, and that's all I need for prototyping.