Programming Languages: What tool is right for which job?

Some languages (e.g. Matlab, Erlang) were designed for specific domains in which they excel, but most were designed to be general-purpose. So, most of those languages are really suitable for most tasks.

There are few things that, for example, one might code in Java, that I would not consider using Python for. The two are very different, of course, but for non-specialized applications, the practical reasons for using one over the other come down to other concerns; ubiquity, your own familiarity, interoperability with your existing infrastructure, etc. Me, just I'll choose the novel general-purpose language out of interest.

Then again, a few languages I've worked with (i.e. PHP, Matlab) deserve thoroughly to be crucified for their crimes against computer science and developers everywhere.

Turns out a query I'd written back when there were 20k votes in the system didn't perform so well now that there are 200k votes in the system. Go figure.

Can ActionScript say that?

It's definitely the case that stock Ruby 1.8 is slow for a great many applications, though. Having said that, there's alternatives, and a lot of the time efficiency of execution is not at the forefront of the objectives for systems written in Ruby.

(The Shootout is not the One True Measure of speed... but it isn't meaningless, either.)

Generally the only thing this really means though is that you wouldn't want to do hard-core computation in pure Perl/Ruby/Python, hence things like NumPy. [1]: http://shootout.alioth.debian.org/u64q/which-programming-lan...

This pretty much rules out e.g. Python and Ruby, because the CPython/CRuby interpreters suck at concurrency (there are other implementations available, which are a little better). You can still do many Python processes, which do lightweight switching between I/O tasks somehow (e.g. with Twisted), but then load balancing between processes becomes a PITA. Node.js is similar, there's only one thread running at once. With Node.js (and twisted) you also have to manually "compile" your code for asynchronous style continuation passing style, something humans suck at but compilers do very well.

Now given the task you said, you could take C and epoll/kqueue and write a small framework yourself, but processing the data you got with C might not be too nice.

Or you could use an interpreter or a compiler that does this automatically for you. This is where the choice of language kicks in. In order for the language implementation to be able to effectively organize parallel execution, it needs some information from the language. C doesn't have any helpful information, which is why it often relies on full OS context switches where all machine state is stored and restored (or a non-portable hack with the stack). Something lisp-y with first class continuations might be helpful, but many lisp implementations don't really do concurrent execution well.

So, for a language that concurrently executes network I/O efficiently while still being high level and fast, my recommendation for your task would be: Haskell. There's years of research and engineering work done on just what you're looking for in Haskell. Just forkIO as much as you wish, write your code in regular sequential imperative style, compile for multithreaded execution (ghci --threaded) and let the compiler and runtime do all the hard work for you.

Erlang might do the job too. If anyone knows of other languages with smart I/O multiplexing and co-operative userland threads or fibers (with concurrent execution!), please reply here!

NOTE: my assumption was that you're actually processing the data you're receiving and you're more or less CPU limited. If you're I/O bound, you don't necessarily need concurrent execution and may get away with Twisted or Ruby fibers or Node.js (without using multiple processes and load balancing or task queues).

Once you know several techniques and several languages, and have started to get a skill at identifying the core bits of a project, the process (or series of questions to answer) becomes:

- what technique will effectively handle to core problems?

- What languages make this technique simple to implement?

and then the real kicker, which in a way comes full circle to the original naive analysis:

- in the best language for $technique, are there any problems with secondary and ancillary problems associated with it? (e.g. i have to do this really fast lookup table of simple operations, C would be great for this, except that the data is irregular and involves a lot of string parsing C sucks at that..., or the central problem here is just a big fold, i'll use Haskell, but it involves a lot of tricky memory optimization to be fast enough...)

Out of this of course arises a meta-technique of learning to partition problems into stages in different languages, which of course has its own problems.

In that context, I disagree with your assesment. It often takes me a week, sometimes more, to evaluate and choose the right solutions for a new product. The techniques that I use when coding matter little during that time.

What _does_ matter: third party services and libraries that will make it easier to build a better product.

However, I agree that choice of language is only tangially significant. It's rarely a "pain point" for me. Gui toolkits, cloud services, hosting solutions, payment solutions, libraries for [you name it], etc. _those_ make up the "tool for the job."

But seriously, i think its great that we can all work in our favorite language, and we can get away with it. I can write a web service in common lisp, and you can use it in your Rails cat picture app, that also uses a brainf*ck script to convert the cat pictures in a suitable format. And your company can also write an iPhone version of your app in objC and a desktop version in C++ for windows and Linux. Man, protocols are awesome :)