Haskell is 4.2x as good as Erlang at lightweight concurrency. Give it some love. | Hacker News Reader

Haskell is 4.2x as good as Erlang at lightweight concurrency. Give it some love.

shootout.alioth.debian.org

125 pointsdfranke35 comments

Go build something with it, like a phone switch, then we'll talk.

Kidding aside, I think the fact that Erlang got thrown into the production ring so early is perhaps a hindrance to its growth in some ways. They can't just go fiddling with the language at this point without some serious breakage, whereas newer languages like Scala and Clojure would probably not create major problems should they go through a few incompatible versions. Yes, people are using them in production, but not for things like phone switches!

Haskell has amazing potential... But it's hard to argue with Erlang (running Ericsson's 24/7 real-money business), OCaml (running Jane Street's 24/7 real-money business) and so on. Is there anyone with serious performance and reliability requirements that's bet the farm on Haskell (yet)?

http://www.galois.com/ is a mostly haskell shop.

It's definitely an interesting language; but I would see it as more competitive with C, C++ or Java as a language for building high reliability components.

I'm starting my grad career under one of the co-founders of Galois (Andy Gill), and one of his projects while there was to build some kind of VHDL generator tool for encryption (I only saw a high level description in a presentation, so I may not be entirely accurate describing it as such).

It created layouts that performed over 10% faster than the best hand-tuned layouts at the time. My understanding is that it was for the DoD.

Haskell needs an equivalent to OTP and cross-process/cross-server message passing before it can seriously replace Erlang.

I say this as someone who would like to see that happen (though I have no resources to do it myself). But until that happens, Erlang has a niche that Haskell can't fill; you can toss robust, multi-machine-scalable servers together in Erlang way faster than you can put them together in Haskell. It's not something intrinsic to the language, it's just Haskell doesn't have the libraries.

There was an effort to do this, but it seems to have fizzled: http://www.macs.hw.ac.uk/~dsg/gdh/

Mature, consistent and simple to use libraries with good documentation is what Haskell lacks most. The foundation is great. Scalable servers could be a good niche for Haskell, but it would probably need backing by one or more companies to get there.

To my understanding, a major reason that Erlang was written was because they needed a concurrent system with soft real-time guarantees - in order to work for telephony, it had to operate within predictable, bounded amounts of time.

Conventional Prolog wasn't an option* , because the backtracking inherent in its execution model leads to significant variance for worst-case timing for individual operations. The same problem can happen with lazy evaluation, for both time and space - while the sophisticated compiler optimizations can allow for some really remarkable amortized performance, individual worst-case measurements can still have some serious outliers.

* Though Erlang was prototyped in SICStus Prolog.

With lazy evaluation, forcing a thunk for a seemingly minor operation can trigger a burst of postponed work. This can be a problem if it means a sudden drop in a game's framerate, for example, much like non-incremental garbage collectors' pauses have made people wary of using GC. Similarly, a JITting compiler can do more elaborate optimizations if it can afford to stall the program perceptibly while performing them, but that would disqualify the compiler for many real-world uses.

Real-time operation can impact performance, much like implementing full transactions slows down databases. Still, it's a fairly significant feature in its own right, and a language that does concurrency faster overall, but cannot prevent occasional erratic timing, is aiming for a different target than those (e.g. Erlang) that have such guarantees. It's not unusual for a program that juggles tens of thousands of connections to need to respond to them promptly, for example. Concurrency isn't just about performance.

(Also, that's why the Erlang movie has that "Declarative real-time programming NOW!" banner in the background. :) )

Also worth nothing that what GHC does in 8 seconds, Ruby does in 48 minutes. This is why I always chuckle when people get super-excited about multi-core concurrency in langauges like Perl, Python, and Ruby -- at best, you will make your app 4-8x faster. But by moving to a different language, you can make your app 360x faster.

Something to keep in mind.

But by moving to a different language, you can make your app 360x faster.

If your "app" is a toy microbenchmark with an under-specified benchmarking methodology, then sure. You could also make your "app" 360x faster by switching to a userspace thread implementation.

Lightweight threads are a much better fit for many applications. If you are writing one of those applications, this benchmark shows that you can get great performance using Haskell... with only a few lines of code.

I am sure C would do better with a proper userspace thread library, but nobody has submitted such a program to the shootout yet. "If it's so easy..."

I see your point but unfortunately using Haskell doesn't magically also make your database (/etc other bottlenecks) 360x faster, so... Ruby it is :)

And you can call Haskell from Ruby

http://github.com/mwotton/Hubris/tree/master

I have written a lot of database-driven web applications in dynamic languages, and the database has never been the bottleneck. It has always been in the application code.

(Actually, not true, when I worked for $LARGE_ADVERTISING_COMPANY, we checked the state of the session in the database for every request. This put an enormous load on the database, and made every page load take at least two seconds waiting for locks. No, I did not design any part of this system, nor was I allowed to fix it; so it doesn't count :)

I'm confused: the benchmark notes explicitly say:

Programs may use kernel threads, lightweight threads; but coroutines, cooperative threads and other programs with custom schedulers will be listed as interesting alternative implementations.

However, the Haskell docs for Control.Concurrent note that in Haskell,

Concurrency is "lightweight", which means that both thread creation and context switching overheads are extremely low. Scheduling of Haskell threads is done internally in the Haskell runtime system, and doesn't make use of any operating system-supplied thread packages.

Which makes the comparison with a C implementation that uses kernel threads pretty unfair: presumably a C implementation that uses green threads / fibers would be much more competitive with the Haskell implementation. Comparing apples to oranges isn't very informative.

As an aside, choosing a language based on how fast its green thread implementation happens to be at passing tokens around a ring is utterly silly.

You can compile your app with the threaded runtime, in which case the threads are scheduled amongst the available CPU cores.

Edit: The benchmark in question does just this, with 5 cores. So while the docs you cut-n-pasted are technically accurate, they do not apply to this particular run of this particular benchmark.

Well, fair enough: presumably GHC is creating 5 kernel threads under the covers, whereas the C implementation creates 503 kernel threads. Therefore the C implementation incurs approximately 100x the context-switching overhead. Once again, apples to oranges.

the threads are scheduled amongst the available CPU cores.

Actually, the per-CPU idle stats suggest that the Haskell program ran entirely on a single CPU, so it wasn't actually utilizing all 4 cores anyway.

I think this is because a thread was never actually blocked.

It should probably be noted that what's actually going on here is that there's some issue in the Erlang runtime that's hurting it substantially on the quad-core box. On the single-core benchmarks, it speeds up substantially and is neck-and-neck with Haskell.

Partly there's an issue here that the priority for Erlang has always been more multi-box distributed systems rather thaan single-box concurrency, and partly there's an issue that Erlang's not-really-open-source nature seems to hold back progress in fixing issues like this. And, partly there's the fact that the Erlang core team doesn't really care at all about micro-benchmarks like this.

I am curious what qualifies as true CPU concurrency vs. threading tricks/shortcuts in the implementation of a language. If you look at java-6, c++, pascal, c, and ocaml, those are the top performers that actually utilize all 4 cores, where as haskell and erlang use 100% of one CPU. Also, in the interesting alternative programs section, the top performer is java (by cheating that they mention Haskell probably does behind the scenes)

See dfranks' message. If he's right, in this context being 100% on one CPU is a feature, not a bug.

I have a question about the CPU column:

> 0% 0% 0% 100%

This was a Quad Core CPU, so my first impulse is to ask, does that mean it was running on only one core?

Yes. It was compiled with the threaded runtime, but the nature of the problem is such that all but one thread should be blocked at any given time, so the Haskell runtime scheduled them all on the same core.

That's pretty cool! Thanks for the informative response.

Yeah, when I looked at the definition of the problem, it sort of made sense.

Le sigh ... dammit, I should give Haskell more of my attention.

I just hate camel casing and type errors :( And if I'm going to procrastinate about writing For Reals code in a language, various lisps have that covered already.

Although, I'm writing a For Reals app in Ruby (w/merb) at the moment, and it's turning into a bit of an uggo. Maybe I'll procrastinate about finishing it, and give Haskell another go.

I hated static typing too until I learned Haskell. I dabbled in OCaml first and hated even that. Haskell and OCaml use the same basic paradigm for their type systems (Hindey-Milner), but Haskell's variant is more expressive. Typeclasses really made all the difference for me.

Also, Haskell is just a really, really difficult language, and it's hard to pass accurate judgment until you've mastered it. When I first started studying Lisp, it took about six weeks of intense study before I really felt like I grokked the language. Coming into Haskell with my existing Lisp background, it took me a year to feel the same way.

That last paragraph is probably what I most needed to read.

I did give The Haskell School of Expression a good whirl -- heh, I brought it down to south america with me, and hacked through a little of it each morning for a couple of months, swinging in a hammock and listening to waves crash and deciding that the water was probably still too cold to go out.

But although I understood what I was reading and understood what the code I had written did ... grok I did not.

I couldn't think creatively in Haskell AT. ALL.

Nonetheless, your comment heartens me. Please reassure me further: are you a person who has ever written or attempted to write a non-trivial mathematic proof?

Yes. I was a math major.

See, there's yer problem.

Err, I mean, my problem. Le double sigh.

You'll notice they all add up to 100%... In this (artificial) test, only one thread is doing work at any time no matter on which core that thread is scheduled. I don't know if it was deliberate (i.e. a clever optimization) but if you're going to use your threads like that it makes sense, avoids cache misses.

I wonder what sort of optimization Scala is employing ;)

You can tell by looking at the colums to the right--they show the load on each of the cores.

What is particularly interesting is that Haskell uses only one CPU to beat Java which is using four.

Has anyone noticed that SML/MLton is gone from the newest round of benchmarks? It used to do quite nicely on the concurrency benchmarks, IIRC.

What surprised me the most on this graph was how well java performed (bearing you buy enough memory ;))

yeah, at over 8x the memory, it beat an aging Smalltalk VM by 4 secs ;)

also of note: Smalltalk VisualWorks 35.86 seconds vs JRuby 6 minutes

That's conceptual more of an apples-to-apples comparison.

Reply on news.ycombinator.com