Benchmarking D vs Go vs Erlang vs C for MQTT broker | Hacker News Reader

Benchmarking D vs Go vs Erlang vs C for MQTT broker

atilanevesoncode.wordpress.com

62 pointsandralex49 comments

> Go 1.2 produces a binary that performs on the order of 10%-15% faster than the numbers above, which might mean equivalent performance to the Erlang implementation.

Huh, impressive for Erlang, beating a newer, statically compiled language, while being interpreted and with dynamic types.

newer, statically compiled language with a minimalist, unoptimized compiler

versus an interpreter that got decades to mature (and HiPE to speed things up some more)

I didn't use HIPE. It was providing no performance improvement for this use case.

Erlang's VM is very sophisticated and mature, it is at least years ahead of Go, imo. For example, compare Erlang's pre-emptive scheduling [1] to "goroutines".

[1] http://jlouisramblings.blogspot.com/2013/01/how-erlang-does-...

> it is at least years ahead of Go,

Yes, Erlang has admittedly had decades to improve the scheduler (versus a couple of years for the gc compiler in Go), so this is quite literally true.

However, you should know that goroutines are pre-emptively scheduled now; the "scare quotes in your comment seem to imply that they aren't: http://golang.org/doc/go1.2#preemption

I'm a go fanboi, but to be fair, goroutines are now more preemptive, but not completely.

In practice that means that if you call a function somewhere in your infinite loop, you'll get preempted. It's still possible for a single goroutine to steal an os thread.

Practically speaking though, I feel like nobody should be scared away from go due to it's lack of full pre-emption. Seriously, don't write an infinite loop with no function calls.

Perhaps even more impressive is that it was written by an Erlang novice. This makes me want to hit the books and start learning Erlang again.

He's an Erlang novice, but the guy is the best programmer I've met. I have no doubts that if he chose to pick up D instead he'd've written code as fast as mine.

i see no mention of "concurrency" in that article. i see that the go programs make heavy use of goroutines so why were they run in "single threaded" mode? just testing their mqttsrv with various levels of GOMAXPROCS rewards with immediate benefits. here are some numbers for pingtest and loadtest in Go for varying levels of GOMAXPROCS for them and the server:

(pingtest/mqtt gomaxprocs) vs pingtest, reporting "elapsed time":

  1x1  5.8s
  2x1  6.0s
  4x1  6.0s
  8x1  6.1s
  16x1 6.1s

  1x2  5.5s
  2x2  3.5s
  4x2  3.4s
  8x2  3.5s
  16x2 3.5s

  1x4  5.4s
  2x4  3.6s
  4x4  3.3s
  8x4  3.4s
  16x4 3.7s

same thing but for loadtest, reported value is "messages/sec":

  1x1  75521
  2x1  76523
  4x1  77477
  8x1  74613
  16x1 73678

  1x2   71439
  2x2  100571
  4x2   93426
  8x2  108938
  16x2  97593

  1x4   71390
  2x4   87426
  4x4   98746
  8x4   99320
  16x4 119692

  1x8   64413
  2x8   87445
  4x8  102953
  8x8  120720
  16x8 106842

(note that the loadtest doesn't look like it takes long enough to be meaningful in its default)

as always with benchmarks, caveat emptor, but in my opinion go does just fine.

Are your numbers in line with the blog's table?

So you get those instead of these?

Loadtest:

Go: 90.9 +/- 11 100.1 +/- 0.1 99.3 +/- 0.2 98.8 +/- 0.3

If not can you make it relevant to the table so people can compare and contrast?

loadtest for 100 500 750 1000, X*Y below means X = loadtest GOMAXPROCS and Y = mqttsrv GOMAXPROCS:

  1x1 80735 67696 67068 66585
  2x1 64380 76531 31346 43268
  4x1 77613 73794 72759 43263
  8x1 77813 64197 32899 46960

  1x2 71641 72521 72715 72695
  2x2 103711 102944 104210 39619
  4x2 73171 23803 90537 46204
  8x2 96624 49698 33714 45729

  1x4 69418 74088 34364 38935
  2x4 88805 102847 113740 108829
  4x4 102406 108110 34672 120161
  8x4 109022 124726 121642 46765

  1x8 62336 68238 71297 68754 
  2x8 104833 115276 118170 44617
  4x8 122292 126075 126101 121306
  8x8 108826 125098 124765 47713

edit: s/pingtest/loadtest/ in the explanation above.

Still not following. For Loadtest:

D: -- [121.7 +/- 1.5] -- [166.9 +/- 1.5] -- [171.1 +/- 3.3] -- [167.9 +/- 1.3]

C: -- [106.1 +/- 0.8] -- [122.4 +/- 0.4] -- [95.2 +/- 1.3] -- [74.7 +/- 0.4]

E: -- [104.1 +/- 2.2] -- [124.2 +/- 5.9] -- [117.6 +/- 4.6 -- [117.7 +/- 3.2]

G: ---- [90.9 +/- 11] -- [100.1 +/- 0.1] -- [99.3 +/- 0.2] -- [98.8 +/- 0.3]

G(2): ? ? ? ? ?

G(4): ? ? ? ? ?

G(8): ? ? ? ? ?

I mentioned concurrency when I said that using more than one system thread didn't help. I tried varying GOMAXPROCS. It either had no effect or made everything slower. And yes, the defaults for loadtest are useless, they need to run for at least a few seconds. Your results aren't the same ones I got. Maybe it's different with Go 1.2. I have to go and check.

the server (mqttsrv) needs GOMAXPROCS>1 for any appreciable changes in the results, otherwise we're just measuring contention in the client.

i ran on a 2-core intel desktop running linux.

"the Erlang version I’m not sure"

At least try to find out if it's HiPE (native compilation) or not?

"an excuse to learn Erlang, and wrote his own implementation"

Why even included Erlang if you're not going to share the source...

He wasn't in the office. I'm going to ask him about it today. Like it says at the end of the blog post, if he puts it up on bitbucket I'll include the link.

There's a link to it now.

"the Go implementation was executed with go run ". I think it would be much more appropriate to build the GO script with "go build" and use the resultant binary. Because the D program was compiled.

http://golang.org/src/cmd/go/run.go?ModPagespeed=noscript

"Run compiles and runs the main package comprising the named Go source files."

So he should have run the D program with dmd -run :D

or #!/usr/bin/rdmd

I found this article a little hard to read;

- #555 on #fff is distinctly too light for a web page's text color (I generally recommend around #202020);

- the font-size of 13px is too small—remove it and let the default of 16px show.

"Go is an opinionated language, which would be fine if I agreed with its creators’ opinions." Spot on.

Would have been cool to see how many lines of code each implementation took including the erlang one.

It's mentioned. It's not easy to measure but they're all in the 800-1000 lines region.

Summary:

I hate Go, So Go Lost this race.

I love D, So it won or tied with C which I don't like but everyone knows is fast.

I wanted to learn Erlang, so I made sure it did better than Go.

I didn't share my SourceCode so you can't see if I was stupid or cheated.

Did I miss anything?

Also there are bugs in my Go-version test, I know about them but I hate Go so, pfff, who cares.

The Go version has been written by a Go fan.

And what it changes? If code contain bugs, only full noob will use results for benchmarking.

Source links for everything but the Erlang version are there at the end of the article.

Quite a few things.

The Go version was written by a person enthusiastic and presumably competent in Go. See their perspective here where he concedes defeat on the performance front: http://blog.nella.org/mqtt-code-golf/

Adding C as a baseline is entirely appropriate.

The source code for C, Go, and D is available. The Erlang version may or may not at the discretion of its author.

Probably a more sensible hypothesis is that you like Go and saw a result that is unfavorable to it, so you had an emotional response.

> As usual, Go programs compare quite favorably to C programs in terms of memory safety...

That's not saying much :)

yes, indeed you did. he did not wrote the erlang implementation but a colleague of him did.

> Go implementation was executed with go run

Why didn't they just use "go build program.go; ./program"?

If you're compiling the other programs it doesn't seem fair to use the interpreter for go.

I thought "go run" did exactly that (compile then run)?

Perhaps I'm mistaken, looking at the source it does seem to do just that: http://golang.org/src/cmd/go/run.go?ModPagespeed=noscript

I'd rather like to see Rust in here.

Me too, but I've decided to not even look at Rust until it hits 1.0.

Good, because we're certainly not ready to compete in serious benchmarks yet. :)

The results of the latency benchmark say that a larger latency is better. Is that a mistake?

No, it was measured in thousands of messages per second.

While your "benchmark" (if we should call it that way) obviously make no sense, I would suggest you guys look into how concurrency and parallelism are different concepts.

The key takeaway in the benchmarks seem is that Go is significantly slower than the rest, much much slower.

Apart from the regular 30-sec interval postings on HN extolling the virtues of Go, it seems to me, people should rekindle an interest, or discover a new interest in D, C or Erlang where performance is a consideration, and maybe Go where a need to feel like part of the post-Ruby crowd.

    >Go is significantly slower than the rest, much much slower.

Everyone knows that Go is a new-ish language and has room to improve on performance. That said, Go seems perfectly fast since all of the languages seem to perform fairly well.

Further, were Go "much much slower", CloudFlare, IronIO, Google, etc would probably not be using it...

    >discover a new interest in D, C or Erlang
    >where performance is a consideration

Erlang on The Game: http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t...

Go looks to be about ~50% the performance of C++, but, from lots of peoples' comments/writings, is much nicer to write so they use it when productivity is more important than absolute performance.

Note: I'm interested in Go (as I am about lots of languages), but am not a user of it right now.

Ya, Erlang doesn't do well on the Benchmarks Game because the Benchmarks Game doesn't measure communication, which is Erlang's strong point. No-one writes algorithms like those in the Game in Erlang; you code them as a server in C or Java or anything and farm work out to them from Erlang.

> Erlang on The Game: http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?t....

The Game is a terrible benchmark for Erlang. Math, computing digits if Pi, string searching, n-body problems. If that is the target domain, don't pick Erlang, or Go, pick Fortran or C.

    >Go is significantly slower than the rest, much much slower.

Did we read the same article? D was faster than the rest, but Erlang/C/Go were not that far apart, unless its too late in the evening and I'm being stupid.

I don't understand those benchmarks, 'insert any languages' can achieve performances than 99% of companies don't even need. And for the other 1% you're probably smarter than those posts on HN.

Reply on news.ycombinator.com