Go, D, Erlang and C in real life: MQTT broker implementation shootout (2013) | Hacker News Reader

Go, D, Erlang and C in real life: MQTT broker implementation shootout (2013)

atilanevesoncode.wordpress.com

115 pointsBufordTJustice49 comments

Open articleView on HN

> What about readability and ease of writing? I can’t read Erlang so I can’t comment on that.

The Erlang code looks very nice. If you read this, great work Patrick!

https://bitbucket.org/pvalsecc/

Nice use of gen_fsm + binary matching.

Here is an example of the client code that takes only 200 lines:

https://bitbucket.org/pvalsecc/erlangmqtt/src/f37505188c1f1c...

Yeah, it all depends on code authors. This is one more example of an mqtt broker in Erlang - VerneMQ; and it's very readable: https://github.com/erlio/vmq_server We are currently estimating it.

Thanks, we're working hard to improve our codebase even more! :)

Umbrella Project can be found on https://github.com/erlio/vernemq as well as our website https://verne.mq

I'll let him know :)

Thanks and thanks for sharing the benchmarks!

Writing code in Erlang is such a pleasure. I really hope more people get exposed to it. The pattern matching alone is just so nice (and concise!).

This being from 2013 and using Go 1.2, I'd be interested to see an update using Go 1.5.

Thanks to the author for including standard deviation numbers and not just the averages like so many other "benchmarks".

Thanks, came here to say this. Go GC and code generation are improving all the time.

I really can't imagine that this benchmark is relevant anymore. Haven't Go, D, and Erlang had major changes since then? E.g., Go is now compiled by a compiler written in Go.

I'm planning on updating it this year once I finally find the time to learn Rust. Developing an MQTT broker has become my go-to task for learning new languages.

I'm really glad that you've done the D implementation. I think D gets less attention than it should, especially compared to Go.

Maybe somebody already proficient in Rust, reading this, can contribute?

No big deal, it wasn't a slight against you, but rather the OP. It's not your fault he reposted your two year old benchmarks.

How about Java, then? It's a little less esoteric than most of those languages.

Funny you should ask: https://atilanevesoncode.wordpress.com/2014/01/08/adding-jav...

So Java is the pingtest winner, and I'm sure that with just a little more work it can be the loadtest winner as well.

I doubt it, Patrick properly profiled and optimised it. Later on I got the D version to within 3% of Java's performance in pingtest.

It's clearly optimized for latency over throughput. For one, it is single threaded -- unlike the Erlang, Go and, I believe D (vibe.d) implementations. Also, I don't know exactly how the benchmark was performed, but Java takes time to warm up (otherwise you're testing a completely different implementation of Java, which is only used at startup time).

I've found that for long-running server-side apps, Java is very hard to beat in terms of performance, especially relative to the effort spent. I also think it's a nicer language than Go (I also prefer it over C++ and D).

How Go is compiled isn't as relevant to its performance as much as what Go is compiled into, which hasn't changed all that much as far as I know. The GC improvements might help with latency though.

>as much as what Go is compiled into, which hasn't changed all that much as far as I know

What are you talking about? Go code generation is getting better all the time.

The translation of the compiler from C to Go was a mostly mechanical transformation, specifically to prevent any changes in the resulting compiled programs. Go compilation is getting better all the time, but the switch itself did not change the resulting programs.

An article from 2013 means that it covers a change from at least Go version 1.2 (released 2013/12/01)) to 1.5 (released yesterday?). The performance section of each document talks about cases where it may go slower for faster in certain instance in every since release, including the one that was the change from C to Go for the compiler[2].

That said, I took the original statement about Go switching from Co to Go to indicate there's been major changes in the compiler, so it would be interesting to see more recent results, not that the change itself necessarily was responsible for major speed improvements, but they could very well have assumed the compiler change would have had a larger affect on the resulting binary depending on their understanding of the Go toolchain.

1: https://golang.org/doc/devel/release.html

2: https://golang.org/doc/go1.4#performance

It's still hard to take any benchmark seriously when given the quote:

> Mosquitto was compiled with gcc 4.8.2, the Go implementation was executed with go run, the D implementation was compiled with dmd 2.0.64.2 and the Erlang version I’m not sure.

The "I'm not sure" speaks for itself, but go run also includes both compilation time and execution time and they're comparing it against just the execution times of the other languages. That's not exactly an apples to apples comparison.

I didn't write the Erlang version, don't know Erlang nor have any idea how it's built. The curious can always check the code out.

I don't know why you think that start-up time has an effect on the benchmarks. It doesn't matter how long the brokers take to start, once they did the measurements were done. `go run` doesn't change a thing.

You're right, my phrasing was a bit misleading. It was just the most dramatic example of a "major change" I could think up.

That was basically my reaction as well. I would also like to see a Rust implementation included.

> pingtest (latency - bigger is better)

This is an odd way to measure latency. Could someone explain further?

As someone pointed out to me later, it's not really latency but latency-constrained throughput.

"Since the Erlang unit tests are in the same files as the implementation, it’s hard to know exactly how many lines long it is. It gets worse since it implements most of MQTT, the D implementation essentially only implements what’s necessary to run the benchmarks."

Benchmarking the entire spec versus only the minimal set is almost certainly part of the problem here. If you want to benchmark implementations against each other, you should probably make sure they implement the same thing!

No, not really. I understand why you'd think that but knowing the MQTT spec and the benchmark, it shouldn't make any difference at all.

Are you sure about that one? I am thinking about the situation where presence of alternative code paths that never actually get executed can lead to fairly large differences in timings, particularly for tight loops. (At least in computational code; I'd expect it to be much less common for protocol handling benchmarks like these...)

Pretty sure yeah, I looked at a lot of profiling logs.

This is slightly offtopic, but MQTT says it can provide "exactly once" delivery. I thought that wasn't possible?

It is possible if you are prepared to wait for an unbounded amount of time.

I'm the author but not the OP.

Out of morbid curiousity, how many lines of code did the C implementation clock in at?

~20kSLOC IIRC. It's not a fair comparison since it does more, but let's face it, if these implementations did the exact same thing I doubt they'd pass the 5kSLOC mark.

I get about 1.5-2x for c vs c++. Genuinely surprised that erlang took so many lines of code.

did you use GOMAXPROCS=$YOUR_CORES ? or just the standard in 1.2? Actually golang is really good in multithreading. Next you also have a Java example, did you warm it up or not? Actually thats why I hate benchmarks, since mostly they are screwed since most people who are writing benchmarks actually knowing just a little of each language. (actually they just want to show that "their" language is good)

Back then I tried GOMAXPROCS from 1 to 8 and it didn't make much of a difference. Later on with a different version of Go (I don't remember which), increasing GOMAXPROCS to 2 made a difference, but any more than that was pretty much the same. In any case (and again, the last time I tried), the Go implementation with GOMAXPROCS=2 was the slowest even though it was using twice as many threads as the other ones!

As for showing "their"/"my" language: there were implementations from 4 different sources, and I didn't write the benchmarks; the guy who wrote the Go implementation did (and that's why the benchmark app is in Go).

It's interesting to see the lowest throughput provided by the C implementation for the 1k case. Any idea why?

No, I didn't dig deeper.

2013 ? Would gladly see the same benches with Go 1.5.

Is there any hardware that uses MQTT?

There is. For example, we use it for our hardware product: https://www.relayr.io/products-services/wunderbar/

My now somewhat defunt ninjasphere uses it. I've also heard it is popular with openHab or other IoT devices.

Karma does: https://blog.yourkarma.com/how-we-use-mqtt-at-karma

update from go 1.2? since there have been many performance improvements since then...

It seems this article was posted in 2013. Not sure where that falls in the Go release timeline, but I'm guessing the version wasn't was far behind then as it seems now.

Edit: After a quick search, it looks like Go 1.2 was released only 4 days prior to this article being posted.

I run Arch Linux, all the versions were whatever was newest at the time.

Reply on news.ycombinator.com