Akka, Haskell, Erlang, Go and .NET Core compared on 1M threads | Hacker News Reader

Akka, Haskell, Erlang, Go and .NET Core compared on 1M threads

github.com

170 pointsatemerev158 comments

Looks misleading to me. Scala code spawns real threads while Go code uses go routines.

If we want to compare apple to apples, we should use thread pool shared across Scala actors: http://stackoverflow.com/a/1597942/341181

Just add 7 lines of code and Scala should get at least 10x faster.

Akka actors are not "real" threads, Akka scheduler runs them through available resources.

Still why didn't you did that:

    import scala.concurrent.duration.Duration
    import scala.concurrent.{Await, Future}
    import scala.concurrent.ExecutionContext.Implicits.global

    object FutureGen {

      def caller(num: Int, size: Int, div: Int): Future[Long] = {
        if (size == 1) {
          Future.successful(num.toLong)
        } else {
          val futures = for (i <- 0 until div) yield caller(num + i * (size / div), size / div, div)
          Future.sequence(futures).map(_.sum)
        }
      }

    }


    object Root extends App {
      // Warmup
      Await.result(FutureGen.caller(0, 1000000, 10), Duration.Inf)

      val startTime = System.currentTimeMillis()
      val x: Long = Await.result(FutureGen.caller(0, 1000000, 10), Duration.Inf)
      val diffMs = System.currentTimeMillis() - startTime

      println(s"Result: $x in $diffMs ms.")
    }

This would give Scala: 499ms and Go: 436ms which is pretty neat, since go actually has no type system. (And I'm using the global execution context, which could also be changed) Currently your akka version is slow, since it's creating a new class on every call, so you would have like a dozen more methods to dispatch than my version.

Could you please open a pull request? Initially, I wanted to compare Akka actors with Erlang actors; perhaps I am unfair to plain old Scala.

I've written inside the PR, actually if you inline the function to the main object you are actually ~50ms faster than go.

Are you sure that is faithful to what the benchmark is trying to simulate?

That's not the same. You are just creating a bunch of completed futures. Your code basically is just summing numbers up in a recursion here. Which is not what OP is trying to benchmark.

actually Future.sequence() will run dispatched. it would just make no sense to run the single number concurrently, as the go version won't do this, too. Future.sequence uses the global Executer.

        sum := 0
        for i := 0; i < div; i++ {
            sub_num := num + i * (size / div)
            go skynet(rc, sub_num, size / div, div)
        }

This is the go version and only the inside will run on a thread so I made a fair comparsion P.S: The go version will actually run single threaded when you are having GOMAXPROCS=1.

Go example seems off too from what the author intentions were. As for the dispatcher, I don't believe it will actually give any resource to a future that is already complete? OP wanted to have messages going through 1M actors, so there would be a moment when each actor gets a thread from a thread pool.

However it makes a difference if I create 1M actors and dispatch them via reflection or if I create some of them and passing results back and forth. actually goroutines would use static dispatching and don't have a lot of overhead. however a scala actor is totally different than any goroutines or erlang actors, they are keeping a lot of state. a promise/future is actually a little bit better in this kind of sense, still the jvm version will actually (even when optimized) use way more memory.

This is a silly argument. If I read you right, you are saying that a goroutine is closer to a promise/future than a Scala actor in terms of resources used, and therefore it's OK to use futures instead of actors in the Scala code and compare them that way.

That betrays the whole benchmark. The basic point of the benchmark is to answer the question, "How expensive is it to create and run 1,111,111 actors in various languages?" You can't make a manual optimization that causes only 111,111 actors to be created and then say that it's a better comparison.

goroutine is not an actor. its a concurrency mechanism. so a goroutine is not an actor, however a future (http://doc.akka.io/docs/akka/2.4.1/scala/futures.html#Use_Di...) is definitly closer to an actor than a goroutine. so you can't compare golang and scala/erlang. too bad, huh?

And when you are clever you would be comparing monads, which actually a goroutine could be. and a future/promise, too.

Btw. Even a Future does more than a goroutine do. it won't crash your whole program when it fails, which a goroutine does.

Yes, goroutines are a concurrency mechanism. Actors are also a concurrency mechanism, so I fail to see why the comparison is unfair. As you say, Actors are safer in that uncaught exceptions in an actor will not bring down a program. They have a few more dials and knobs, you can store references to them, and they are more heavyweight. They have more overhead as can be expected from these facts and this is exhibited in benchmarks like this one.

Futures are not a concurrency mechanism precisely. They are a monadic data structure around possibly-concurrent execution. You can wrap a Future around concurrent execution by an Actor, or you can do what you have done and wrap a Future around a sequential recursive computation. As with the synchronous .NET core implementation of the benchmark, the performance is better due to not having the overhead of Actors or other concurrency. Like the synchronous .NET core implementation writing the computation this way is not a fair comparison to the other languages in the benchmark and the high performance does not indicate anything about the efficiency of the concurrency mechanisms available in Scala.

The goroutine is actually in a separate concurrently executing computation. The "go skynet()" call from the Go example is a real concurrent call that will block on the "c <- xyz" lines and resume execution when a reader consumes the value. Calling Future.successful is not the same thing.

the compiler will definitly inline this part:

     if size == 1 {
	c <- num
	return
     }

No it will not, and can not.

"c <- num" is a blocking statement. Since c is an unbuffered channel, execution cannot proceed past the statement until a reader consumes a value. The reader in this case creates 10 goroutines executing "c <- num" and only then starts consuming values. If the statements you wrote were inlined execution would immediately halt in a deadlock.

BTW, there's a merged PR which makes Akka a lot faster; can someone measure that on OS X (MB Air or MB'2015) ?

Just gonna put this out there: Scala is full of these scenarios where very small changes to the code have disproportionately large changes to the outcome. It's one of the reasons I'm really not a fan of the language. There are so many features that are mentioned in the literature and lore that you have to learn to not use, for fear of pervasive performance penalties.

This is a great example. I've done a lot of Scala and o found the original version very obvious to read, but only experience tells me why the compiler can't make it efficient. Your version is not really worse in any way, but I think it's fair to say that it shouldn't be obvious to everyone why one is suboptimal at first glance.

even if your downvoted. I think what you wrote is somehow true. it's really hard to optimize. but I don't think that Scala is the only language which has this downside. Also you could write really really awful code with scala.

Yes, if we make all the implementations do the same thing then we should expect the same results!

The point is exactly to compare different implementations of the same concept. That's always the point in performance comparisons!

The parent comment is precisely saying that they don't all implement the exact same concept. Also, different runtimes will give different results, even if the concept is exactly the same.

Similarly the Haskell and Erlang are using lightweight threading.

I think a lot of people are overlooking something really important here. There are numerous comments about optimizing the code to perform better on this task. I might even go as far to say people are taking offense because their language of choice didn't perform as well they'd hoped. Many of these comments add complexity to the solution and make it harder to maintain. I like the fact that the author kept things simple. It's kind of like using the default settings for each language and then comparing the results. Both the simple version and optimized version are important data points, and they give you a better view of each languages limitations.

sorry but the scala version is actually pretty pretty dumb.

also go is not better than all of these languages. what we learned here, is actually that go is easier since you only have one common way to do this task. go has goroutines, but go doesn't have runnables, futures, actors, completionstages, executioncontexts. GO has GOMAXPROCS and not dispatchers, thread pools, etc..

Also the JVM eve Java only code would actually beat golang in many ways, but thats nothing you should be scared of. Golang is 1 1/2 year old, the JVM lives like 20+ years and they spent numerous times about the GC implementations (they could be even changed so that you could use a better GC for conurrency or for parallelism or for synchronous execution). Go is good for the task it is created. Other languages maybe tend to be more general so that you could do many things and due to their years of knowledge they are actually more optimized than go could be / they could be more optimized than go could be.

> go doesn't have runnables, futures, actors, completionstages, executioncontexts

Some of us (most of us, I would contend) who have been successfully programming highly concurrent software without them for 20+ years are totally OK with that.

> Some of us (most of us, I would contend) who have been successfully programming highly concurrent software without them for 20+ years are totally OK with that.

I work on parallel software every day and I would not be OK with having threads and channels as my only tools. I would agree that they're usually the best thing to reach for initially, but it's essential to be able to have low-level control over the scheduling if you want to get good parallel performance out of the hardware. I have workloads that today result in 3x speedups on 4 cores but would become slower than the sequential algorithm on 1 core if I were to rip out the highly-tuned parallel scheduler and replace it with threads and channels.

To be fair, though, I'm working on CPU-bound software and if my workload were I/O bound I'd likely have a different outlook on this.

Actually, Go is at least 6 years old; the original announcement was Oct 30, 2009:

https://youtu.be/rKnDgT73v8s

yeah sorry however I would actually use go version 1 as a reference. everything before was changing just too often, there was go fix however still many changes. also everything before 1.0 wasn't targeted too much about performance. also I can remember how much the template spec changed. thats what I used these days quite heavily. I have a book about go that dates 2011, and I think that was go 0.7 as far as I remember.

Fair enough; in that case Go 1.0 is just about 4 years old:

https://blog.golang.org/go-version-1-is-released

I am scared that Java takes an order of magnitude more memory than Go with its 20+ years of optimization and all. Java's GC does not appear to be significantly better than Go's one. Also Go's runtime and GC both written in Go, something Java has not been able to do so yet. So Java appears less general purpose to me. Yes Java has humongous number of libraries and frameworks but that is expected of a popular language that is 20+ years old.

> Java's GC does not appear to be significantly better than Go's one.

Java's GC is generational, allowing bump allocation in the nursery. That is a huge advantage.

Considering Go is showing under ~10ms GC pauses. Java's advantage seems theoretical rather than real world. I work in Java at my job and I have not seen such low GC pauses in Java ever. edit: Also considering huge garbage idiomatic Java libraries and user code generates, Generational GC in Java is a basic requirement not a powerful advantage over Go.

Bump allocation in the nursery is a huge advantage. There's a reason why fast Go code goes out of its way to avoid allocations (notice that the benchmark tool contains an allocation profiler), and it's because allocations are expensive when you don't have a generational GC. If you do have a generational GC, then allocations are incredibly cheap: just a pointer bump and check. It's like 4 or 5 instructions compared to the dozens in a typical malloc. With that kind of collector, you don't have to spend nearly as much time tuning your program to avoid allocations. This is the reason why Java has been slow to add value types: when you have a bump allocator, they really aren't needed as much (which is not to say they aren't needed at all).

In the HotSpot VM, you can tune the GC to get any type of max garbage collection pause you want (-XX:MaxGCPauseMillis). This is basic functionality that any concurrent garbage collector supports. The downside, of course, is that pauses will happen more frequently, reducing throughput. That is why just saying "it's under 10 ms, so it's really fast" is very misleading: when your GC is interruptible, you can of course get any max pause time you want. But the GC may not be able to keep up with your allocation rate, and it may start being invoked too often. That's why you need to not just look at latency, but throughput--something that sound bites won't capture.

Side note: many functional languages use the same bump alloc method because they tend to have higher allocation rates than imperative programs.

I really hope this doesn't become a meme.

Pause time is only one metric that's relevant when looking at GC, another very important one is throughput.

In fact you can get very low (10-50 msec) pause times with the latest JVMs even with very large heaps, you can just request it at the command line, but you'll pay for it with slower execution. There is always a tradeoff. Go doesn't have anything fundamentally new in this field.

There is a report here on an adapted HotSpot (not integrated into the mainline JVM though) that gives <10msec pause times at the 99th percentile and an 8 gigabyte heap:

   http://welf.se/files/OL15.pdf

But you pay for it with a 15% CPU overhead.

Go's GC seems to actually do a full concurrent mark/sweep of the entire heap every single time. That suggests to me that throughput could be rather poor.

> Go's GC seems to actually do a full concurrent mark/sweep of the entire heap every single time. That suggests to me that throughput could be rather poor.

Not to mention that you lose bump allocation in the nursery. Nongenerational GC is a lose-lose all around.

Right, scanning large heaps means someone has to pay along the way. The Go GC pays with lowered throughput much like the adapted HotSpot VM is showing. The 10ms pause times for go were reported for a 250 gigabyte heap, however, and that is pretty amazing.

But the lowered throughput has to be present as you are scanning the heap and you have enabled the write barrier to make the concurrent scanning work its invariants. In fact, the newest versions will make aggresively allocating goroutines run slower by using some of their timeslot when they request data do run the mark phase.

Have you tried using the G1 garbage collector? http://www.oracle.com/technetwork/articles/java/g1gc-1984535...

See the -XX:MaxGCPauseMillis=NNN argument, you might be able to set it to 10ms on highly performant hardware.

Then "Skynet" should be removed from the description, as well as Actors. I guess the word "whatever" deserves to stay though.

Summing a bunch of numbers is a silly task anyways

In the end, isn't that all we're really doing?

This has no external data so workers may avoid a cache fetch let alone having enough data for regular cache misses. Even a simple strlen call in each worker could begin to add the kind of issues that affect meaningful performance.

How a language handles misses/blocking/threading is more important than what it can do in a perfect cpu only world.

A true coder wanting to implement things near correct with algorithm complexity (big O), will try to find how to implement the right code by taking time to research understood and time tested patterns they've learned (it is an applied science after all) and implement them.

Saying the average joe doesn't care to understand what his/her Lang is doing under the hood which will doom them to repeat complexity that many here will respond with "duh" is not justification that a Lang is better - because it supports that behavior "better".

I don't know erlang that well but I'll be damned if I don't spend the time to find out how to properly do M:N with efficient tail recursion.

If one doesn't care for efficiency at all then who cares about benchmarks?

You missed the point. People aren't posting better algorithms, they're posting domain specific knowledge to improve efficiency. The kind of stuff that takes you away from building new features that add value. Benchmarking and efficiency is an important datapoint when picking a language. It's important to understand how un-optimized code performs too.

In my comment I support domain knowledge experts.

In my comment I purposely mentioned erlang because to implement algorithmic efficiency you must have domain expertise.

"I don't know erlang that well but I'll be damned if I don't spend the time to find out how to properly do M:N with efficient tail recursion."

Tail recursion is an efficiency, doing it right in erlang requires domain expertise.

Building new features and adding value is business. Someone with domain knowledge in any of the langs will be able to implement those business values in the best possible way.

If your business is built with shitty coders doing shitty things you will eventually find yourself in hot shit. Even if it takes 5 years. What comes around goes around.

I would take the domain knowledge developer over any monkey anytime. That's your 10x developer. That's who built all that awesome open source stuff you use all day. Not the business feature value guy.

He builds businesses that fail 99% of the time. That other guy building that obscure Lang which turns out useful for that niche topic (I.e concurrency) lives on even in concept to build or inspire other langs/ers.

You see, that domain expert knows to pick the right algorithm for the right job.

He knows the right Lang for the right job.

Even if that job is only business value.

Long live domain knowledge experts and those who wish to attain it.

Absolutely. Benchmark should be done with idiomatic code you'd like too see in a documentation. Not ultra optimized ones. At least not only.

I'm pretty sure this doesn't measure what it claims. It's a mostly sequential flow with very little meaningful concurrent work going on. I would imagine a for loop would put all of those to shame. I'm not sure why anyone would optimize their runtime for what's basically yet another process ring benchmark but I know it's gotten slower over time on a couple of the measured cases because it's just not a real example and making this fast trades off other things like good scheduler balancing.

Also check out Pony if you want something were spawn and send become basically free. It'll run pretty close to for-loop speed for the number adding which should be a baseline calculated in all implementations.

> I would imagine a for loop would put all of those to shame.

Indeed, see ".net sync" version. The point is to measure process spawn/message passing performance.

He has a valid point though. Goroutines are more like for-loops than processes. You can't kill them, can't manipulate them, can't store references to them, etc. Comparing that to actors is incorrect.

Bingo

There's updated Akka version which is a lot faster, btw.

Someone should hack together C coroutines using Duff's device and throw it up there. It'd win handily, probably around 1ms. You've got around 100 million instructions to use in 1ms on that i7-4770 and no cache to hit, go nuts.

>> basically yet another process ring benchmark

For those who don't know what you're referring to, an example:

http://benchmarksgame.alioth.debian.org/u64q/performance.php...

Another microbenchmark flamebait on top of HN? Seriously?

Those things never describe real workloads - in such a view a Porsche is always better than a truck because it accelerates faster. Tough luck if the workload is shipping 20 tons of manure. Put that in your Porsche.

The Haskell version was not compiled or run as a multithreaded application. Given sufficient heap to avoid a GC in the middle of the run I'm seeing ~ 375ms +/- 25

https://github.com/RayRacine/skynet/blob/haskell/haskell/Sky...

This is hilarious.

there is a major flaw in the erlang version which causes unnecessary list allocation and garbage collection multiplied by 1 million times. I would try to replace lists:seq(0,9) with a tail recursion. Even that it's a small list of 10 integers, it is still created 1 million times.

https://github.com/atemerev/skynet/blob/master/erlang/skynet...

EDIT: it seems that list:seq overhead is insignificant - so it doesn't matter, maybe erlang compiler already optimizing it. I even inlined the calls to spawn - it still the same result. But I found another problem: it seems that some processes are stuck - so if you run benchmark the 3rd time - yo get out of processes.

EDIT2: Note, that unlike other solutions. Erlang has process isolation, i.e. if there is a panic in one of your goroutines - your entire server will crash, but if there is an exception in your erlang process - only that process will crash, not affecting other processes in your server (unless they linked with it).

> there is a major flaw in the erlang version which causes unnecessary list allocation and garbage collection multiplied by 1 million times.

There's probably an alloc cost, but there should be no GC cost: erlang processes get a 233 words heap by default, a list of 10 integers is 21 words, so the process should die without needing to run GC.

yes, when process ends/dies all the heap is reclaimed without regular GC. I also tried spawn_opt/5 with reduced min_heap_size option - it doesn't help.

> I also tried spawn_opt/5 with reduced min_heap_size option - it doesn't help.

Yeah whether using spawn_opt or VM options (-hms and -hpds) it doesn't look possible to reduce the process heap (or process dict) below the initial size, only to increase it.

Just tune the VM: https://github.com/atemerev/skynet/issues/33

and speed doubles :)

Go is extremely performant. But then again, isn't the author comparing apples to oranges so to speak? A Go program runs as a native binary, whereas Erlang and Scala/Akka run via their respective VMs. Wouldn't these results be expected?

You mean, he should implement a VM for Go or in Go? So the comparison is fair, but totally useless by ignoring the way programs in those languages are typically run? It wasn't meant to be fair, it was meant to be relevant.

Of course the results are more or less expected. JVM is by far the worst, because it uses OS-level threads, while Go and Erlang use userland threads, which are way cheaper to spawn and manage.

[edit] According to some other comment around here, Scala is supposed to use userland threads as well.

Nope. I just think the comparison should be nuanced. I agree that these data are relevant.

Yup.

Scala runs on JVM and uses OS-level threads.

However, Actors are not threads, so in a way it's like user threads. One million actors will be scheduled to run on a pool of OS threads.

The author also measures .net core, which has the best performance of them all. I believe that is also running non native.

Moreover, the core assumption that native == faster is not true. One advantage of the jvm is that as a program runs, the runtime can internally profile and adjust the byte code on the fly in order to increase performance.

Hey! Yeah, I just noticed the author added .Net core (this wasn't apart of the benchmarks when it was first posted to HN). I agree that the assumption is violated. On the topic of native code though, check out the numbers for .Net core on Win 8.1 vs. Mac OS X (almost 3x faster on Windows, which is expected).

(dotnetcore version author here)

Please note that Windows and OS X is benchmarked on different CPUs!

You're welcome to re-test at dual-boot MacBook together so we can meaningfully compare OS X/Windows

If anything is unfair it's that they're using long ints. Java (and I assume other JVM languages) have the overhead of allocation for most of the longs.

Testing with allocation in all languages will give a better idea of performance (and I won't be surprised if .Net core fares very well).

JVM has primitives for longs

Is the .NET version actually spawning any threads at all? It uses Tasks to collect the results, but all the work is done in the main thread by the looks of it.

.NET version has 2 variants for exactly this reason :) Single-thread version is fastest of course.

The async version also doesn't spawn as much Actors/Threads/whatever than all other implementations. It only runs the Task.WhenAll continuation in the ThreadPool - and probably even that not because it can recognize that all child tasks are ready and execute the continuation synchronously. The leaf tasks, whose amount is the biggest, are only synchronous function calls that return a fulfilled task instead of spawning anything. A fair comparison should use at least use something like `Task.Run()`, which would make it a lot slower. And then you can make it a lot faster again by using a custom TaskScheduler.

I think there are by far too many knobs to turn to make this comparison meaningful.

Updated to use Task.Run, got only 15% perf decrease, which can be due to the delegate creation too...

That was a quick update! Looks better now. I would have expected the difference to be much bigger. I made some similar comparison about a year ago (with normal .NET), and it performed slower than Go. However with a custom Actor implementation (similar to the Scalaz implementation) I was a lot faster. F# with the Hopac library also performed very good.

Another .NET version arrived, 2x faster LOL

Its also running in debug mode...

.Net is compiled to native code, either jit or at deployment time.

What you're saying could be taken two ways.

1. That the time (including start-up time) for something on the JVM would be longer because the JVM needs to start up.

2. That Go producing a native binary is automatically faster at things.

In reference to the first, the benchmark timings are started when the software actually starts running.

In terms of the second, being a native binary doesn't mean faster. Go still has a runtime. It's just included in the binary.

Go's performance advantage is probably a few things. Go was made to spawn millions of goroutines and Go uses primitive types. I don't know all that much about Scala, but I believe it doesn't do primitive types which means that every one of those Int objects requires following a pointer to get to the value. That's overhead. EDIT: this looks like it's wrong as Scala has a bunch of pre-defined classes that extend AnyVal and correspond to primitive types in Java.

It also looks like the Scala code is passing back a Long even when the value is still small enough to fit in an Int. That's doubling the amount of memory when that memory only needs 64-bits near the end stages. By contrast, the Go code does `sum := 0` which I'm not actually sure how that's working. If I initialize a 32 bit int without specifying it, I see that I wrap around (http://play.golang.org/p/PiiMZmEzKp). So, I'm not sure how initialising it to 0 and then adding to something over an int32 works. Maybe Go uses a different default int depending on platform?

Go's implementation is also simpler. In the Scala version, there's a receive method and the sub-actors just pass back when they're done potentially causing a bunch of switching. Go, on the other hand, creates all the goroutines and only when that's done starts receiving from them. In the Scala version, the program might think that there's value in switching from creating the sub-actors to receiving from them - and they should pass back very quickly. The implementation here in Go doesn't allow for that switching.

When dealing with something as synthetic as this, all of these little things matter.

It would still be fair as long as they are equally idiomatic in their respective languages. If one solution avoids slow startups,requires less of programmer's 'think cycles', the need to wait twiddling thumbs while something compiles or warms up; then there is still a legit point to be made. Not a fan of Go by any means but have to give it credit for making concurrency more mainstream

> A Go program runs as a native binary, whereas Erlang and Scala/Akka run via their respective VMs. Wouldn't these results be expected?

No? Erlang and Scala both have JITs. The JVM does quite a bit more optimization than Go's compilers do.

Erlang doesn't have a JIT. BEAMJIT hasn't been released yet.

The point to understand here is that Go uses lightweight concurrency and has a layer of indirection between goroutines and hardware threads. Goroutines allocate a tiny stack and Go uses a software scheduler to jump between routines. Then that Go scheduler starts hardware threads as workers to start executing the routines.

The net result is that Go is relatively efficient for naive programs that start hundreds or thousands of threads and let them all run.

I don't know Scala or Erlang, but based on the results I would assume both are allocating a posix thread for each thread in the software and so it is using the kernel scheduler to switch between them with all the overhead that entails.

But a program can be refactored in all the languages to have a smaller number of workers and a queue of work to be consumed. This would be faster than the Go version, but require more code and complexity.

I have been considering threading in Rust vs Go and that is the biggest difference I see between the approaches. However it should be possible to implement Go style goroutines in Rust using custom dispatch code like old C coroutines. I think I have already seen libraries like that, but I haven't dug in the details to know for sure.

No, Scala and Erlang don't allocate pthreads. Their threads are also lightweight and in-system.

JVM threads map to native threads

That's true but there is some nuance here because you can use the JVM fork/join executor service & see behavior similar to coroutine's.

Unclear wrt Scala, jakozaur asserts the bench does use pthreads.

Erlang does use lightweight processes, but the configuration isn't tuned for high creation/destruction throughput: on my system, the smallest possible process is 333 words, or 2.5Kb (control structures, default stack and default private heap)

Are you sure about scala? I would assume it just uses JVM threads, which map to OS threads on most modern implementations.

Yes, I am sure. M:N dispatcher is used here.

http://doc.akka.io/docs/akka/2.4.1/scala/dispatchers.html

Scala runs on JVM and uses the same OS-level threads. What's going on here is the actors are being scheduled to run on a thread pool.

Scala does not have lightweight threading. It uses the same thread pools from Java. All your millions of Actors are scheduled onto one of these thread pools.

Which means you need to program with futures in order to not do blocking IO on your limited thread pool that the actors are multiplexed across. So at that point you need to write node.js style code.

Coming from an Erlang world to akka, this blew my mind.

> I don't know Scala or Erlang, but based on the results I would assume both are allocating a posix thread for each thread

No. Erlang is only ~five times slower than Go, according to the benchmark run, which is a marvellous result given the Erlang's VM is slow as hell, even with HiPE. Scala/Akka is slower by thirty times under this workload.

> But a program can be refactored in all the languages to have a smaller number of workers and a queue of work to be consumed. This would be faster than the Go version, but require more code and complexity.

Oh, I bet it wouldn't be faster, but it surely would add tons of complexity. Erlang was specifically designed for programmer to spawn a thread for each activity. New client connected? Spawn. Need a background computation? Spawn. Need a connection to external entity? Spawn. Need to do cleanup (reliably) after this computation terminates? Spawn. Go shares this approach to some extent.

If you do file system related things in a goroutine it sometimes spins up a new OS thread, and if you are not careful you'll have thousands of them in no time and they never seem to die.

> I don't know Scala or Erlang, but based on the results I would assume both are allocating a posix thread for each thread in the software and so it is using the kernel scheduler to switch between them with all the overhead that entails.

Incorrect.

> I have been considering threading in Rust vs Go and that is the biggest difference I see between the approaches. However it should be possible to implement Go style goroutines in Rust using custom dispatch code like old C coroutines. I think I have already seen libraries like that, but I haven't dug in the details to know for sure.

There is https://github.com/dpc/mioco for I/O-based coroutines. For CPU bound tasks you want something like https://github.com/nikomatsakis/rayon.

In go version you're using unbuffered channel, which will cause a lot of extra goroutine swithces please try to change line 10 to: rc := make(chan int, 10) for example.

After this change my results changed from: Result: 1783293664 in 61586 ms. to: Result: 1783293664 in 28401 ms.

How about running this on a 36+ core machine? Testing parallelism on a dual or quad core isn't that exciting. Maybe we'd realize this benchmark isn't really testing parallelism but primarily sequential thread creation/deletion. This could very well be fast on a scheduler optimized for a single core.

And just to be clear, the interesting property to test is parallelism, not concurrency.

Except that schedulers optimized for single cores do not grow on trees, either. Python 3.5, which I think is a language with very high incentives for such single-core optimizations (because of the GIL), does almost two orders of magnitude worse than Golang (1.7s vs 63s) on a single core.

I wrote a py3.5 asyncio coroutine implementation of this benchmark, and I hope it's just my inexperience with asyncio that makes it get such horrible results.

Here it is: https://github.com/ociule/skynet/tree/master/python35-asynci...

I think it would be interesting to try http://docs.paralleluniverse.co/quasar/ too (fibers, and actors on top, for the JVM).

Ye olde thread-ring was less about startup ;-)

http://benchmarksgame.alioth.debian.org/u64q/performance.php...

I had thought looking at the title that this was about a benchmark on a machine with one million hardware threads, which would have been interesting. Turns out to be a job of spawning loads of unprioritized identical threads that block, do negligible work and exit. It looks like time costs are going to be almost entirely dominated by setup and tear down, Which is very nearly useless for any situation in which you expect threads to exist for more than infinitesimal time. Setup and tear down are constants. They won't dominate the runtime of most programs.

MacBook Pro, 15-inch, Early 2011, 2.2 GHz Intel Core i7, OS X 10.8.5. Yeah yeah, i know. At least it was more or less unloaded when i ran the tests.

    $ git show --no-patch --pretty=oneline HEAD
    82f318ad418aa10eb5553bcbfad1783e544c76cd Merge pull request #26 from bitemyapp/master
    
    $ java -version
    java version "1.8.0_11"
    
    $ go version
    go version go1.5.3 darwin/amd64
    
    $ (cd scala; sbt compile run)
    Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384m; support was removed in 8.0
    [info] Set current project to skynet (in build file:/Users/twic/Documents/Code/skynet/scala/)
    [success] Total time: 1 s, completed 15-Feb-2016 22:59:34
    [info] Running Root
    Result: 499999500000 in 8832 ms.
    Result: 499999500000 in 4866 ms.
    Result: 499999500000 in 2457 ms.
    [success] Total time: 17 s, completed 15-Feb-2016 22:59:51
    
    $ (cd go; go run skynet.go)
    Result: 499999500000 in 399 ms.
    
    $ (cd java; ./gradlew run)
    :compileJava UP-TO-DATE
    :processResources UP-TO-DATE
    :classes UP-TO-DATE
    :run
    Streams
    -------
    Result: 499999500000 in 18 ms. (sequential)
    Result: 499999500000 in 38 ms. (parallel)
    
    RxJava
    ------
    Result: 499999500000 in 204 ms. (immediate)
    Result: 499999500000 in 216 ms. (computation)
    Result: 499999500000 in 188 ms. (io)
    
    BUILD SUCCESSFUL
    
    Total time: 10.784 secs

Get off my lawn.

This kind of benchmark can show off what v8 is capable of (when used with a library that has been carefully tuned for performance)

Code is pretty-much copy pasted from .NET Async version with bluebird promises instead of Tasks. Ends up being almost 2 times faster:

https://github.com/atemerev/skynet/pull/16

(Only 50% faster if we return Promise.resolve(num) though)

Returning num instead of Promise.resolve(num) is unfair compared to .NET version ;) but I merged your PR as-is

I just tried it with Promise.resolve and after JIT warms up its 350ms, while .NET (async) after JIT warmup is 300ms

Just to drive the whole "this comparison is misleading" point further: https://github.com/atemerev/skynet/pull/23

Lwt is a cooperative multi-threading library with a particular semantic: A task that doesn't block doesn't even yield, it returns directly.

Yes, it's comparing apple to oranges.

When i saw 1,111,111 threads i knew Go would be a winner here, because Go uses green threads which are not system threads - no syscalls needed etc.

Erlang doesn't use system threads either.

Yeah, isn't it weird that the Go implementation is so much faster than Erlang one?

Probably not entirely, Erlang's lightweight processes have their own private heap (~1.8KB by default on a 64b system), a process dictionary and likely more control structures, and the language is mainly interpreted. A 4x different in performance may be on the high side (idk), but I wouldn't expect erlang processes to be quite as fast to spawn as goroutines.

edit: and the erlang bench allocates a bunch of lists on startup: https://github.com/atemerev/skynet/blob/master/erlang/skynet... `lists:seq()` will allocate a 10-elements linked list, considering this is called recursively and concurrently it might contribute to the issue (or not)

Minor clarification -- Erlang is not interpreted, it's compiled to bytecode and executed on the BEAM virtual machine.

> Erlang is not interpreted, it's compiled to bytecode and executed on the BEAM virtual machine.

And Python is compiled to bytecode and executed on the CPython virtual machine. That's still interpreted.

The major problem, without any measurement and just guesswork, is that you have a larger allocation overhead in Erlang. Not only do you need the process control block for the process, you also need to go to the memory subsystem and grab a heap for the process. In the Go variant, you are allocating out of the same heap and you can usually avoid some allocation there because everything you allocate is used. Not so in the Erlang system.

Were these processes/goroutines to do anything more complex, then the difference goes toward zero. But the constant overhead at the lowest level is probably higher.

Yeah, I was wondering that. There could always be the general overhead in the VM, but the HIPE version (which is compiled IIRC) should be closer. A commenter mentioned that the erlang version would be creating intermediate lists that would have to be garbage collected (see all the lists:seq(0, Div-1) ), but I wouldn't know how to avoid that without writing unidiomatic code. I'm no erlang expert though.

I don't think the lists would have to be collected, erlang processes have a relatively hefty private heap (233 words) by default, so chances are the process just dies without needing to run a GC[0]. The lists do need to be allocated and initialised though, which may or may not contribute to the runtime.

[0] the way I understand it, each process would generate a 10-elements list of small integers, which would be 21 words (1 word + 1 word/element + the sum of the element sizes — 1 word each for small integers)

Good point, it would be the allocation, not the GC that would be the problem. My mistake.

For go version changing line 10 to rc = make(chan int, 10) make it 2x faster

Another one for the list of meaningless benchmarks

From the description, almost 90% of created tasks in this benchmark are tiny "leaf" tasks, which are quite likely to be optimized away completely. So it looks like the benchmark compares how well these compilers optimize one particular case, and not general performance of goroutines/tasks/whatever...

Though not unexpected, it's good to know what the order of difference is when choosing tools for the job at hand.

akka/erlang actors are not equivalent to go routines, a closer concurrency mechanism would be futures/tasks

See .NET Core implementation with tasks :)

This is a joke, ok take this C++ version that takes 0 time to finish: https://github.com/atemerev/skynet/pull/26#issuecomment-1845...

This test was performed on a 2-core machine. I'm not familiar enough with multi-threading benchmarks to know what's standard, but that sounds suspicious to me, like the benchmark might not show how well a language takes advantage of a 32 core server.

Anyone know if that would be a problem?

actually comparing jitted vm's without a warmup is extremly dump.

Second: using the default thread dispatcher for this kind of scenario is pretty dumb on scala (also I think erlang could be made faster, too. but since I'm not an erlang guy...) Fixed Thread Pool (GOMAXPROCS vs ForkJoin)

All timers are started after the warmup.

How do you know when warmup is finished?

Warmup's a tricky issue, see this brand new paper: http://arxiv.org/abs/1602.00602

There's an update for Akka version, which measures 12/8/4.4 seconds on subsequent attempts, so yes, warmup is important... but somehow that seems kinda unfair for other languages too.

No? In the scala code the timer is started and Actors are created. No warmup anywhere. https://github.com/atemerev/skynet/blob/master/scala/src/mai...

I agree that using the default thread dispatcher is not how this should be used. The application's actors should create and use its own ExecutionContext.

/cc @atemerev

Haskell version is naive too.

My laptop runs the original in about 3.07s and my improvements bring it down to 1.20s though I think there's room for improvement.

[EDIT] My improvements were actually not the same algorithm as the original's, with algorithm corrections my version is actually slower!

A combination of tuning the runtime heap size, number of threads and compiling with -threaded made the same implementation run in 0.5s for me.

The Erlang code is far from optimal... I'll open a pull req later today.

You are most welcome! I am not an Erlang expert.

This is a terrible benchmark, but

https://twitter.com/bitemyapp/status/698975974277271553

Added Erlang on Windows numbers, added .NET synchronous numbers.

The languages tab on GitHub (click on the multi-coloured bar) is another interesting metric, is it somewhat representative of the conciseness of the languages tested?

That division looks expensive.

He says, with absolutely no profiling whatsoever... I would like someone to offer some evidence though :)

"async (8 threads)" is confusing. async and multithreaded are different approaches to parallelization.

Is the Akka version doing reflection on each iteration or does it just look like it?

Just ran the checked in Crystal code on a 2015 Macbook (1.2ghz):

$ ./skynet

Result: 499999500000 in 294ms.

theres a lot of nitpicks on code and go vs scala.. but .net is more flexible and annihilated both by a fair margin as well..

I like how the language banner is so colorful.

I think the .NET core implementation is flawed... Task.FromResult doesn't actually happen on another thread, it returns an already completed Task synchronously. You might want to use Task.Run instead.

.NET core impl has 2 versions, async and sync. Task.FromResult is synchronous, yes. Look closer :)

Doesn't matter. Unless you do TaskFactory.StartNew or Task.Run, it's not going to try to schedule work on another thread. Everything is happening in the main thread in your code.

See the updated version. Runs about 15-20% slower, which is far less than the difference between single-threaded version and task-based one. Also, it's pretty easy to see in Task manager how all the cores get loaded.

That looks better. However, I am really not sure what you are trying to achieve here. You didn't specify any scheduler, it means your tasks will use a default scheduler and that scheduler uses threadpool [1] and it will never create 1M threads. And as there is no actual work performed in the Task, things will run reasonably fast.

I am having hard time understanding what you are trying to test here. I am worried that .NET code is not really testing what you think it is.

[1] https://msdn.microsoft.com/en-us/library/dd997402(v=vs.110)....

You can't create 1M threads on Windows (or OS X, for that matter) because there won't be enough RAM for stack space. (also, running more threads than physical cores on CPU-bound task is pretty pointless).

Maybe you mean we should schedule something which has a complete event loop? That would be a lot more fair wrt the Akka version, but the Go version does not create any threads either.

Not sure what you mean by complete event loop. But I have a strong feeling that you are comparing apples to oranges in your benchmark.

It's indeed apples vs oranges vs pies (Actors/Coroutines/Futures). Probably should make a table :)

Even after the change, there's nothing async about the "async" version. It's multithreaded, not asynchronous.

I'm surprised that none of the compilers optimized this benchmark away completely...

I don't follow. What optimization technique would fundamentally change this program?

If a compiler could proof that creating the channels/go routines/what-have-you does not influence the arithmetics (e.g. because only associative operations are used and the order doesn't matter), it could constant-fold the loops and just output 499999500000 directly.

I don't know any compiler (and language implementation) that can do that, though.

Haskell maybe?

Doesn't seem out of the question, but it would require a very good type system. I think out of these languages, only Haskell is at that level.

Any of the VM languages could probably do this heuristically as well: the "if size == 1" base case depends only on one of the parameters, you could hoist that out of the concurrent thread/coroutine and into the caller. Then the code path that hits only ever assigns to a future/promise/array/whatever and immediately returns so it won't block and has no other side effects so it can run synchronously for less than the cost of a context switch.

In general programmers are pretty good about not kicking off expensive computations they don't need to, so the easy pickings here are probably pretty small. And as soon as there are any side effects at all (which includes any writes to shared memory) it becomes very hard to reason about moving computation from one thread to another. So it's easy to see why compiler writers are not super excited to start optimizing across concurrent threads.

Reply on news.ycombinator.com