Hacker News Reader

Comment by chrisseaton | Hacker News Reader

> Your Erlang program should just run N times faster on an N core processor

But only if your program is embarrassingly parallel with at least N times available parallelism in the first place! If you have one of those it's already trivial to write a version that runs N times faster on N cores in C, Java, multi-process Python, whatever.

If your program has sequential or less parallel phases or needs to communicate then you are subject to Amdahl's law like you always were.

Armstrong has that claim in the Erlang book and I was gobsmacked to see it written down with no caveat or mention of the limits from Amdahl's law whatsoever. I was sure it was a joke and it would be followed by 'ha ha ha of course not - nobody knows how to achieve that despite decades of intensive research', but no it's a serious claim made with a straight face. Erlang's solved it!

Erlang helps you write parallel programs... but only if you program is entirely parallel in the first place.

View on HN

That's true, but when I read that sentence I place the emphasis on should. I see it as aspirational.

A better way to phrase this would perhaps would be to say that perfectly preemptive scheduler should be able to keep all cores hot. Blocking of one Erlang process should not halt the program's progress.

As Erlang requires message passing overhead by design, it will never see perfectly linear scaling behaviour. Still, its scheduler, its GC model and M:N threading implementation are extremely robust and can be relied upon to write extremely reliable networked services.

I also interpret it this way.

If you recall the famous Wide Finder benchmark for parallel programming by Tim Bray, back in 2007 as well [1], those were essentially the results.

Languages like Erlang or JoCaml (Ocaml plus joint calculus), whose models are inherently parallel yield simple and fast solutions.

Other languages too, but they require a lot of expert optimization and ugly code.

[1] https://www.tbray.org/ongoing/When/200x/2007/09/20/Wide-Find...

> That's true, but when I read that sentence I place the emphasis on should. I see it as aspirational.

Despite all the arguing I'm taking part in, this is probably the true and reasonable explanation. I read 'should' as 'if it doesn't there's a problem'.

Yes, indeed, don't forget his very next line is "Is this true? Almost. It’s early still, but we’re optimistic (Wildly optimistic; I haven’t seen this kind of optimism for the last 20 years!)."

> As Erlang requires message passing overhead by design

That's Erlang's main weakness. For parallel code, structurally shared data is immensely useful for processes running in the same address space. With Erlang, you have to give that up.

OTOH, not having this possibility makes distributing processes to other nodes a lot simpler.

You should be able to share data in Erlang's model, because the data are immutable. You just can't modified the shared data.

Yes, I meant immutably shared data.

> If you have one of those

And how would you have one of those? Erlang encourages you to design and think about programs in a certain way, that typical C, Java or Python does not. I've seen Armstrong in one of his talks to actually emphasize the importance of learning distributed systems if you wanted Erlang to be useful for you. That's way beyond "embarrassingly parallel" point of view, Amdahl's law has no use here.

Obviously said "program" must be a server that services independent requests. I mean, not obviously, but if you work in telecom, web, etc. that's pretty much all programs you see.

I get that, but as I say we already knew how to scale those linearly in most programming languages. Erlang brought nothing new there.

It does have advances in things like reloading and supervision.

Erlang was invented 30 years ago to solve problems that were not easily solvable by the back then prevalent programming languages. The claim that Erlang hasn’t brought anything new to the table couldn’t be further from the truth. Designing concurrent, fault-tolerant programs was not a problem solved in a satisfying way by the programming languages and technologies that were mainstream back then, which is exactly why Ericsson pursued the path of inventing their own programming language. I suggest everyone to read Joe Armstrong’s paper “Making reliable distributed systems in the presence of software errors”. It’s easy to read and mind expanding even if you do not intend to use Erlang. It can also save one from public embarrassment when one claims ridiculous things as parent and many other remarks in the other comments.

I acknowledged the areas where Erlang did bring advancements such as fault tolerance.

I read that paper, more than once, and his thesis, when I was doing my own PhD on parallelism, so I know the area and related work fairly well.

I don't think I'm embarrassing myself when I say that it's silly to claim that an arbitrary Erlang program should achieve linear scalability. But we don't need to debate with vague claims to originality - as I say Amdahl has done the maths and shows that it is only possible for a program that is embarrassingly parallel.

Concurrency is not parallelism. You conflate parallelism with concurrency in your comments both here and above. I should also mention that for a big part of its life the Erlang virtual machine has had no support for parallelism. Erlang was not invented to solve parallelism so your hypothesis is downright wrong. Parallelism was just a consequence of the concurrency which was not exploited by the Erlang virtual machine until a much later point. And yes, if you read the paper I have referred to in my previous comment you will happily find out exactly the reasons for why Erlang was invented and what sort of problems it has solved. Also, to clarify, I did not claim that you were wrong about Amdahl’s law; you are however completely wrong in your claim that Erlang has brought nothing new other than supervisors and hot code reloading to the table.

I think OP specifically talked about this assertion, "Your Erlang program should just run N times faster on an N core processor." This assertion is about parallelism, not concurrency.

I don't know what to say apart from that I disagree with how your describing my argument back to me.

Funny how often that's the only reasonable response. Nitpick: s/your/you're/. I do my part to stop language drift.

I searched many times the comparison of Erlang BEAM and JVM regarding performance but couldn't find much. May be it is not done, or maybe it is not publicly available. At my work some folks keep pushing RabbitMQ over ActiveMQ for no other good reason than it is in Erlang so we will get much better performance. I know it is not true but could not find web links to support that.

BEAM is slower than the JVM for sequential code but it doesn't have a block the world GC and it has preemptive multi processing. It'll have worse steady state performance but a lot of the worst cases that Java can have are heavily mitigated.

It's also realllly good at routing; the control logic is a somewhat slow more-or-less interpreted language speed but then the act of routing bytes around runs at native speeds (as long as you keep things in binaries and don't convert them to "strings"), with the Erlang framework making sure by the nature of how it works that there's no contention or anything getting in the way because everything's immutable.

It's sort of like how people talk about how NumPy can be so useful; Python is itself one of the slowest languages (especially since when it comes to NumPy we can't necessarily count on PyPy, though that's coming along) but it doesn't matter because you just use the slow language to script up the computation you want to do, which then goes and runs in highly-optimized assembler implementations. You can look at Erlang being very similar when being used in its native domain for networking, using a slower language to decide how to route bytes around.

So it's quite possible for competent Erlang to beat out competent Java for such a task. Java optimized to within an inch of its life can certainly beat Erlang, but it's going to be a much uglier code base to get there.

You are making the error of comparing BEAM with a specific JVM implementation.

Not all available JVMs on the market have "a block the world GC" and the way multi-processing is done is implementation defined.

I could be wrong but I was under the impression that the four major JVM gc's all had a stop the world phase and while multi-processing is implementation specific I'm not aware of a robust pre-emptive multiprocessing model built for the JVM. If you have robust examples that are contrary to this I'd be interested in hearing about them.

Azul doesn't. It's not a differnt GC for the hotspot VM, it's a separate VM altogether

"Trivial" to write parallel programs in C?

You should stop doing whatever it is that you're doing, because you are in the top 0.00000000001% of programmers, in the Universe, and I want to work on whatever multi-billion dollar startup you undoubtedly have going on.

I said it is trivial to parallelise a C program that implements an embarrassingly parallel algorithm. Embarrassingly parallel means no resources need to be shared, and there needs to be no communication, so no need for locks or messages, even in C. That's why I say it'd be trivial to parallelise it. Just add a pthread_start. Of course almost no applications are embarrassingly parallel.

And then you also need to implement mailboxes to receive new data, supervisors to watch the threads and revive them when they fail, initialization methods to get back to the original state, etc.

You can trivialize any issue if you ignore all the facts, but it doesn't make your argument have any correctness.

I'm really surprised at the arguing in this thread.

Embarrassingly parallel algorithms are trivial in pretty much every language. That is why is embarrassing.

>And then you also need to implement mailboxes to receive new data

If you did, it is not embarrassingly parallel.

You know, a huge amount of parallel work is done in C/C++. It was the language used for multiple parallel algorithms courses in my university. Being able to do basic MPI type stuff in C does not make you a top programmer. And not being able to do an embarrassingly parallel algorithm in C would ring alarm bells.

People's obsession with language choice often cloud their judgement.

Parallel programs really are not that hard to write. The hard part is figuring out how to make a problem into embarrassing parallel. Once that's figured out, the coding is simple. In this case, the language choice won't help.

> Parallel programs really are not that hard to write.

Yes... but!

> The hard part is figuring out how to make a problem into embarrassing parallel

The other hard part is the debugging - especially when you fall short in the "embarrassingly" department.

It's not until C11 that the language itself had any notion of multithreading without reaching out to a separate standard - POSIX - and various nonstandard compiler extensions required to implement it all (e.g. for memory barriers to constrain the reordering of code)

I've further found it distressingly frequent that in porting code from one platform to another that basic primitives are plainly broken to the point where I can write unit tests to catch the issue - usually leveraging e.g. ARM hardware and it's weaker memory ordering semantics, a topic which few fully understand.

For example, a "lock free" circular queue one might use to implement a task system to roll your embarassingly parallel programs atop of. Which were indeed lock free - and volatile free - and atomic free - and memory barrier free - and used in parallel from multiple threads...

Even the experts occasionally forget a mutex lock or something. I play catchup with additional static analysis annotations when new compilers expose new options to catch this kind of thing, but...

Static analysis is, of course, very restricted in what it can give you answers to, although I'd still love to have more advanced tools widely available.

What I really consider a game-changer for concurrent programming are the dynamic checkers integrated into recent versions of GCC and Clang (and the Go compilers, too). Those make it super easy to flag data races and don't have any false positives. Caveats apply, naturally, as they can only flag those races that actually happen during a particular execution of your code.

While Valgrind and similar tools could do that for years, the difference with TSan is speed. It's fast enough that you can use it all the time during development. Same for AddressSanitizer and the undefined behaviour checkers.

You mistake concurrency with parallelism. If each of your "synchronous" call to a method is an asynchronous message, suddenly, all your program is embarasingly concurrent. Wait... was that what Alan Kay was saying with SmallTalk ? Would we have all lost our mind on RPC and synchronous calls for so long?

Yes. Yes we did.

If every call is an async message but the next step in your algorithm always depends upon the result/outcome of that async message you are programming a sequential algorithm wrapped in concurrent clothing.

I would claim that even if you use Erlang and distribute it across multiple processes your program doesn't actually exhibit concurrency. If you want to argue otherwise that's fine, but you just push the problem out: your program is "concurrent" but you gain zero benefits of concurrency, and when you distribute it across multiple cores to try to make it run in parallel it might be concurrent but by definition it will not be parallel.

Either way you want to slice it you have a big problem with this line of thought.

Edit: from my experience it is easy to write Erlang programs that exhibit something not exactly the same as, but similar to, this sort of behavior.

"Embarrassingly concurrent" is not embarrassingly parallel. Parallelism is not merely making things concurrent nor making synchronous calls into concurrent async calls.

once you are concurrent, what stop you from being distributed to multiple cores ?

I hate to do this but I have to quote you, "You mistake concurrency with parallelism."

Not really. Once again. What stop completely concurrent code to run on different schedulers ?

What you described is concurrency. What OP was talking about is parallelism.

>> Your Erlang program should just run N times faster on an N core processor

> But only if your program is embarrassingly parallel with at least N times available parallelism in the first place! If you have one of those it's already trivial to write a version that runs N times faster on N cores in C, Java, multi-process Python, whatever.

You've made two claims, one irrelevant and one false:

1. only if your program is embarrassingly parallel is irrelevant, because a almost every program is embarrassingly parallel in Erlang. The language is built around concurrency to the point that parts which wouldn't be obviously parallel in another language are in Erlang. Further, slow hashes in crypto have taught us that is actually quite difficult to make something which can't be parallelized.

2. it's trivial to write a program that's faster in X language I'm not sure how toot define trivial, but I've yet to find a language that can communicate between threads as performant-ly. Even languages like Clojure which use similar thread semantics can't do what Erlang can because the underlying threads aren't as lightweight. Spinning up a million threads in Erlang isn't even unusual, whereas in any of the languages you mention it's either crippling-ly slow (Python or Java) or very difficult to synchronize (C most, but Python and Java aren't easy).

I've shipped Erlang code and spoken at Erlang events and even I don't believe that every program in Erlang is embarrassingly parallel. There are very real limits on what you can do.

Erlang has never been a language about amazing performance. It's been about an environment and library that delivers amazing distributed concurrency with shockingly little effort.

This is an amazing accomplishment and should not be undersold. In making unsupported claims about how Erlang has magical parallelism sauce that somehow ignores Ahmdal's law we ignore the real benefits for a fiction.

>only if your program is embarrassingly parallel is irrelevant, because a almost every program is embarrassingly parallel in Erlang. The language is built around concurrency to the point that parts which wouldn't be obviously parallel in another language are in Erlang. Further, slow hashes in crypto have taught us that is actually quite difficult to make something which can't be parallelized.

OK. Not an Erlang user, but I can't let this statement go.

I've studied parallel numerical algorithms. Many/most of them will involve blocking because you're waiting for results from other nodes.

If you're saying Erlang has somehow found a way to do those numerical algorithms without having to wait, then I'd love to see all those textbooks rewritten.

Amdahl's Law reigns supreme.

I'm not devishard, but I parsed his statement slightly differently. He's not saying that Erlang makes things parallel magically. Rather, he's saying that Erlang forces tasks that /could/ be parallel to be parallel by default. Thus, Erlang will tend to maximize the sections of your program that are run in parallel compared to other languages.

Which would make the original commenter's point valid:

>Your Erlang program should just run N times faster on an N core processor

No, it won't. It will only be true for tasks that /could/ be (completely/embarrasingly) parallel (as you say). Which is kind of circular.

Not really. Think about every object you'd have in Java that's being passed around your system. Now imagine each of those objects are their own processes and you're passing around references to them.

Just on that one case, you've taken huge chunks of a linear execution pattern and parallelized it. Now make that your norm and amplify it to everything. Now realize that the message passing allows this mode of operation to spread each part of this workload over not only more cores but more machines across the network.

And then realize that you can deploy updates to this codes individual parts while other parts continue running without taking down the whole system.

Then you get Erlang.

None of what you said will prevent the need for waiting for the majority of numerical algorithms.

No one's disputing Erlang's prowess at parallelism. What the critic in this thread was saying was that you can only get Nx speedup on an N core processor for a limited set of algorithms. Most parallel algorithms will not fall in this category. Amdahl's law is a general truth - it doesn't matter what your architecture/language is. There is nothing special about Erlang that will make any parallel algorithm scale linearly with nodes.

On top of that, Erlang has a scheduler which reduces the need to rely on the operating system's concurrency model.

I'm interpreting "embarrassingly parallel" to mean that it's obvious the task can be parallelized, and I'm saying that many tasks where this isn't obvious in a more serial language are obvious in Erlang.

No, I'm not claiming Erlang breaks Amdahl's Law. I'm claiming that Amdahl's Law applies less often than people think it does.

The prescheduler caps the execution window for anything that would block. If one piece of the system would take a long time to finish, it doesn't interfere with the other parts of the system completing their work on schedule. That one blocking piece will finish more slowly, but ever other moving part in the system will keep responding as expected.

>If one piece of the system would take a long time to finish, it doesn't interfere with the other parts of the system completing their work on schedule. That one blocking piece will finish more slowly, but ever other moving part in the system will keep responding as expected.

That's how it works in pretty much any language that supports message passing. I used to do MPI programming in C. Nothing you said is not true for MPI in C. Later I did some MPI in Python. True there as well. If you have MPI in any language, it is true for that language.

My understanding is that those languages rely on cooperative scheduling within a thread, meaning that the running code has to relinquish control to the scheduler. Threads themselves are prescheduled at the OS layer but OS threads are much heavier and limited in how many can be running. A Java thread is 1024kb for example, compared to an Erlang process that's 0.5kb.

>My understanding is that those languages rely on cooperative scheduling within a thread

MPI doesn't require threads.

I think people here are confusing parallel programming with multithreaded programming. One is a subset of the other.

I'm not saying Amdahl's Law is wrong, I'm saying that it doesn't apply to as many problems as people think it does. I said "almost" for a reason.

> almost every program is embarrassingly parallel in Erlang

I can't agree with that, and it's key to my point. There are some problems which we just don't know how to make embarrassingly parallel. Take something classic like mesh triangulation or mesh refinement. Nobody knows how to make those embarrassingly parallel. If you write it in Erlang, it's still not going to be embarrassingly parallel. And it won't scale linearly to N times faster on N cores no matter which language you write it in.

So it's just not true to say that any Erlang program should scale linearly. If nobody on earth knows how to make mesh refinement scale linearly, how will Erlang do it?

Maybe you mean you wouldn't choose to write those programs in Erlang? Well then I think it's a meaningless claim to say Erlang will linearly scale your program, but only if it is a program which is naturally linearly scalable anyway. Erlang hasn't helped you do anything there so why make a claim about it?

"almost every program" != "any program"

WTF. How do you turn a problem which cannot be reduced into parallel sub-components into an Erlang program then? Are there just a huge class of computations that cannot be done with Erlang? Of course not.

Of course you can implement an inherently serial algorithm in Erlang. It just won't run N times faster on N cores.

> I've yet to find a language that can communicate between threads as performant-ly.

Minor nitpick: Erlang uses processes, not threads. Your comment doesn't explicitly say that Erlang uses threads though, so perhaps I'm being overly pedantic.

But an Erlang process is not the same as an OS process, or even an OS thread. It's managed by the Erlang VM and is very lightweight.

Reply on news.ycombinator.com