Multithreading Magic - Why Everything You Thought You Knew about Concurrency... | Hacker News Reader

Multithreading Magic - Why Everything You Thought You Knew about Concurrency...

slideshare.net

66 pointsadulau36 comments

It's great to get feedback on the talk. Impossible to even introduce 0MQ properly in 10 minutes, so this presentation was really just to drive discussion. I'd apologise for the polemical title but that's just how I talk.

To some of the comments here...

- Parallel programming may be taught in CompSci courses, for sure, and it's a popular use for 0MQ, but (a) it's not applied to mainstream application development, and (b) it is not designed to scale to networks of any size.

- The Actor model is not key to building a successful message-based concurrent application, but it's a good example, and helps people understand that there are alternatives to shared-state concurrency.

- It's IMO useless to ask people to learn Erlang or Scala to get message based concurrency. People use Java, C, C#, PHP, Python, and probably still COBOL somewhere. So the challenge is how to give this mainstream a toolset that lets them build large parallel applications.

Perhaps next year at FOSDEM we can do a devroom and take the time to see a real application evolve from a simple stand-alone process into a real distributed one. 0MQ is very hard to grasp as a theory, one needs to actually use it for a few days before the beauty a cheap, universal, fast, intelligent, easy to use, and asynchronous queuing messaging fabric hits home.

Cheers!

I've glanced at the 0mq sources a bit, and I'm curious about the error handling. For example, zmq::tcp_socket_t::read() handles following errnos from recv(): EAGAIN, EWOULDBLOCK, EINTR, ECONNRESET and EPIPE. If recv() fails for any other reason (e.g. ENOBUFS, ETIMEDOUT) and returns -1, the following is performed:

  errno_assert (nbytes != -1);

which translates to

  #define errno_assert(x) \
    do {\
        if (unlikely (!(x))) {\
            perror (NULL);\
            fprintf (stderr, "%s (%s:%d)\n", #x, __FILE__, __LINE__);\
            abort ();\
        }\
    } while (false)

The first thing I noticed is the use of perror() and fprintf() for error reporting. Any daemon that closed its stdin/out/err and not had them redirected to /dev/null will most likely output errors to other file descriptors (sockets, other open files etc.). Is this by design or something that was missed?

Secondly, the use of abort() in an API is probably also not the best choice. Say I'm writing a (robust) database server and an error happens on one connection - the error will bring the whole server down, which kind of defeats the purpose. Is this also by design?

[Edit: formatting]

I like ZeroMQ but this slideshare doesn't deliver any substance at all. For an overview, skip it and check out the white paper (if you can call it that) cowritten by the same guy (Pieter Hintjens) instead:

http://www.zeromq.org/whitepapers:multithreading-magic

These slides were part of a "lightning talk" on ZeroMQ (video here: http://streaming.fosdem.org:8000/ferrer/0mq.ogv ). So I think Pieter just wanted to make the audience curious and not dive into the details. I do agree that it might be better to use more slides for patterns and use cases and fewer generic/actor-slides. Also the argument that it's not taught in university was a bit weak.

There's a copy of the video here: http://www.youtube.com/watch?v=CCBYzKfmQ4U

The FOSDEM streaming server seems to be timing out right now.

Dunno, made perfect sense to me. It's just that many (most?) programmers, in my own experience, don't fully appreciate the insanity that is traditional "shared data" concurrency.

What I see is that they program something buggy, but where the bugs are rare enough to make the program work most times.

For many types of problems, shared state concurrency is not the nightmare it's often made out to be. For example, if your problem can fit into the consumer producer model then it's really easy to make that work on the jvm today.

The most widely used languages, C and C++, do not offer any support for concurrency. Programmers roll their own by using threading APIs. Languages that do support concurrency, such as Java, Python, .NET, and Ruby, do it in a brute-force manner. Depending on the implementation - there are over a dozen Ruby interpreters, for example - they may offer "green threading" or true multithreading. Neither approach scales, due to reliance on locks. It is left to exotic languages like Erlang to do it right.

You know how I wrote my first concurrent programs? Create threads with pthread_create(3), and pass ownership of objects by sending pointers over connections created with pipe(3). Which sounds an awful lot like this cure-all actor model (which I hadn't heard of yet) that requires languages almost nobody's ever heard of.

Of all the approaches taken to multithreading, only one is known to work properly. That means, it scales to any number of cores, ...

clone(2) and socket(7)

... avoids all locks,...

Your threads/actors still block waiting for messages. I've seen an example of someone using this to write a mutex in Erlang.

... costs little more than conventional single-threaded programming,...

Probably depends how much you have to twist your code to fit this functional+actor model.

... is easy to learn,...

If you discount having to learn a whole new language and turn your code inside-out.

... and does not crash in strange ways.

Instead of crashing, it hangs because you accidentally wrote a mutex.

I've built quite a few actor systems in C++ for real-time platforms. You use a threading library like pthreads, which works fine in C++, use thread safe libraries, create an Actor object with a queue that encapsulates a thread, and then have your async messaging system route and drop messages in the right actor queue. As long as application writers can manage their impulses to share state between the threads it works excellently. The only problem is usually the shared state inherent in the memory allocator, so some sort of thread based memory allocation scheme works best. You also get to choose your response latency via priority, which is something you don't get with Erlang.

Keep in mind that the original point of erlang was "reliable software", not "high performance" (quotes because those are fuzzy terms) - there are many classes of errors that simply cannot happen in erlang that happen all the time in real-world C++ programming. You also get distributed computation for free via a well-tested VM.

No doubt, I just wanted to point out that C++ does support concurrency and in some use cases can even be superior.

> Your threads/actors still block waiting for messages. I've seen an example of someone using this to write a mutex in Erlang.

This statement suggests to me that you've got some fundamental misunderstandings about the nature of actor based programming. Your actor may block on its message queue (and an infinite computation would mean you'd never consult the queue again), but during a message wait it doesn't necessarily block a thread. That's sort of the point of a good Actor model implementation.

If you write a series of symmetric calls that create a dependency, then yes, you will deadlock; the Actor model can't work its magic when you explicitly model blocking calls.

Perhaps you don't know this, but every lock&block system has an equivalently performant Actor-based system, and vice-versa. (See: http://www.sics.se/~adam/pt/duality78.pdf) What this means, for programmers, is that whichever model fits our current problem best is the one to use. We can make either one performant.

P.S., Just because you haven't heard of these languages doesn't mean they are not widely used. What does your personal experience have to do with the viability of a technique?

> Your actor may block on its message queue (and an infinite computation would mean you'd never consult the queue again), but during a message wait it doesn't necessarily block a thread. That's sort of the point of a good Actor model implementation.

A thread can block on a syscall or a "while(true);", but won't necessarily idle one of your CPU cores. That's likewise the point of a good task scheduler.

Actors are threads, just they tend to be implemented in userland rather than as OS threads.

> Perhaps you don't know this, but every lock&block system has an equivalently performant Actor-based system, and vice-versa.

Knowing that they're equivalent (up to "frictionless surface" assumptions for performance, ignoring what primitives the hardware provides) is a significant part of why I don't take claims of the actor model's magicalness seriously.

> P.S., Just because you haven't heard of these languages doesn't mean they are not widely used. What does your personal experience have to do with the viability of a technique?

I have heard of them, always in terms of "this is awesome, what's wrong with people that almost nobody actually uses it?".

What I got out of this presentation: "You thought the only way to do concurrency was by using synchronization primitives? Surprise! You can also do it by message-passing between processes!"

What I think of this: Say whaaaaaaaaaaat? Message-passing as a concurrency model? That's crazy talk! Brilliant crazy talk! I wish this was such common knowledge that it was being taught in undergraduate "Parallel Programming" courses everywhere. Oh wait... IT TOTALLY IS!

Pfft - I've seen way too many code bases with random threading code.

Speaking of Java here - what's needed is that it's simply not possible to write new Thread(foo).start(); - the message passing model has to be enforced.

I mean, read the Java documentation! What they say about threading is this: "Something slow? Just throw it in a background thread!". It's inane.

The carryover from computer science education to practice is so weak that you have to teach and persuade all over again. People leave school, get a job, and spend five years only using knowledge from their sophomore "Data Structures in C" class while struggling to catch up on all the critical stuff that their CS program taught them nothing about (SQL, source control, release management, etc.) Having misunderstood the purpose of undergraduate education, they believe this experience discredits everything they were taught in school, so they hold you to a higher threshold of proof if you're pitching ideas they heard in upper-level CS courses than if you're pitching ideas they've never heard before.

Is that really true? I meet very few people, even fresh out of good schools, who have any practical knowledge of the Actor model. Many have heard of it, but most don't know that about even basic concepts surrounding it like Needham's Duality.

While my CDP class didn't explicitly discuss the Actor Model, message-passing as a concurrency model was studied and we did implement algorithms using it.

How basic is the Actor Model for implementing concurrency using message-passing, besides the obvious advantage of formally defining a common language of terms and proven theorems for what you're doing?

Whaa?? "Concurrency is hard... Actors... ZeroMQ... End scene!"

This is far from enlightening.

:-) If you hope for enlightenment from watching a 10-minute talk, you're going to be disappointed. I do like your summary of the presentation... "Concurrency is hard... Actors... ZeroMQ... End scene!". Nice.

Enlightenment requires that you change in some way. For me, like many people who have taken the step of downloading and learning 0MQ, the feeling of "wow, this is too easy, where's the catch?" comes only after a few days writing code, and then your brain makes a slight adjustment, and it's all obvious, and that is a small but real enlightenment.

For what it is worth this is the model Redis, and Redis cluster, are beating on, and why we are not going multi threaded.

You meant "betting on", right? Not trying to be that guy, just wanted to make sure I understood your comment.

Yes, thanks and sorry for the error. Unfortunately it is no longer possible to edit the comment (but I agree with the edit timeout FWIW).

CSP (Communicating Sequential Processes) also seem similar:

http://en.wikipedia.org/wiki/Communicating_sequential_proces...

There's a patch for Python that does CSP!

http://code.google.com/p/python-csp/

Once upon a time there were even dedicated CPUs that had their own programming language based on CSP:

http://en.wikipedia.org/wiki/Transputer

http://en.wikipedia.org/wiki/Occam_%28programming_language%2...

ZeroMQ is fantastic for low latency communication between processes, but probably not the best choice for building 'actor' based systems. The slides don't really provide much basis for the argument.

Erlang & Scala (for a more object-oriented feel) are both excellent for building actor based systems.

Message passing is not a golden bullet. It is a synchronization primitive for distributed systems. Basically on the same level as mutexes, semaphores, and monitors.

Badly designed you are in the same concurrency hell.

Badly designed, anything can become hell. Crazy has no limits. The point of the talk was that message passing is fundamentally not at the same level as mutexes, semaphores, and monitors. Shared state does not scale linearly, no matter how well you use it, whereas message passing does, given cheap, fast, universal messaging.

A message queue is shared state. It may be more convenient shared state, and since it only needs to be got right once it might be more bulletproof shared state, but you have multiple threads that are serialising on access to a queue no matter how it is implemented.

You're right that a message queue is shared state and has to be gotten right but that can be done (and ZeroMQ does it) without locks. It's thus invisible to application developers. You're wrong to say that multiple threads serialize on a queue no matter how it's implemented. ZeroMQ lets threads write full speed to a queue while another thread reads full speed from the queue, without wait states.

I wish Pieter had a bit less sensational title than "Multithreading Magic - Why Everything You Thought You Knew about Concurrency is Bogus, if not actually Totally Insane" considering that the slides contain only common knowledge.

Having read up a bit more on ZeroMQ, I have to ask: How is this, if at all, superior to, say, MPI[1]?

[1]http://en.wikipedia.org/wiki/Message_Passing_Interface

Significantly simpler, faster, and more general. MPI is like one of the patterns ZeroMQ provides (the pipeline pattern) with a lot on top. That makes it great for parallel programming, but not so great for (e.g.) data distribution (pubsub) and other forms of message queuing.

...the bar man said "we don't serve event loops here"

Node.js walks into a bar...

Naming Magic: Why Every Other Link Including This One Is A Bait

Whenever I need an Actor, I go with Nathan Fillion.

Reply on news.ycombinator.com