Hacker News Reader

I only have around ~1 year of twisted experience, but I felt like the viral aspect of twisted was even worse than that.

Not only everything needs to be asynchronous, but using deferred leaks everywhere in your API. Deferred callback firing mechanism 'recursively' passing the result of the previous callback is the most egregious aspect of it. Yet, I think deferred is by far the most readable/maintanable method to write async in twisted. For example, inlineCallbacks looks nice on paper, until you start using them and realize stacktraces are often wrong, and that you loose a lot of concurrency by forcing things that could run in // to be serialized.

I think explicit async is simply the wrong model for a dynamic language. Often, async is just an optimization, and more efforts should be spent on how to make it happen under the hood.

> but using deferred leaks everywhere in your API

Exactly. To expand on it. Glyph and/or Guido, I have heard say how the fact that IO is happening is very important and one should an obvious explicit callback in the algorithm. (Those using yield based concurrency also agree but just say to stick a yield in there instead of a deferred+callback).

I disagree. If one particular API function happens to use Twisted shouldn't affect the user or caller of the functions. Using a particular IO library should be an implementation detail.

Agreed. I make heavy use of greenlet/gevent, and one of the primary advantages is that you can import and use naive sequential libraries, and they won't block your process (assuming they're pure python so the monkey patching works properly).

As for "not knowing when you might switch coroutine" - I contend that if it matters, you're doing it wrong. Note that, unlike proper threading, things can't happen at the same time / interleaved with simple statements, so it's still very easy to do atomic actions without special locking - eg. there's no way self.x += 1 could switch in the middle (unless someone is badly abusing the __add__ method).

Oh I agree with you. I just gave that point in quotes to illustrate the common response that Guido and others have given to greenlet based IO mechanisms.

The problem is still that other coroutines can mess with your shared state, and you have no queues as to where they get to do that. Neither Twisted nor tulip has that problem. The fact that it can't happen in __iadd__ is just one place where it can't happen. That's a far cry from not having to care about this problem.

> The problem is still that other coroutines can mess with your shared state,

I think you mis-understood his point. The point is if you treat your green threads like regular threads and you properly release and acquire locks ("with" contexts help here too) then it won't happen. The other way is to use queues for example.

In most small examples, say updating a shared dictionary or list, many won't bother serializing green thread access to it. That should work fine.

> The problem is still that other co-routines can mess with your shared state, and you have no queues as to where they get to do that. Neither Twisted nor tulip has that problem.

greenlet based green threads dispatch underneath just like Twisted's reactor does based on a select/poll/epoll/kqueue system call. Like it is possible to update half the items in a dictionary, then call a function that does IO from a green thread, and another green thread gets dispatched and now updates the dictionary, so, now one thread sees a logically inconsistent shared piece of data.

Now replace the above threads with callbacks. You have a TCP listening socket (is it called a factory or protocol, I forgot my terminology already) that dispatches callbacks when data arrives. In those callbacks you update some part of the dictionary then realize you need data from a data base so you make a database call, get a deferred and add a callback to continue updating when data arrives back from database. In the meantime your original socket fires and another callback goes to your shared dictionary and now sees it in an inconsistent state.

This thing exists, I have used it a few number of times:

https://twistedmatrix.com/documents/8.2.0/api/twisted.intern...

It is a semaphore in Twisted. It is also used to reduce concurrency but it can be used to protecte shared data if needed. And I did need it in a couple of places.

This myth that callback based programming frees one from the need to use locks and somehow it magically infuses code with unicorns and easy shortcuts is very common. The only way to dispell it to use the source. Just read the source of twisted and see how it works. Look at greenlet and eventlet's hub's source (or gevent) and see how it works.

Yes, greenlet + monkey patching libraries can hide IO dispatching points your code. It is a valid downside. So to be safe you an just treat them as regular threads. But it allows you to use libraries -- one of the biggest reason one would pick Python for.

Now that we are on the subject, what is one sane way to drastically reduce the need for locks and semaphores? Use actors like Erlang does. A class with a thread and a queue attached to it. Then avoid sharing data instead send messages. You'll be hit with a serious performance issue because of memory copying, because well, there is no free lunch. Twisted and other callback mechanisms are certainly not a free lunch either.

> ... and you have no queues as to where they get to do that.

Heh heh, an accidental funny typo. As actually if you did have queues, you might not have to worry about shared data access ;-)

I don't see the parent suggest that you use treat coroutines like threads at all. In fact, he's explicitly suggesting that the same issues that exist with threads don't happen, or are at least significantly diminished.

I'll happily admit that if you treat coroutines like threads and lock where you're supposed to, the failure modes are the same. However, since nothing appears to break if you get it wrong (synchronization-related failures that only show up under load, and when they do, they show up by silently clobbering some data), I suggest that there's way too much room for people to get it wrong there. (My personal anecdotal evidence appears to corroborate this.)

Furthermore, in many cases there are simple ways you can make a synchronous library look async in twisted: deferToThread is the most common/obvious one.

So, yes, you get all of that wonderful manual CSP stuff! I guess I just don't agree with you that that's a benefit.

Can you give a practical example of where you used a semaphore to synchronize shared mutable state access? I've only used it for, say, concurrency limiting, I think.

I'm actually very interested in actor-based concurrency, and I think many of my fellow Twisted developers with me (we have a good contingent of actor + obj-capa fanboys). I have a few systems where the reactor is pretty much that queue for all intents and purposes :)

>I don't see the parent suggest that you use treat coroutines like threads at all. In fact, he's explicitly suggesting that the same issues that exist with threads don't happen, or are at least significantly diminished.

Right, but by implication if you don't want them to happen and are afraid someone will override __iadd__, then by no means just use a lock. Both eventlet and gevent I believe will monkeypatch threading module to now become "green" threading module.

> However, since nothing appears to break if you get it wrong

But nothing would appear to break in the deferreds case I suggested either. It would still be accessing shared data in an inconsistent way.

> I suggest that there's way too much room for people to get it wrong there.

That is why a private queue with a green thread attached to it (a actor!) is a better way to build large concurrent systems.

> Furthermore, in many cases there are simple ways you can make a synchronous library look async in twisted: deferToThread is the most common/obvious one.

Yes it is there and it works but it is clunky. After switching to gevent we, for example, got about about a third to a half as much code as before. A lot it was boiler plate code. inlineCallbacks and yields. Our own counter-parts to already existing libraries to make a Twisted alternative.

> Can you give a practical example of where you used a semaphore to synchronize shared mutable state access? I've only used it for, say, concurrency limiting, I think

Pretty much the exact problem I described above. A server was processing requests from the user. It was listening on a TCP socket and reading commands delimited by a new line. One of the commands initiated a business operation consisting of multiple steps. That operation involved going to some databases, calling a number of sub-processes all of which was handled via callbacks. As the operation was taking place, a local piece state data was updated, tracking the progress etc.

Now if before that request finishes, a new command comes. And starts firing the same sequence of callbacks, but the previous one hasn't finished yet.

So that's it. You are accessing a shared piece of data which is now in a logically inconsistent state in respect to your internal business logic and you have no idea it happened (just like with threads).

So you need to acquire a deferred lock and then release it when done.

> I'm actually very interested in actor-based concurrency, and I think many of my fellow Twisted developers with me (we have a good contingent of actor + obj-capa fanboys). I have a few systems where the reactor is pretty much that queue for all intents and purposes :)

Yeah we switched to that. Actors are basically an object attached to a green thread that a main method. Actor has a queue. Outside processes can send it messages by putting them in the queue.. Actor in main method can do self.receive() for example to get messages off the queue. Basically what Erlang does. You have choice if you want to duplicate data (make copies before putting in queues) or use objects with locks to serialize access to them (say if they are large and copies won't work).

Now that we are on the subject, what is one sane way to drastically reduce the need for locks and semaphores? Use actors like Erlang does. A class with a thread and a queue attached to it. Then avoid sharing data instead send messages. You'll be hit with a serious performance issue because of memory copying, because well, there is no free lunch.

In Golang, you can avoid the serious performance issue by sending a pointer over a channel and treating that as a transfer of ownership of the data.

Yap. But it breaks isolation. Some places need that. There is no free lunch again. Otherwise semantically it is accessing the same data from multiple concurrency contexts. Once that concurrency grows it becomes very hard to reason about who and when changed what if you get a crash. Setting write watch-points in gdb is one way I found to do it in C++ for example.

You can always pass around pointers to immutable data structures which only implement read interfaces.

Exactly. Or maybe copy-on-duplicate-access detection. So keep a single copy on heap but when accessed first time by second context/thread then make a copy of it. Azul's Java GC plays some of these games with read barriers. They detect reads to certain pointers and set callbacks to run custom code when they happen.

On the other hand, the nice thing about promises in tulip is that they are generic enough that you can hook up an existing async library, so long as it also deals with promises - even if they're a different kind of promise. For example, callback-based native APIs exposed by the OS or its standard libraries.

The viral nature of twisted is not limited to twisted and shows up in lots of places. It's actually a consequence of the problem that "concurrency is Hard".

There is a tradeoff to be made between pre-emptive and cooperative multitasking. Every platform must choose one or the other or a mix of both, and then deal with the consequences. Threads, of course, have a well-known class of very dangerous bugs that make using them difficult. Processes make pre-emptive multitasking much safer by erecting certain barriers, but at the cost of being much more tedious.

These days, callbacks, events and other forms of cooperative multitasking are the new shiny. They avoid the big problems of threads and multiple processes, but have some problems of their own. First and simplest is that you can only use one CPU per process without mixing in threads or multiple processes. Second, if something doesn't cooperate, it blocks everything else. You have to be very careful about when and where you yield the CPU. Because of that, everything must be event-driven or use callbacks, or you risk trashing your throughput and/or latency.

Threads have a similar problem: if you use them, you're stuck using thread-safe libraries. It's not as bad as the cooperative case though. The thread-safety of a library can be independant of the code that uses it. In the cooperative case, you don't have that independence. You're not just stuck with cooperative-safe libraries. You are restricted to events or callbacks that are compatible with the cooperative framework used in your application. If your application uses events, then the library must use events instead of callbacks, and it must use the same types of events to plug into your event loop. If not, then you need a compatibility layer. Some cooperative languages reduce the problem by mandating how events and/or callbacks work for all libraries. Python doesn't have that, which where the viral nature of twisted comes from. That's why you're talking about twisted-compatible libraries and not just python libraries, and why this a non-issue with node.js database libraries.

edit: wording

> Threads have a similar problem: if you use them, you're stuck using thread-safe libraries.

One pattern that I like personally is the actor pattern. Building systems out of independent isolated concurrent components that share data _mostly_ by sending messages. Erlang at the top of this paradigm but one can replicate that in other languages and libraries. There is Akka for Scala and others as well. It is basically an object, a thread running one of its methods and a queue. This object also has an address or handle. Objects communicate by sending messages to each other's handles.

Now of course as you say one needs to use libraries that allow this concurrency.

So far I find this pattern best matches real world scenarios. Real world is built of many concurrent pieces but each individual piece could be a small sequential island. This doesn't apply just to intelligent agents but to server requests as well. Each client connection and context can be sequential but it can run concurrently with other potential client context.

Ideally if one such client request crashes it shouldn't crash the main OS process. If one such client goes into a tight loop it shouldn't block the rest of the requests. This is hard to do without sacrificing heavily by copying data and or using isolated heaps (can use real OS processes up to some point, but also it is very resource heavy).

> That's why you're talking about twisted-compatible libraries and not just python libraries, and why this a non-issue with node.js database libraries.

Very good point. I didn't think of it that way. Say if Python was born with Twisted bolted on to start we wouldn't be having this problem. But it is interesting how a mono-culture of concurrency is a good thing in certain respects. And to give Guido props I think he is trying to bring that in -- as with Python, it is desirable to have one best way to do the same thing. Concurrency wasn't like that historically. And he is trying to fix it. I personally disagree with the approach he took (either in promoting or looking at deferred or explicit co-routines based on generators)

> I can. Twisted is viral (not in a good way always). Once you start using Twisted you are doomed and stuck in a parallel reality of Twisted-only libraries forever. I have used Twisted professionally for 4+ years, this is not something I just read on a blog.

This seems to be a common sentiment. I'm writing a talk which I'm submitting to PyCon 2014 that hopes to demonstrate that that isn't (or doesn't have to be) true.

I've made the repo[1] and preliminary slides[2] available :)

[1]: https://github.com/lvh/TwistedMixing [2]: http://blog.lvh.io/TwistedMixing

That's a very good presentation. Thanks for sharing.

Yes it is possible to use deferToThread and I like run_in_reactor trick. Very nice. eventlet by the way was integrated with a Twisted reactor as well. So it could co-operate with it. But, it was never a question of "impossible" it more of a question of it being "annoying".

There seems to be a bug on "Deferred vs blocking API" slide:

  d = async_read()
  d.addCallbacks(on_result, on_failure)

on_failure() is not called if on_result() raises an exception but it might be called in the blocking code:

  try:
    result = blocking_read()
    on_result(result)
  except SomeError as e:
    on_failure(e)

The blocking analog of the addCallbacks() call could be:

  try:
    result = blocking_read()
  except SomeError as e:
    on_failure(e)
  else:
    on_result(result) # on_failure() is not called

Oops! Good catch :)