I think you mis-understood his point. The point is if you treat your green threads like regular threads and you properly release and acquire locks ("with" contexts help here too) then it won't happen. The other way is to use queues for example.
In most small examples, say updating a shared dictionary or list, many won't bother serializing green thread access to it. That should work fine.
> The problem is still that other co-routines can mess with your shared state, and you have no queues as to where they get to do that. Neither Twisted nor tulip has that problem.
greenlet based green threads dispatch underneath just like Twisted's reactor does based on a select/poll/epoll/kqueue system call. Like it is possible to update half the items in a dictionary, then call a function that does IO from a green thread, and another green thread gets dispatched and now updates the dictionary, so, now one thread sees a logically inconsistent shared piece of data.
Now replace the above threads with callbacks. You have a TCP listening socket (is it called a factory or protocol, I forgot my terminology already) that dispatches callbacks when data arrives. In those callbacks you update some part of the dictionary then realize you need data from a data base so you make a database call, get a deferred and add a callback to continue updating when data arrives back from database. In the meantime your original socket fires and another callback goes to your shared dictionary and now sees it in an inconsistent state.
This thing exists, I have used it a few number of times:
https://twistedmatrix.com/documents/8.2.0/api/twisted.intern...
It is a semaphore in Twisted. It is also used to reduce concurrency but it can be used to protecte shared data if needed. And I did need it in a couple of places.
This myth that callback based programming frees one from the need to use locks and somehow it magically infuses code with unicorns and easy shortcuts is very common. The only way to dispell it to use the source. Just read the source of twisted and see how it works. Look at greenlet and eventlet's hub's source (or gevent) and see how it works.
Yes, greenlet + monkey patching libraries can hide IO dispatching points your code. It is a valid downside. So to be safe you an just treat them as regular threads. But it allows you to use libraries -- one of the biggest reason one would pick Python for.
Now that we are on the subject, what is one sane way to drastically reduce the need for locks and semaphores? Use actors like Erlang does. A class with a thread and a queue attached to it. Then avoid sharing data instead send messages. You'll be hit with a serious performance issue because of memory copying, because well, there is no free lunch. Twisted and other callback mechanisms are certainly not a free lunch either.
> ... and you have no queues as to where they get to do that.
Heh heh, an accidental funny typo. As actually if you did have queues, you might not have to worry about shared data access ;-)
I'll happily admit that if you treat coroutines like threads and lock where you're supposed to, the failure modes are the same. However, since nothing appears to break if you get it wrong (synchronization-related failures that only show up under load, and when they do, they show up by silently clobbering some data), I suggest that there's way too much room for people to get it wrong there. (My personal anecdotal evidence appears to corroborate this.)
Furthermore, in many cases there are simple ways you can make a synchronous library look async in twisted: deferToThread is the most common/obvious one.
So, yes, you get all of that wonderful manual CSP stuff! I guess I just don't agree with you that that's a benefit.
Can you give a practical example of where you used a semaphore to synchronize shared mutable state access? I've only used it for, say, concurrency limiting, I think.
I'm actually very interested in actor-based concurrency, and I think many of my fellow Twisted developers with me (we have a good contingent of actor + obj-capa fanboys). I have a few systems where the reactor is pretty much that queue for all intents and purposes :)
Right, but by implication if you don't want them to happen and are afraid someone will override __iadd__, then by no means just use a lock. Both eventlet and gevent I believe will monkeypatch threading module to now become "green" threading module.
> However, since nothing appears to break if you get it wrong
But nothing would appear to break in the deferreds case I suggested either. It would still be accessing shared data in an inconsistent way.
> I suggest that there's way too much room for people to get it wrong there.
That is why a private queue with a green thread attached to it (a actor!) is a better way to build large concurrent systems.
> Furthermore, in many cases there are simple ways you can make a synchronous library look async in twisted: deferToThread is the most common/obvious one.
Yes it is there and it works but it is clunky. After switching to gevent we, for example, got about about a third to a half as much code as before. A lot it was boiler plate code. inlineCallbacks and yields. Our own counter-parts to already existing libraries to make a Twisted alternative.
> Can you give a practical example of where you used a semaphore to synchronize shared mutable state access? I've only used it for, say, concurrency limiting, I think
Pretty much the exact problem I described above. A server was processing requests from the user. It was listening on a TCP socket and reading commands delimited by a new line. One of the commands initiated a business operation consisting of multiple steps. That operation involved going to some databases, calling a number of sub-processes all of which was handled via callbacks. As the operation was taking place, a local piece state data was updated, tracking the progress etc.
Now if before that request finishes, a new command comes. And starts firing the same sequence of callbacks, but the previous one hasn't finished yet.
So that's it. You are accessing a shared piece of data which is now in a logically inconsistent state in respect to your internal business logic and you have no idea it happened (just like with threads).
So you need to acquire a deferred lock and then release it when done.
> I'm actually very interested in actor-based concurrency, and I think many of my fellow Twisted developers with me (we have a good contingent of actor + obj-capa fanboys). I have a few systems where the reactor is pretty much that queue for all intents and purposes :)
Yeah we switched to that. Actors are basically an object attached to a green thread that a main method. Actor has a queue. Outside processes can send it messages by putting them in the queue.. Actor in main method can do self.receive() for example to get messages off the queue. Basically what Erlang does. You have choice if you want to duplicate data (make copies before putting in queues) or use objects with locks to serialize access to them (say if they are large and copies won't work).
In Golang, you can avoid the serious performance issue by sending a pointer over a channel and treating that as a transfer of ownership of the data.