Unladen Swallow 2009Q3 Released with LLVM optimized Python
code.google.com
code.google.com
Something I noticed though, in the Project Plan page:
http://code.google.com/p/unladen-swallow/wiki/ProjectPlan#20...
They state that one of the goals for "2009Q3 and beyond" is "to remove the GIL and fix the state of multithreading in Python." Which I think is an awesome and ambitious goal that I would find very useful as will many others.
So I hoped the GIL thing would fall into the "2009Q3", but apparently it's in the "beyond" category. Anyway, they need to update that project plan.
lock();
++value;
unlock();
The wiki for it is a good start: http://en.wikipedia.org/wiki/Non-blocking_synchronization And I'm going to go ahead and plug the work I've done in the area: http://people.cs.vt.edu/~scschnei/streamflow/
Many of the lock-free algorithms use CAS native machine instructions, however. What exactly do you mean by "reasoning about them is hard"? Do you mean in formal methods sense, or just difficult to grasp in general? I am just glancing over your ISMM'06 paper which looks very interesting--it seems you are using CAS instructions there, too (which makes me wonder that I was probably too unspecific in my "simplifying locking", which at least for my sloppy thinking includes "lock-free" algorithms too, sorry for that...)
Reasoning about lock-free algorithms is hard because there is no mutual exclusion. Locks enforce a critical section; you guarantee that only a single thread will execute in a critical section at any given time. This allows you to change state local to that critical section without worrying about other threads interfering.
Lock-free algorithms have no critical sections. All threads can execute all instructions at all times. This means that when you update shared state, you have to think about the implications of what happens when another thread touches the same state this thread is trying to update. Keeping this level of concurrency in your head at all times is difficult, which means that reasoning about the correctness of an algorithm is difficult.
It has a Queue interface for passing objects, but also allows shared state.
You'll find their publish-subscribe message passing is quite simplistic, and the support is great.
Does that math make sense in some way I'm not seeing?
This 930% basically means that 2009Q3 can use as little as ~1/10 of the memory of 2009Q2.
(Edit: Or did you mean that the 2009Q2->Q3 savings was 930% greater than the Q1->Q2 savings?)
From the release notes:
Lowlights:
* LLVM's JIT and other infrastructure needed more work than was expected. As a result, we did not have time to improve performance as much as we would have liked.
* Memory usage is still 2-3x that of Python 2.6.1. However, there is more overhead that can be eliminated for the 2009Q4 release.