Hacker News Reader

If you carefully read what I said, I think you'll find there's less disagreement between what you posted and what I said than your tone suggests you think. I did, for instance, mention C methods, but you seem to imply that that matters to all Python implementations. It only matters to CPython. Barring what last I knew was some fairly experimental stuff from PyPy, Python extensions are not generally compatible between the implementations. (Where you see libraries that appear to work on multiple implementations, AIUI you're actually seeing multiple implementations of the binding code.)

Personally my feeling is that if you care enough about performance to be threading in Python, you care enough about performance to not be using Python anyhow. It's a nice language, it's by far my favorite of the scripting languages of its family, but even with all the PyPy JIT and other magic tech words you can throw at it, it's still a slow language. When that doesn't matter, great; when it does, you eventually reach the point where you're stuck. (I've never been overwhelmed by the "implement it in C then" argument, for many reasons, and nowadays to an increasing degree if you're going to implement your "core" in another language for speed, there's a language you can choose that is both nice to use and already fast for your task, and that's only going to get more true as time goes on.)

The C methods do unfortunately matter to all implementations. They gave thread safety and almost all Python code in the wild relies on that property, so it's become an undocumented property of the Python language specification and an intrinsic flaw of the language itself.

Reference counting and the C API are properties of the CPython implementation, not Python the language. Other Python implementations do NOT use reference counting and don't offer the C API, but do make guarantees about atomic operations. I think the problem with backward compatibility is with wanting to preserve the C API and the existing extension modules implemented in C, not necessarily with Python the language.

I should have been clearer that the other Pythons don't use reference counting. Preserving the C API can be done, see Ironclad[1] which allows using C extensions in IronPython.

But assuming threadsafe collections is still a problem - that will never match single-threaded GIL Python running in multiple processes unless some efficient way can be found to optimistically make the collections not threadsafe and only pay the synchronization costs when they're really shared between threads. I think that's a solvable problem, but not with the STM approach PyPy took. Still performance will likely lag the mutli-process solution which doesn't have that cost.

I would rather see a Python that introduces a way to allocate PyObjects in shared memory and with no thread safety. Since it's a new system, that can be done without breaking compatibility. Then sharing between processes could be as easy as sharing between threads and just as efficient. The GIL could stay in that case and the C API could be left unchanged.

[1] https://github.com/IronLanguages/ironclad

all operations on Python collections like lists and dicts, are atomic, making those collections safe to share and modify across threads. Most python code in the wild that utilizes threads makes use of those guarantees. So you either use fine-grained locking, or lock-free data structures, or STM, etc to protect ALL collection instances, or you've basically made a Python 3000 that breaks compatibility with all existing Python code.

Do IronPython and Jython, Python implementations which do not have a GIL, use fine-grained locking? How is multi-threaded performance on these implementations? Surely not as bad as this talk implies CPython with fine-grained locking would be?

IronPython uses a combination of lock-free techniques and fine-grained locking. Reference counting is of course replaced with the .NET runtime GC. Single-threaded and multi-threaded performance suffers since this approach means pessimistically synchronizing all accesses, including those that are single-threaded.

You should watch the video. I won't try to repeat it all here, but Hastings identifies the performance problem areas and lays out some plausible directions for tackling it, and a plausible implementation/changeover path that doesn't create another big breaking change.