CPython internals: A ten-hour codewalk through the Python interpreter (2015)
youtube.com
youtube.com
More remarkable is the fact that antirez updated the documentation in response to a post in Reddit. https://www.reddit.com/r/redis/comments/3re0aw/any_pointers_... Thank you antirez! :-)
That is, I think local variables and constants are looked up by a small integer which the CPython compiler produces by stack analysis.
But any globals must be looked up by name: functions, classes, modules, global variables. And methods on classes, attributes on classes.
I'd be interested to get clarity on that, and any pointers to relevant code/docs. Is this addressed in the videos? I have looked through the CPython source a lot, and even patched it, but the lookups are a little hard to follow. I've played with the "dis" module and code objects.
EDIT: Answering my own question, it seems like I was confused about the index into co_names, which is a small integer into a list of strings, and then the lookup of that string. So it's a 2-step process?
LOAD_NAME, LOAD_ATTR, and LOAD_GLOBAL are all lookups by name, and are used for everything else: globals, object attributes and methods, modules, etc.
It seems that if Python had a static module system, all the lookups by name could be compiled down into lookups by number.
https://docs.python.org/3/library/dis.html#python-bytecode-i...
https://www.python.org/dev/peps/pep-0509/
Dictionaries got some sweet upgrades for v3.6.
I love Python, and use it a lot for ETL type work, but if threading worked well, I could/would possibly use it for far more purposes.
Because with multiprocessing and greenlets, 99.99% of concurrency problems are trivilially solved by current Cython.
So if I want to call two Fourier transform functions at the same time in Python I can, because neither of them is implemented in Python and so they don't hold the GIL.
That's the kind of parallelism use case I most often see come up, so although the GIL dismayed me early on I've come to see it as pretty irrelevant.
But maybe it makes more sense for other applications, for the performance critical parts of the code to be actual pure Python. I do mostly numerical simulations, so pure Python is usually a non-starter, you fix that long before you think about parallelism.
Day to day, I'm writing Python code which is actually parallel because a large fraction of the run time is dominated by the by things that aren't pure Python. I suspect this is true even for people who are not going out of their way to make it true. It's simply the case that most RDBM systems, Fourier transforms, etc with Python bindings are not written in Python.
The GIL sounds scary, but I think people overestimate the fraction of time it is actually held in their code.
I'm writing a transpiler that uses global information from codebases, and so it transpiles potentially hundreds of files at once and creates rather complex data structures. Compute bound for quite a while, so I tried speeding it up with multiprocessing (since multithreading would be useless). But with multiprocessing it took longer to serialize/deserialize the complex datastructures for each process, so I had to give up. Next time I have time for this I'd probably try to use Jython as a drop-in replacement and see whether I can get it to run with GIL-less multithreading.
(I'm pretty sure this is the video I'm thinking of) It's 30m, but worth it if you're interested. Not sure what progress has been made since then.
Also, most of my Python scripts run on both Linux and Windows, so I have that restriction, as well.
If your code is CPU bound and you're using native Python, you're going to be making a tonne of heap allocations and pointer dereferences. This will be very slow.
If you implement the relevant stuff in Cython, even without using multi-threading you'll likely see 10x performance improvement, and can often see up to 100x.
Removing the GIL makes Python worse at the stuff it's good at, for questionable improvements in the areas Python is really terrible. This is not a good trade.
A interpreted language does not need to be compiled into bytecode. Some languages are compiled to bytecode some are interpreted as is.
Python 3 is not even remotely close to a Python 2 rewrite. Much changed UI-wise, but the core is very similar if not identical.
https://tech.blog.aknin.name/tag/internals/
I spent a lot of time reading it while working on the mixed-mode Python debugger for Visual Studio (which, coincidentally, supports both Python 2 and Python 3 - they do really share a lot of things). Much of that work involved parsing and writing internal Python data structures directly, since Python interpreter may be unusable when the current instruction pointer is inside native code (GIL not held, various system locks held etc).
The lectures are very interesting and if you have a spare evening it's possible to just blast through the first 3 or 4 without sweating too much.
This looks awesome!