The Design and Implementation of the CPython Virtual Machine
blog.codingconfessions.com
blog.codingconfessions.com
This was long enough ago that it was in the Python 2 days, so I don't know if the method survived the product's transition to Python 3. I do know the method would probably not have survived a dedicated reverse engineer who wanted to figure out how to recover the unencrypted bytecode.
So far, it has remained recognisable from one version to the next - someone used to reading (disassembled) CPython 2.7 bytecode would feel right at home reading CPython 3.11 bytecode, for example. But there's no guarantee that it will remain so, and with the faster-cpython project going on, we should probably expect it to change more in the future than we've been used to.
However, AFAIK, Python currently makes no guarantee at all about bytecode compatibility between versions, and there are no plans to change that.
I am unfortunately very familiar with CPython's bytecode [1], and do not recommend trying to do anything with it:
- It is extremely unstable; the same opcodes might do different things between versions. For instance: for the opcode `JUMP_IF_TRUE_OR_POP`, in Python 3.10 and below, its argument is an absolute address. In Python 3.11 and above, it is a relative address.
- It does not have synthetic (compiler-introduced) variables; this means the iterator of a loop must be kept on the stack, and as a result, exception handling must keep some items (namely, the iterators) on the stack.
- It is poorly documented (although it is getting better nowadays); there where numerous times where the stack layout an opcode expected changed between versions but were not yet included in the dis documentation.
One notable difference between CPython and other bytecode interpreters is a lack of multiple compilers. For instance, in Java, there are typically three compilers:
- `javac`, which produce class files that are interpreted by a basic, slow virtual machine.
- C1, which generates instrumented machine code to collect profiling data that C2 will use (still relatively slow).
- C2, which generates optimized machine code from the profiling data of C1 (significantly faster than C1).
CPython shoves the job of all three into its single interpreter. For instance, its CACHE instruction serves the purpose of recording the types on the stack for the subsequent instruction, which the specializing adaptive interpreter [2] uses to specialize an opcode (for instance, BINARY_OP to BINARY_OP_INPLACE_ADD_UNICODE).
All of the above makes it extremely hard to develop against CPython's bytecode, and is probably why there are few libraries for reading or transforming CPython's bytecode. Although most modifications can be done by either modifying the class object directly, wrapping the class/function, or extending the class, some cannot. For example:
- Change calls of function `foo` to function `bar`.
- Compute the automatic differentiation of a function [3].
- Translate python code blocks to another language for interoperability (for instance, to Java or SQL).
The way most library authors do it instead is by reading the AST, which is mostly stable.
[1] https://github.com/TimefoldAI/timefold-solver/tree/main/pyth..., a Python bytecode to Java bytecode translator, written to avoid excessive overhead caused by FFI calls (the new foreign Java API largely removes the overhead).