A Python cross-version decompiler
github.com
github.com
A modern decompiler framework tends to take a very different approach. For native executables, the hardest problems are actually disassembly and higher-level type recovery. Bytecode is easier in that the type information is essentially retained and code is very nicely and obviously differentiated from data; no need to worry about jump tables or constant pools.
Control flow for a decompiler is recovered via some form of control flow structuring; the problem is essentially a graph grammar parsing problem, but this is not an easy problem to solve. In practice, it's the existence of goto-like edges (gotos, breaks, and continues) that really screws things up. It's usually easy to tell what the structure should look like without these edges, but identifying which edges fall into this category turns out to be challenging in practice. Especially if you start trying to undo jump threading.
The other interesting issue is working out what the decompiled variables are and should have been. SSA is actually a pretty clean way to separate out different variables that get assigned to the same register (this identification problem exists even in bytecode, as local variable slots due get reused whenever possible), but doing the PHI elimination and finding out where you've got bitcasted type magic going on (more of an issue for native code) can be a challenge.
When there is a translation, is it always unambiguous? No. The same bytecode could get decompiled "a and b" or as "if a: b".
Why this happens less often in Python, especially earlier versions fo Python, is that it's code generation is very much template driven and there very little in the way of the kinds of compiler optimization that would make decompiling control flow otherwise harder.
Using the example given above, I believe earlier versions of Python would always enclose the "then" statement/expression with "push_block", "pop_block" pairs. Even though in certain situations (no new variables are introduced in the "then" block) this isn't needed. But the fact that those two additional instructions are there can then distinguish between those two different constructs.
It is sort of like matching ancestry based on the largely "junk" DNA strands that exist in a person.
Finally, I should note that Python with its control structures with "else" clauses like "for/else" "while/else" and "try/else" make control flow detection much harder. And this is the major weakness of uncompyle6.
That's why I started https://github.com/rocky/python-control-flow to try to address this. In contrast, the current python uncompyle6 code which has a lot of hacky control flow detection, this creates a control flow graph, computes dominators and reverse domninators in support of distinguishing these high-level structures better.
Didn't think of for/else and friends, yeah, that's tricky.
Definitely using doms/idoms is the right approach.
Spoiler alert: no is it no more a glorified pattern-matching approach any more than a dragon-book-style compiler is.
We include Dropbox's Python 2.5 bytecode
https://blogs.dropbox.com/tech/2018/09/how-we-rolled-out-one...
Currently (custom) Python 3.5, I guess.
And the paper: https://www.usenix.org/system/files/conference/woot13/woot13...
Basically, dropbox remapped the op codes to different values and shipped rc4 encrypted pyc files with a custom python binary that had the key inside it.
Using Centos 7, I managed to install Python 3.6 (https://www.rosehosting.com/blog/how-to-install-python-3-6-4...) quite easily. And of course Python 3.4 is available in the EPEL repo if you need something more "official" (https://fedoraproject.org/wiki/EPEL).
But if you don't want your target machines to install anything, then use nuikta (http://nuitka.net/) on the dev machine to make a stand alone executable out of your python program, and distribute that.
The dev machine needs to be a centos 7 to make sure it will work on the targets (there are other possibilities, but more complicated). Other than that, you won't need to provide anything else. Python will be embedded in the resulting executable, will work transparently, and will not conflict with any existing setup.
As a bonus, the program will run up to 4 times faster, and won't need any admin rights to be installed.
Alternatively, you can use the python-future library (https://python-future.org) to make you program compatible with both python 2 and 3. But it's much more work.
See http://rocky.github.io/pycon2018.co/#/16 for a video showing use in this context.
As for round-tripping, I make use of that idea in testing the decompiler and/or looking for bugs (which are numerous). Python has an extensive test suite for its many library programs (in later versions in excesss of 800) which are largely written in Python.
The cool thing about a unit test program is that it is self checking.
uncompyle6 is far from perfect, so no, it doesn't roundrip all of the unit test programs that it could. (There are some unit test programs that are too fragile, like those that expect the line number to be exactly the same which might occur in tests testing a debugger).
The biggest problem as I've said many times is handling control flow properly. However that could be solved much better if someone who had more time to spend on this wanted to.
At most you might lose the names of local variables inside a function (i.e. variables that, due to internal scope, have a limited lifespan), but member variables, global variables, function names, class names, method names, etc. would all need to be kept in the bytecode.
There are maps in the code object that will turn the index back into a name, but that's only used when some sort of introspection occurs, such as you might find in a debugger or traceback.
The names found in an "import" of course is a string so there might some small advantage there in a shorter name.