Nuitka: a Python compiler
nuitka.net
nuitka.net
Number of people that tried it when I told them that to make it work they had to download python, pip, paramiko and install the pycrypto binary? Zero. Number of people that tried it after I just gave them a zip file with an executable? No longer zero.
I tried different libraries but Nuitka was the only one that made everything work seamlessly. I owe this guy a beer.
Edit: I no longer owe Kay a beer. I found the donation link on his website.
Find Guido's quote in the first comment.
https://ep2013.europython.eu/conference/talks/nuitka-the-pyt...
> The good thing [GvR] did to Python is to make his opinion be just that. I can do Nuitka without and beyond his control.[1]
[1]: https://ep2013.europython.eu/conference/talks/nuitka-the-pyt...
Kudos to anybody who is trying to move Python into a performance envelope similar to Lua and Javascript, not to mention Golang, Julia, or Clojure.
Of all the current dynamic languages Python is the slowest moving on almost every front. Ruby become popular around the time Python 3.X was coming out, at the time it was much slower and riddled with 1.8/1.9 issues. Since that time Ruby has surpassed Python almost everywhere whereas Python 3.X which should have had the freedom to do great things considering it broke backwards compat with 2.X has languished in it's slow performance.
Sure there is great things happening in and around the numpy community.. but that is tiny compared to the great things happening in Julia, Rust, Ruby, Golang and Javascript. (I include the last 2 despite my personal opinion being that they are not as good, but one can't deny they have made progress).
I loved Python but it's standing still and I can't afford to do that anymore. I said my very solemn goodbyes to programming Python full time a while ago and I think it's one of the best decisions I have made.
That being said. Good on this dude for doing something about Python performance without doing Cython style stuff.
/rant
Separately though, sometimes when you want to break the dead hand of the committee-driven, value-destroying hold of a small group of individuals, you have to mercifully push aside the figurehead that provides their credibility. That figurehead is Guido van Rossum, and his post speaks more than what I could about how his time is over.
While a large number of people use Python because of NumPy, my work in chemical structure analysis and search doesn't depend on NumPy. I use the package about once a year, and few of the chemistry analysis tools I have a dependency on it. I used to be involved with the Biopython project, and while NumPy is strongly recommended for a couple of the modules, it's not a requirement.
Then there's all the people who use it because of Django, and Zope before that. (I remember that the 2000 Python conference seemed to be half Zope developers.) Plus the win32 extensions get about 7,000 downloads per week, implying a pretty active base in that area.
Checking the PyCon talks, only about 5% of them seem to deal with numeric work. (Then again, there's SciPy ... but there's also DjangoCon and local non-NumPy meetings; certainly few in our local user group meeting work with NumPy.)
How then did you draw your conclusion?
Python-the-language did make three changes to support the language: multi-dimensional array slicing, the Ellipsis notation, and most recently the infix operator for matrix multiplication. But I believe that any language which couldn't handle a NumPy-like module would not be that successful in the first place.
django
Downloads (All Versions):
25082 downloads in the last day
171415 downloads in the last week
743271 downloads in the last month
Downloads (All Versions):
12920 downloads in the last day
80640 downloads in the last week
327163 downloads in the last monthAnother point (to the grandparent posts): putting Numpy into a "science" pidgeon hole is erroneous. It is massively used in finance, engineering, bioinformatics, and statistics.
... and there can still be more scientists using Python than web devs using it.
Python web development is niche, despite the good frameworks. If you look at job postings, it's eclipsed by RoR, .NET, Java... maybe even PHP.
> Let's not forget that Numpy and its direct linear ancestor, Numeric, has been dominant for more than a decade. Show me the Python web framework which can say the same.
Django was released almost 10 years ago, and I remember it was already popular, a lot of people migrated from Plone.
...and just about every aspect of engineering and a non-trivial part of finance.
But you cite a very important point: all the improvement, modest as it is for the majority of users, is going to 3.x and yes, that's why I've moved. But I can tell you it's only because of the constant nagging and threats about abandoning 2.x. Nobody would have moved for any actual "feature" of 3 were it not for the fear of being abandoned. It's a stick-only strategy. No carrot.
Just out of curiosity, who do other pythonistas think would be the best candidate for new BDF"L"? Travis Oliphant is one who comes to my mind.
Regardless of whether or not Hayden succeeds here, good on him for at least trying. GVR hasn't even made an effort to improve Python's performance (i.e, by most accounts, Python 3 is even slower than Python 2). In fact, he seems intent on actively discouraging such efforts. For example, why is CPython still using a stack-based interpreter, when several people have already worked toward implementing a register-based one and were only met with derision?
I generally agree with GVR's general approach of regarding premature optimization as a mistake, and developer time is usually more important to optimize for than processor time.
On the other hand, sometimes you have to optimize code, and dropping down to C is unnecessarily risky (not to mention tedious, although that comes with the territory). The fact that there's even an alternative today with Numpy and Cython seems to have happened in spite of GVR, not because of him.
I'm sad that someone as influential as GVR would be so consistently rude and dismissive toward an effort at improving Python, no matter how misguided he viewed the effort. This is the kind of attitude that makes leading open source projects suck.
Edit: By the way, I don't mean to imply that I share the pessimism some have about Python's future. Between Python's considerable rate of adoption in upper education, numerics, machine learning, bioinformatics, and various other scientific computing fields, combined with novel approaches to the language like PyPy, Pyston from Dropbox, Nuitka, and Numba, I'm overall pretty optimistic. But it's becoming clear that the committee-driven approach of CPython, driven/impeded by the BDFL, isn't bearing the fruits it has in the past.
I miss the Monty Python jokes and the freewheeling 'BASIC, evolved' feel of old 1.x sometimes even.
It is about being able to have a easy, dead-simple way to provide Windows users with a executable. Without changing my code, without weird build systems. Like it or not, Windows users are still the majority out there.
I love Python, it's my favorite language by far - but I love all those different tools (numba, pypy, hope, nuitka, cython etc) that make you sacrifice a small amount of dynamic magic in exchange for significant speed ups.
I don't have to use them - but when I need to write fast code, it's really nice to be able to do so in Python.
See also https://groups.google.com/forum/#!topic/python-static-type-c... if you wish additional evidence.
That Google Groups thread shows that he's irritated because he thinks that the Nuitka author doesn't understand the issues: "... he is incredibly naive about what kind of optimizations he'd like to apply. (He basically doesn't seem aware of the difficulties arising with static analysis of Python.)" While the Unladen Swallow and PyPy developers do understand the issues.
I think it's in character. I gave a talk at another EuroPython on measuring performance. Partway through he asked "isn't this just a repeat of the timeit module"? My response was "yes, I'm explaining why it works the way it does." I think that was sufficient to mollify his irritation.
http://blog.behnel.de/posts/indexp241.html
Nuitka's author's reply here:
https://webcache.googleusercontent.com/search?q=cache:fB1tgJ...
I sympathise with the developer because I think the tone of the criticism he's been getting is very far short of nice.
...But...On a purely technical level...I see a lot of problems with his response.
I think there's real merit in this project on the distribution side of things --- not the optimisation side. I hope he pivots the project in that direction.
There are previous projects, like Michael Salib's "Starkiller: a static type inferencer and compiler for Python" ( http://dspace.mit.edu/handle/1721.1/16688 ), and John Aycock's "Converting Python Virtual Machine Code to C" (http://legacy.python.org/workshops/1998-11/proceedings/paper...) which explored that optimization space, and no doubt others.
So when the talk is titled "for the first time, there is a consequently executed approach to statically translate the full language extent of Python, with all its special cases, without introducing a new or reduced version of Python", a listener would expect that it be more advanced than previous work. To be fair, neither Aycok's nor Salib's work was complete, but they did enough groundwork to show that static analysis of the type that Nuitka was exploring would not be able to achieve its stated goals.
But all of this pointing to a discussion from 2012 is pretty pointless, as people do use it for distribution - something which wasn't touched on during the talk, as I recall - while the author did talk about aiming for type inference at compile time, when all evidence is that the speaker had little idea of the actual issues, as previous reported in multiple previous attempts.
I honestly don't know what to do when someone presents a talk as a pure hobbyist, puttering around on a project with little interest in what others have done, and who doesn't understand the audience enough to gauge which details are of interest and which aren't. There's a culture mismatch, certainly, but that's what makes it harder to assign fault.
Should Hayen have realized that the project wasn't at the right level for EuroPython, or that the future project goals were overstated? Did EuroPython get more advanced over time? Did van Rossum not allow that weekend hobbyists don't have the experience to judge things? Should van Rossum never say anything negative in public about a project (and if so, at what level of fame does one need to bear that in mind)? I certainly have no answers to those.
It's a hard problem, I wish more people would attempt to take care of it.
At some point, the effort of letting someone down gently fifty times a day gives way to a curt but efficient form of communication.
In this case, GvR is just laying out what he sees as issues, take them or leave them. If you don't care for his opinion, fine, if you do, there you go.
This is pure condescending tone...
It's easy to compile it, but making the compilation actually useful for a dynamic language is HARD. There's a reason that most dynamic languages will either interpret or JIT, and it's not because the JIT writers overlooked something obvious.
A naive translation of the code will remove a little bit of overhead from the bytecode dispatch, but the resulting code bloat will blow out instruction caches for any reasonable sized code. In small programs with one translation unit, analysis can sometimes work to speed things up, but it quickly becomes either undecidable or intractable.
Basically, on dynamic languages, you can often see static compilation being worse than a naive attempt at native code, especially once the hot path for the interpreter fits in cache, but the generated native code no longer does. Wanting to see results on real world benchmarks is entirely reasonable.
And when adding compilation into the game, I don't see CL as any less dynamic (probably more) than Python, and yet it goes to near C speeds too.
Sometimes it's not even possible (!) to get C speed because of things like float boxing across function boundaries.
- Naive interpreter (CPython). Everything is a dict. Slow. - Transliterate to some hard-compiled language, but all data is still one kind of dynamically typed object (Nuitka). A little faster, and compatible, but has limited optimization potential. - Infer types and try to create appropriate code in a faster language (Shed Skin). Hard to do, but promising. (Shed Skin has one implementor.) - Restrict the language (RPython) Potentially much faster, but incompatible. - Build a JIT compiler/interpreter combo and handle all the hard cases that require recompiling during execution (PyPy). Hard to do, and results in a huge system, but almost compatible. After 10 years of work, it's finally happening.
If you're willing to restrict the language, it's much easier. RPython was written only to help build PyPy, but the concept could be extended to allow most of Python. Both Shed Skin and RPython insist that type inference succeed at disambiguating types. If you're willing to accept using an "any" type when type inference fails, you can handle more of the language.
The big boat-anchor feature of Python is "setattr", combined with the ability to examine and change most of the program and its objects. This isn't just reflection; it's insinuation; you can get at things which should be private to other objects or threads and mess with them. By string name, no less. This invalidates almost all compiler optimizations. It's not a particularly useful feature. It just happens to be easy to implement given CPython's internal dictionary-based model. If "setattr" were limited to a class of objects derived from a "dynamic object" class, most of the real use cases (like HTML/XML parsers where the tags appear as Python attributes) would still work, while the rest of the code could be compiled with hard offsets for object fields.
The other big problem with Python is that its threading model is no better than C's. Everything is implicitly shared. To make this work, the infamous Global Interpreter Lock is needed. Some implementations split this into many locks, but because the language has no idea of which thread owns what, there's lots of unnecessary locking.
Python is a very pleasant language in which to program. If it ever gets rid of the less useful parts of the "extremely dynamic semantics", sometimes called the Guido von Rossum Memorial Boat Anchor, it could be much more widely useful.
Isn't __slots__ made for that specific usecase? (when you want to optimize your code by specifiying specific attribute names)
I think the "dynamic" behavior is a sane default (since most people don't need that optimization).
(Threading is a separate issue, though.)
Nope, we just need a large company willing to spend tons of money funding the effort.
PyPy has made incredible strides in this area, especially for long running processes where the JIT has time to warm up. But they need a lot more funding if people ever want Python to get fast.
JavaScript is a bit easier because it doesn't have concurrency. In Python, you can change a method of an object while an instance of that object is being executed in another thread. So you have to worry about invalidating code currently being executed asynchronously.
Long term, it won't be as fast as a full JIT, as the Facebook HPHP team showed, but Python doesn't have a JIT of the same caliber, so this is probably useful for tons of people.
Dunno if the author is around, but they might find some of the stuff from my PhD relevant, especially how the static analysis worked and some of the challenges of compiling using the C API: http://paulbiggar.com/research/#phd-dissertation
Bought a beer to Kay :-)
[0] - https://github.com/kayhayen/Nuitka/graphs/contributors
Holy cow. And here I'd set aside Python in part because of the difficulty in getting performant executables (I was getting visible lag in a turn-based SDL game ...)
Also, using a JIT is not possible on some platforms (iOS, or if the CPU architecture isn't supported by pypy).
Not to say that pypy isn't better for certain tasks of course.
It's a hassle getting your head around it, and getting it set up, but you'll never look back once you do.
If you rely on PyPy, you get what you're given and then you're stuck. You can't really guess how to rewrite your code to make it faster.
A small blog post I wrote on this: https://honnibal.wordpress.com/2014/10/21/writing-c-in-cytho...
Here's a non-trivial example: https://github.com/honnibal/thinc/blob/master/thinc/learner....
This code is driving a library of very fast NLP tools that I'm writing.
...not really suitable for games.
kivy-ios should have all the details for building everything into one big Python blob, if that's what you need.
The shootout, however, is from August 2013 so getting a bit old by now.
[0] https://www.youtube.com/watch?v=d1a4Jbjc-vU
I've been writing little helper libraries on top of Python 3.4 + pysdl2 as I've been working on games and demos for my (unfortunately cancelled) Pycon talk. One area I've only played with, unsuccessfully, is getting packaging going on Nuitka or some other compiler. If anyone wants to get together to make it awesome get in touch.
If you filter by Python, you can compare some results running on both CPython and PyPy. I would be curious to know what it is about these benchmarks that makes PyPy perform poorly. I would also be interested to see how Nuitka performs.
At the moment I'm also very excited about Pyston from Dropbox: https://github.com/dropbox/pyston
Anyway, all that is to say: I really like PyPy, and we use it a lot, but those _unexpected_ crazy slowdowns are unfortunate.
What STM (as I understand from blog posts from the pypy team) is that it provides real threading without having to worry about pickling.
And yes, it only helps if you have multithreaded programs. However, multithreaded and parallel programs are very relevant today.
Have you audited that compiler's source and building the compiler from it with a known good compiler? Are you inspecting the resulting .pyc, and in this case, the resulting PEs? It's a super easy way to inject a compromise into a package that will probably get widely distributed.
full disclosure: link is down for me, so haven't read the article. Been a comment that has been building up for a while for me, and not specific to Nuitka. Same goes for new frameworks/languages/etc.
Because nobody is that paranoid?
There's much lower hanging fruit.
BTW how much faster is this than PyPy or CPython? That is the question that I think is most on people's minds.
This seems immidately relevant to address pythons biggest problems; its slow, and its hard to distribute. Relevant quote:
All code generation is done to C++ rather than architecture
specific machine code. The cost of porting and maintenance
of architecture specific code generation is now more amortisedSidenote: A very long time ago I had some great luck with Perl's packagers: PAR, PerlApp, etc. but they all basically just bundle in the basic Perl runtime with the guts of your program so there's no real benchmark difference. I suspect the same is not true of efforts like this for Python.
I couldn't get it to work against this obfuscated hello world though: http://benkurtovic.com/2014/06/01/obfuscating-hello-world.ht... :)
I think Python is average in this respect. By comparison, here's an incomplete list of C compilers: http://en.wikipedia.org/wiki/List_of_compilers#C_compilers and you can also look at the list of Pascal compilers.
I suspect we would have seen this more with languages like Perl, except Perl is virtually impossible to parse.
And quite unlike why we have that many JavaScript transpilers.
A few of the compilers might be running on the AST, and thus have no complete alternative implementeation. A good module support for that helps, although it's still not close to VM-based languages like Java.
So, people in different realms (MSoft IronPython, JVM/Enterprise Jython, restofus CPython, scientists/performance Cython/PyPy,Stackless,Psyco, academics/experimenters many more) all want to use it and make versions that integrate with their tools.