Pythran: A Python subset to C++ compiler that takes advantage of SIMD | Hacker News Reader

Pythran: A Python subset to C++ compiler that takes advantage of SIMD

github.com

51 pointsmaxmouchet20 comments

There are quite a few attempts to make faster python implementation these days from pypy (A JIT built on top of rpython), pyston (yet another JIT that which uses llvm as a backend), pyparallel, nukita and I'm pretty sure there are a lot more.

Either the python community is being very prolific, or has no clue what will actually solve the problem, hence its trying everything.

Or there are multiple ways to solve the speed problem, with different approaches having different benefits.

Well, these things are fun to hack on and then each might solve a part of the generally near-unsolvable problem, and for each such part there are people caring about that part the most so the solution has its uses. (Incidentally, my own preference to date has been to only use CPython and fall back to C extension modules when speed starts to matter... But I could have a different preference in a different environment working on different things.)

Kind of like making C or C++ safe is a near-unsolvable problem for most values of "safe" and there's an ever-growing set of static and dynamic analysis tools detecting things like uninitialized memory access, out of bounds access, data races, integer overflow, etc. etc. etc. And each of these tools gives you a different mix of false positives, false negatives, build time overhead (including getting the program to build with the tool at all) and runtime overhead (including getting the program to run), not to mention availability on a given platform and pricing. And different people have a different preference with respect to what this mix should be for their tool of choice. But the problem is unsolvable which is why it's worth to be "trying everything."

> But the problem is unsolvable which is why it's worth to be "trying everything."

The problem is solvable by switching languages, but what these people all try to do, is to solve the problem while keep using the language, which is by definition unsolvable.

As the original causes of the problem won't go away.

But it is good, if in the end at least helps to improve the overall situation.

Switching languages may solve one problem while introducing another. If it were so simple, no one would use Python.

As a longtime programmer of Python, C & C++ (and a longtime appreciator of your FQA Lite, by the way), my preference until recently was also to use CPython & fall back on extension modules for speed.

But I grew weary of writing all the Python C-API boilerplate -- especially checking parameter types and managing reference counts properly.

So I created (and recently open-sourced) Pymod, a Nim+Python project that auto-generates all the Python C-API boilerplate & auto-compiles a Python C extension module, to wrap the functions in a Nim module: https://github.com/jboy/nim-pymod

As a pragmatic Python programmer, Pymod may be of interest to you too...

(As someone who learned C++ in 1999 and spent most of 200x working in C++, I also grew weary of staying on top of C++0x's new features & new gotchas. So about a year ago, I switched from C++ to Nim-lang and haven't looked back. To me, Nim combines all the best features of Python & C++, while avoiding most of the worst: C++ speed, C++ static types, C++ generics, C++ operator overloading; combined with a clean Pythonic syntax & Lispy macros.)

Cython (and before it Pyrex) occupies a similar space between C and Python.

Was any of your design inspired by them? I also wonder how practical it would be to reuse parts of it when making a bridge for another language.

Actually, Pymod was designed to be almost an anti-Cython. :)

My issue with Cython is that it's a limited sub-language within Python, where you add Cython elements incrementally & iteratively (diverging from Python in the process) until the code runs "fast enough". I'd rather work directly in a full-featured, internally-self-consistent language from the start. Nim has a clean Pythonic syntax, with all the best parts of C++ (including its runtime speed).

Hence, Pymod takes the form of an `exportpy` annotation (a user-defined Nim pragma) onto existing Nim functions, which are then auto-compiled into a Python extension module.

So there's no gradual divergence of my Python code (as it becomes more "Cythonized"); rather, the high-performance code is written directly in pure Nim. :)

Right, but Cython also supports the external library use case:

http://docs.cython.org/src/tutorial/clibraries.html

Interesting! Thanks for the link.

After browsing that page, I observe that even when Cython is wrapping an external C library, there's still a notable difference between the way Cython does things & the way Pymod does things.

The explanation on that Cython page begins: "To get started, the first step is to redefine the C API in a .pxd file, say, cqueue.pxd". So to wrap an external C library using Cython, you still need to define an entirely new header-like definition file in Cython's intermediate language.

In contrast, Pymod only requires an `exportpy` pragma annotation at the end of an existing Nim procedure function header -- you don't need to create any intermediate definition files. And the `exportpy` pragma is inert unless you invoke the "pmgen.py" script (a Python script in Pymod that determines the Python C-API system configuration & automates all the compilation), so your Nim code is still completely valid Nim code after you apply the `exportpy` annotation.

I'm now largely out of my depth, but much of that will just come down to what is available from Nim and what is available from C? And perhaps some of it from the complexity of supporting arbitrary C code in a flexible way?

The Cython wrapper code is still pretty trivial compared to the C code ( https://github.com/fragglet/c-algorithms/blob/master/src/que... ), and this is a pretty small piece of C.

Oh absolutely, Nim should definitely get the vast majority of the credit.

It's not just because Nim compiles to C; it's also due to Nim's powerful macro system, which: (1) allows me to define my own first-class pragmas such as `exportpy`; (2) supplies my pragma with a detailed & expressive AST of the Nim function onto which my pragma was annotated; (3) enables my pragma to auto-generate the C-API boilerplate code, and write it to disk as a newly-created C source file, all within the Nim compilation pass. Nim is a fantastic language, perfectly suited to this scenario.

I merely spotted an opportunity. :)

Yes. I use Cython mainly to wrap C++ and it is fairly pleasant. Yes there are still some boilerplates but the wrapping code is clean and easy to read. One can also cut down on the boilerplate by sticking to std library as much as possible at the api level.

Each of these projects has different goals if you look a bit more closely than "make python faster" and has made design decisions and trade-offs accordingly, and as a result each has use-cases it is better suited for than others.

Nuitka for anybody searching. There is also Shedskin and Numba.

It'd be nice if this project mentioned why they decided not to continue/enhance Shedskin.

I think another possible explanation is that compiling a subset of Python isn't such a huge project. Only a couple of the projects mentioned are anywhere close to even just supporting the standard library that ships with CPython.

I just wondered if anyone had any insight in how this differs from shedskin - https://code.google.com/p/shedskin/

From the shedskin website: Shed Skin is an experimental compiler, that can translate pure, but implicitly statically typed Python (2.4-2.6) programs into optimized C++. It can generate stand-alone programs or extension modules that can be imported and used in larger Python programs.

Don't know about the differences, but I think shedskin had died some years ago.

Nope, they are still responsive on the bug list, I think it lives - just very slowly.

But does it retain the python semantics? for example:

  def fac(n):
      if n == 0 :
          return 1
      else:
          return n * fac(n-1)

  print fac(100)

will yield a big string, while any naive C++ translation will print the string "0" due to overflow.

Good question (and I don't know the answer, as I'm not familiar with this project). Note that a naive c++ translation will also error out on simply i=2 * * 524. Automatic extension/promotion of numbers is pretty fundamental to python.

Reply on news.ycombinator.com