PyPy warmup improvements | Hacker News Reader

PyPy warmup improvements

morepypy.blogspot.com

166 pointshencq17 comments

Is it possible to get cython to build C extensions that are compatible with pypy? If not, is that something the pypy developers are interested in? A lot of the code I use depends heavily on C extensions, but it would be nice if I could the power of pypy's JIT compiler on the pure-python portions of my codebase.

You might want to edit that comment. I suspect "depends heavily on pypy" should be "depends heavily on C extensions".

AFAIK it's partially supported and under development. http://pypy.readthedocs.org/en/latest/faq.html#do-cpython-ex...

Your link mentions CPython (the reference/official Python interpreter), not Cython (optimizing static compiler and extension of the Python language) so I'm not sure it answers the question raised by parent.

You're right. The parent didn't provide enough information on the use of Cython. And frankly, there's no better way to find out whether PyPy supports a particular C extension than to try it.

wouldn't it be feasible to write a Cython compiler that doesn't compile to CPython API but to the ffi that PyPy likes more?

...or to RPython to expose more surface area to the JIT optimizer.

BTW very curious of our login name. There used to be this type inferred language called Wirbel that compiled a subset of Python to native binary. Are you anyway associated ?

"Wirbel" ist german for whirl, swirl, vortex, ... -- and "wirr" means roughly "confused" or "light in the head" -- my guess is that this user speaks German and the nick is a pun ;-).

not associated. It is indeed a kind of a pun. wirbel is whirl/swirl and also metaphorically "fuss", wirr means confused. So I put those words together, as BlackFingolfin described.

see my comment above - in short, very feasible, but someone has to care enough to do (I'm not volunteering)

Currently it should work via cpyext (CPython C API legacy infrastructure), but that's gonna be very slow

It's possible to write a Cython backend that would generate cffi bindings instead of C, but the work has not been done yet, so it's a bit unclear when it'll happen. Would be a good summer of code project for example.

Some more info/caveats of Cython's cpyext support can be found here:

http://docs.cython.org/src/userguide/pypy.html

https://github.com/cython/cython/wiki/enhancements-pypy

Off topic, but you might consider flagging some bugs as good starter bugs for joining the project. You have great technology but it's a challenge diving in to it, even with the docs.

If PyPy gets good enough, some of the need for C extensions should go away. Things such as "numpy" are written in C because numbers in Python are always boxed, that is, contained in a object. PyPy tries to figure out when that's not necessary, so it can use hardware arithmetic.

PyPy has to work very hard to do this, because you can muck with almost anything from anywhere with "setattr()" Even from other threads. So any guess PyPy makes has to be un-doable at any time. Python, the language, is what a naive interpreter such as CPython can do easily. In CPython, almost everything is a dictionary, and there are a lot of lookups. PyPy gets rid of much of that overhead. I'm enormously impressed that the PyPy crowd got it to work as well as they did.

These libraries are not just used because of performance. Some of them are quite old, large and partly written in Fortran. Unless you can do better in other ways than performance, you have to be quite a bit faster to make the effort of porting all these libraries worthwhile.

PyPy already is good enough for the need for C extensions to go away in a very large majority of cases.

Another option is to write an interpreter for the C extension source code that works like the Python interpreter in PyPy. This is what we do to support C extensions in JRuby+Truffle http://chrisseaton.com/rubytruffle/cext/.

Reply on news.ycombinator.com