PyPy IO improvements | Hacker News Reader

PyPy IO improvements

morepypy.blogspot.com

107 pointsjerven7 comments

kudos to PyPy team, always delivering the goodies. I just wished it was more CPython (extension) compatible.

I hope that the "pinning" work in the GC code is a first step for better CPython extension support. Classic C interop requires that some memory pointed from the C side in the Python heap stores the same thing. i.e. is not moved. Pinning is one way to do this.

Otherwise the same idea that Jruby/Graal did with co interpreting C code can work for PyPy as well (http://www.chrisseaton.com/rubytruffle/cext/). Although that would need someone to step up and implement a RPython C interpreter, which to be honest is not a trivial exercise.

That's not really a solution when using precompiled native extensions. Not much of a solution either when the native extension (precompiled or not) is a wrapper against a separate native library (which may or may not be coded in C).

Basically, it only works with "native accelerators", libraries which provide a pure {language} implementation as well as an optional self-contained C version of a subset of the library. Which is a case where you'd hope a high-quality JITed implementation would have a good speed on the "pure" implementation (and would probably prefer work be done on improving the JIT to make the pure version fast)

It doesn't work against precompiled native extensions, but it does work against wrappers against separate native libraries, like a database driver or libpng or something like that. The C interpreter can interface with native code, and the JIT has special support for making those calls very fast [1] (it's not like JNI, for example).

So the only thing it doesn't work for is precompiled native extensions where you don't have these source code - so maybe a proprietary C extension. Do you get many of those in practice?

[1] http://dl.acm.org/citation.cfm?id=2500832

> So the only thing it doesn't work for is precompiled native extensions where you don't have these source code

Or you can't build it because you're on windows, or the extension was installed precompiled by the package manager (on Python, that's exactly one of the use case for wheels, I don't think wheels ship C code ever), or the extension is coded in non-C (which is very much explicitly supported by CPython: https://docs.python.org/3/extending/extending.html#writing-e...)

Microsoft has done work to make the Windows situation at least incrementally better:

https://mail.python.org/pipermail/distutils-sig/2014-Septemb...

(I'm crassly attaching a bit of information to your post more than trying to respond to anything in it)

cffi is mostly our answer. The cpyext layer (the layer that emulates CPython C API) is there too, but it's prone to be slow due to the need to emulate some other model of a VM. It's possible to do a better job, but just noone really feels like fighting with this monster.

Reply on news.ycombinator.com