Python by the C side | Hacker News Reader

Python by the C side

paypal-engineering.com

145 pointstype012 comments

oscrypto, in attempt to support both CPython and PyPy well, uses cffi or ctypes to interface with the crypto libraries. That said, when cffi is present, only the ABI (dlopen) mode is used. No shared libraries are compiled.

My experience using both for the project has lead me to characterize ctypes as something for Python developers wanting to interact with C, and cffi as something for C developers wanting to use their code from Python. ctypes plays nicer with the Python GC. cffi is oriented towards C terminology/semantics, and (even knowing from the beginning that weak refs must be held) was far easier to crash by accidental use-after-free.

That's really useful to hear, especially from you, (presuming you're the same wbond who wrote oscrypto)! Have you written or considered writing a longer-form expansion on these experiences?

And yeah, in spite/because of all its simplisticness, ctypes really does seem to win out among those who share Python's aesthetics.

Yes, I am the author of oscrypto. Unfortunately one of my unfulfilled interests is longer-form technical writing. Trying to balance open source and family means that for now, that isn't in the cards.

I think one of the biggest benefits of ctypes is that it is part of the stdlib. Not requiring a compiler means it opens up a whole swath of users who aren't Python developers, but have Python installed. Part of the reason I'm interested in this is being heavily involved in the Sublime Text community, where every user has Python and ctypes, but requiring a C compiler and Python headers eliminates probably 99% of users.

Trying to distribute pre-compiled shared libraries was a path I went down, but gave up on. Having to compile 10 versions of the cryptography package every time a new release came around was a disaster. If you chance upon the cryptography-dev IRC channel, you'll see it is a part time job to get the package compiled reliably, and they have dedicated hardware provided by Rackspace to help them.

I am not too experienced in Python as much as I am in C. In general, do you think Cython might be better middle ground, or am I completely amiss of the various fits between cffi, ctypes and Cython? End use is mainly DSP-like code, neural networks (Darknet in C), and OpenCV and CUDA.

Cython is intended for making Python extension modules. So if the end goal is to make something that you interact with only from Python, then Cython is a good choice. You can still use it to simply wrap some existing library, but ctypes or cffi might then be a bit simpler since it they don't require an added compilation step.

Cython is really fantastic. It makes you realize that there is often not a huge reason to leave Python if that's what you are comfortable with. The community has simply produced so much.

cffi is awesome! You can even use it to wrap Python code in a dynamic library to use in existing C apps (to create plugins, for example). I gave a talk about how to use cffi and using it for embedding a few months ago. You can see it here: https://www.youtube.com/watch?v=T5dtiplkCIc

Very cool! If you're familiar with CFFI already just skip the first half of the video or check out the blog post: http://codelle.com/blog/2016/5/embedding-pypy-in-a-c-applica...

There also Nuitka for compiling python into a native module.

http://nuitka.net/pages/overview.html

I had a great experience interfacing C++ code with Python using both SWIG and SIP. While latter seems to be a bit more powerful for binding C++ code (as it handles many of the intricacies of operator overloading, polymorphism, virtual functions and class inheritance with ease), SWIG does support other languages besides Python (also I think SIP is modeled after SWIG).

For calling C functions cffi or ctypes is usually easier and incurs less overhead, but for rich C++ interfaces there is just no alternative to using a binding generator, as the required code becomes large very fast.

I love Cython, its extremely fast... Has someone experience about the stuff the guys at magic.io are building. They seem to have developed a really fast implementation of asyncio event loop on top of libuv...

I swore off SWIG years ago after being forced to use it for a very large set of C++ bindings. Recently, though, I needed to wrap a smaller C library of about 50 functions and a few structs. I tried cffi but its workflow for wrapping the complete contents of a few header files still seemed more complicated than SWIG. I ended up using SWIG again. The initial results were very easy but SWIG doesn't handle arrays very gracefully. May need to go back and put some more effort into cffi.

Reply on news.ycombinator.com