Interfacing Python and C: Advanced “ctypes” Features | Hacker News Reader

Interfacing Python and C: Advanced “ctypes” Features

dbader.org

80 pointsdbader17 comments

Seems like a reasonable article. But that Makefile is a disaster:

1. The clean rule doesn't delete the generated .html file.

2. The clean rule returns an error if some of the output files are missing. It needs a -f adding.

3. It doesn't implement dependency tracking of the header files or the Makefile itself.

4. On my clean Ubuntu 16 LTS install, the .o:.c rule doesn't get called. Instead Make uses its built-in rule which doesn't include -fPIC, so the build fails. I guess it is missing some % symbols.

More controversially, I think Makefiles for simple purposes like this are a form of premature optimization. It would be simpler to create a bash script called build.sh containing:

  CFLAGS="-Wall -Werror -fpic"
  gcc $CFLAGS -shared Point.c Line.c -o libline.so
  gcc $CFLAGS -shared Point.c -o libpoint.so
  ./testPoint.py
  ./testWrappedPoint.py
  ./testLine.py

That's much easier to understand, doesn't depend on Make being installed, rebuilds when a header is changed and doesn't generate .o file litter. The price is that it takes 0.1 seconds longer to run in the case where there's no building to be done and there's no "clean". I think it is a net win.

Good article. A while back, I wrote a framework to support Python modules in a multi-threaded C program, so the control flow is from C to Python and then back again.

http://www.linuxjournal.com/content/extending-glusterfs-pyth...

For the "embedding" part (C to Python) it's very important to get the GIL stuff right, but it's not totally obvious how to do so and it's hard to debug when you get it wrong. For the "extending" part (Python to C) the issue is going to be maintaining references for Python objects. Dealing with all of the edge cases (e.g. decorators and ctypes-generated function wrappers) was quite educational. Mixing C and Python is definitely a fun exercise.

pybind11 is the nicest foreign function interface I've encountered:

https://github.com/pybind/pybind11

Personally I like pybind11 much better, it is incredibly easy to wrap even fairly complex c++ (and by extension c) projects with it. The best thing about it is its very good and thorough documentation. It is even fairly simple to autogenerate most of those bindings using libclang (The example he gave fall in that category, it gets trickier if you want to be able to subclass in python).

I've always thought ctypes a bit of a hack as you get crashes and so on if you're not constantly careful about updating function definitions. It would be much nicer if the compiler could catch the errors, e.g.

    #include "mylib.h"
    PYTHON_WRAP(myfunc, void, float, float...)

Even with cython (please correct me if I'm wrong), the definition needs to be duplicated in the cython script to use it.

You are describing cffi http://cffi.readthedocs.io/en/latest

Sort of, though I was thinking of more of a system external to python at the C/C++ level. Maybe it's possible to use cffi like this.

Boost Python looks somewhat similar to that type of interface too. http://www.boost.org/doc/libs/1_62_0/libs/python/doc/html/tu...

Why would you use CTypes instead of CFFI?

It's in the standard library and doesn't require having the whole C toolchain available and working (non-trivial if you're not talking about standard things in /usr/{include,lib}).

If all you need is a simple interface to a couple of functions and you want portability across system variants, that might be the path of least resistance and the performance is often negligible if the amount of work being done in the C code is significant.

Personally, I prefer cython most of the times to write C binding/lib. All Gil helpers are very useful.

why not rust? much simpler and safer with libs like https://github.com/PyO3/pyo3

I might be wrong, but it seems PyO3 is more for going from Rust to Python, versus ctypes being for Python to C.

Sure, coupes is fine if do not need to write anything and ctypes is enough. But as soon as you need to make c to python bridge it safer to do in rust. Pyo3 handles ref counting and Gil management for you

That sounds like the opposite direction: ctypes is for when you have a Python program which wants to call a C library, not when you want C code to call Python. As a concrete example, I work with JPEG 2000 imagery. I use ctypes so Python programs can load an image using a JP2 codec such as OpenJPEG to decode the image and save image tiles. Given the complexity of OpenJPEG, it would be a major project to port it to any other language and since it does a massive amount of computation the overhead of ctypes calling it is a small rounding error in the total processing time.

Unless the PyO3 documentation is completely leaving out a major application, it doesn't support this scenario.

Pyo3 support both. You can expose rust to python, and would look to python as native functions and classes or you can call python code from rust

Pyo3 uses rust refs ownership to enforce proper Gil usage. You code won’t compile if you use Gil in wrong way

Because the library you are trying to use it is written in C / C++, which is more likely than that is written in rust.

Reply on news.ycombinator.com