Hacker News Reader

Comment by squeedles | Hacker News Reader

Had to write a fairly substantial native extension to Python a couple years ago and one of the things I enjoyed was that the details were not easily "Googleable" because implementation results were swamped by language level results.

It took me back to the old days of source diving and accumulated knowledge that you carried around in your head.

https://www.dave.org/posts/20220806_python/

View on HN

I made some small contributions to cpython during the 3.14 cycle. The codebase is an interesting mix of modern and “90s style” C code.

I found that agentic coding tools were quite good at answering my architectural questions; even when their answers were only half correct, they usually pointed me in the right direction. (I didn’t use AI to write code and I wonder if agentic tools would struggle with certain aspects of the codebase like, for instance, the Cambrian explosion of utility macros used throughout.)

This was around 2021 so AI code tools had not yet eaten everyone. One of the most interesting challenges was finding the right value judgements when blending multiple type systems. I doubt any agentic coding tool could do it today.

I blended the python type system with a large low-level type system (STEP AIM low level types) and a smaller set of higher-level types (STEP ARM, similar to a database view). I already was familiar with STEP, so I needed to really grok what Python was doing under the covers because I needed to virtualize the STEP ARM and AIM access while making it look like "normal" Python.

Oh, that's very interesting work. And, yes, I'd also be surprised if (today's) agentic tools were at all helpful for that: it's way outside of distribution, and conceptual correctness truly matters.

There's a file on docs.python.org explaining the C api. The rest is pretty straightforward, at least until recently when free threading was introduced (IDK about now). Main hassle is manually having to track reference borrowing etc. Understandable in Python 2, but another tragedy in Python 3.

Great write up, thank you for sharing it! Quick question though, in your first code example (dynamic enum with a metaclass) what is "m" in this line towards the start?

    Py_DECREF(m)

Is it the metaclass?

That's a standard error clause. In the case PyImport_ImportModule threw a Python exception, you need to Py_DECREF any C local variables which are new references(not borrowed references) and return -1.

From the later call PyModule_AddObject, it's clear this code has come from the PyInit_ module initialisation function. This code is running on import of the C extension to initialise the "FruitEnum" module attribute. https://docs.python.org/3/c-api/extension-modules.html#c.PyI...

Exactly so. I didn't notice that missing def when I put together the blog post, but you are right to call it out. In this case that decref was copypasta from some other code -- I don't decref on the other error returns. I combined code that was in several places and omitted the decref for mod_enum too!

The module init function is where you would normally create the module object (PyModule_Create) and decref it if an error occurs. The blog example is utility code that you would call within the module init function to add an enum.

Someone should really create a blog post compiler to catch these sorts of things :-)

Reply on news.ycombinator.com