Show HN: Inline C in Python | Hacker News Reader

Show HN: Inline C in Python

quasiquotes.readthedocs.org

14 pointsjoejev9 comments

This is great. Wasn't too clear on exactly how this is used from the documents (e.g. do I just run 'python') - didn't seem that there where any 'imports' either.

Found this example which seems a little clearer: https://pypi.python.org/pypi/quasiquotes/0.2.0

I was considering linking to the README (the index of the readthedocs page) but I wasn't sure if that would highlight the C inlining as much as I wanted. To summarize from the readme: After `pip install`ing the package, you can run quasiquoted code by including `# coding: quasiquotes` and then running under the normal CPython interpreter.

This still doesn't make it easy to understand how it is executing. Is there a hook on import that detects the file in which it's imported, searches for the # coding language pragma line, and then actually alters the text of the source code before the Python interpreter resumes parsing the rest of that file?

Initially I thought this was something like James Powell's "astexpr" literals example (towards the end of < http://www.youtube.com/watch?v=2O7yj-Nh6AY >), which would require actually implementing a new C-API type, adding parser logic for its literals, and recompiling Python.

If you can add a comment or two to clear up my confusion, I would really appreciate it.

Great work on this!

I think some of this might be addressed in the implementation section of the docs. http://quasiquotes.readthedocs.org/en/latest/impl.html To summarize: there is no import hook or ast parsing happening. Instead, this works by manipulating the way python reads the bytes and converts it to strings. When the raw bytes are read, it looks for token patterns that match the syntax of a quasiquote. It then replaces those patterns with normal function call syntax that is valid. A cute thing that can be done (that I need to document) is that you can "translate" any python from quasiquotes by doing something like:

'[$qq|this is the body|]'.encode('utf-8').decode('quasiquotes') "qq._quote_expr(0,' this is the body')"

Ahh, that makes sense now. So it's leveraging the built-in hook for custom decoding. I actually did not know that the `# coding: utf-8` happened at parse time, nor that it allowed for custom decoding function.

I'd be very happy to help better document this stuff in the code. Searching around, there are not very many good documents describing how to create and register a custom decoder.

I have read and re-read your setup.py file a few times now, and followed the trail to the .pth file and to the locations where codecs.register is used. But it's still not clear to me how this functionality gets "installed" such that the decoder hook that parses your decoder's name ("quasiquotes") is able to map that back to the search_function from the installed module's source code.

Basically, how does the codec registry become permanently aware that "quasiquotes" maps to the search function in your installed package?

The .pth imports a file which at module scope registers a search function with the codec system. This function checks if the name is 'quasiquotes', if not, return None otherwise return a custom object that defines all the methods needed to turn bytes into string objects or go the other way. Because it is a pth in the site-packages, it gets executed at interpreter start. If you wanted to chat more about this you can email me with the email on my gh page. Not sure how long hn threads normally stay up.

wow that is pretty nuts! do you think it would be possible to add a quasi quoter that could inline cython?

Cython would definitly be possible. It would probably work the same way the C quasiquoter does where it creates a function that accepts the scope and then executes the code. It would even be easier because it wouldn't need to do a first pass compile to get the missing names. I will try to work on this soon.

thanks! looks really cool

btw - I suspect a different title might help get more traction on hn - inlining C isn't that interesting (scipy.weave has done this for years) but this approach to inlining is super cool, and can be extended to other langs. none of this is obvious from the title

Reply on news.ycombinator.com