Hacker News Reader

Comment by gjvc | Hacker News Reader

What if I'm (in Python) doing non-numerical stuff like parsing text and generating code? What JIT / AOT tooling (if any) is suitable?

View on HN

I have personally gotten a lot of mileage from just writing the compute heavy parts of my code in C++ and exposing it to Python with a tool like PyBind11 [1] or NumpyEigen [2]. I find tools like numba and cython to be more trouble than they're worth.

[1] https://github.com/pybind/pybind11 [2] https://github.com/fwilliams/numpyeigen

I prototype in python or whatever, then, if the project survives into market and has legs I either buy more hardware or rewrite the expensive parts in C++.

Reduces calendar time, risk, cost. And I'm likely to make better decisions once the code and market is better understood after the prototype is tested under real world conditions and the requirements have changed (like they always seem to do).

+1 for pybind11. I wrote python bindings using pybind11 for two C++ based simulators: MOOSE and Smoldyn. It was surprisingly easy to use given how badly Python C-API and c++ tooling suck. Though you have to create binary wheels for every version of python and platform separately.

> Though you have to create binary wheels for every version of python and platform separately.

cibuildwheel makes this easy.

Pypy, probably. You could also consider writing pre compiled extensions for your "hot" code, eg. in Cython.

pypy is great if you are not already using numpy heavily. Pure python libraries like networkx and myhdl showed 20x speedup when I used it a couple of years ago. For pure python code, pypy provides free lunch.

I think what will be the most maintainable and bring you the least long term pain is Cython

I think most parsing-heavy code are just use C/C++ extension.

Example I can think of include:

1. pyyaml’s parser in C vs the Python version get a huge speed up on large files

2. some parsing table (~GB size) using pandas vs self-implemented Python code with a lot of for loop gain 20x speed up at least.

As a slight contrast to the other responses, I found setting up maturin (Rust + Python) very straightforward since the documentation is recent, and I find it's easy to write parsers in Rust because the ADT syntax is very terse.

Reply on news.ycombinator.com