Hacker News Reader

Comment by chazeon | Hacker News Reader

Software from our group (cij[1], qha[2]) were developed when numba seems to be the best option for JIT. It generates more pain in the hindsight. It generates a lot of depreciated warning due to unstable API, locked numpy to a certain version (i remember 1.21) due to compatibility issues, and when M1 Mac comes out, there were for a long time lack of llvmlite porting to the new platform, so cannot run on these new Macs.

If I had to do it again I would just use plain numpy or use the JAX from Google if JIT is really necessary.

[1]: https://github.com/MineralsCloud/cij

[2]: https://github.com/MineralsCloud/qha

View on HN

What if I'm (in Python) doing non-numerical stuff like parsing text and generating code? What JIT / AOT tooling (if any) is suitable?

Pypy, probably. You could also consider writing pre compiled extensions for your "hot" code, eg. in Cython.

pypy is great if you are not already using numpy heavily. Pure python libraries like networkx and myhdl showed 20x speedup when I used it a couple of years ago. For pure python code, pypy provides free lunch.

I think what will be the most maintainable and bring you the least long term pain is Cython

I have personally gotten a lot of mileage from just writing the compute heavy parts of my code in C++ and exposing it to Python with a tool like PyBind11 [1] or NumpyEigen [2]. I find tools like numba and cython to be more trouble than they're worth.

[1] https://github.com/pybind/pybind11 [2] https://github.com/fwilliams/numpyeigen

I prototype in python or whatever, then, if the project survives into market and has legs I either buy more hardware or rewrite the expensive parts in C++.

Reduces calendar time, risk, cost. And I'm likely to make better decisions once the code and market is better understood after the prototype is tested under real world conditions and the requirements have changed (like they always seem to do).

+1 for pybind11. I wrote python bindings using pybind11 for two C++ based simulators: MOOSE and Smoldyn. It was surprisingly easy to use given how badly Python C-API and c++ tooling suck. Though you have to create binary wheels for every version of python and platform separately.

> Though you have to create binary wheels for every version of python and platform separately.

cibuildwheel makes this easy.

I think most parsing-heavy code are just use C/C++ extension.

Example I can think of include:

1. pyyaml’s parser in C vs the Python version get a huge speed up on large files

2. some parsing table (~GB size) using pandas vs self-implemented Python code with a lot of for loop gain 20x speed up at least.

As a slight contrast to the other responses, I found setting up maturin (Rust + Python) very straightforward since the documentation is recent, and I find it's easy to write parsers in Rust because the ADT syntax is very terse.

Reply on news.ycombinator.com