Python extension language using accelerators | Hacker News Reader

Python extension language using accelerators

github.com

47 pointsprecsim8 comments

From the Quickstart Guide (https://github.com/pyccel/pyccel/blob/master/tutorial/quicks...):

Pyccel's main goal is to resolve the principal bottleneck in scientific computing: the transition from prototype to production. Programmers usually develop their prototype code in a user-friendly interactive language like Python, but their final application requires an HPC implementation and therefore a new production code. In most cases this is written in a statically compiled language like Fortran/C/C++, and it uses SIMD vectorization, parallel multi-threading, MPI parallelization, GPU offloading, etc.

Sounds interesting!

The two other related projects I know are Numba and Cython. Worth checking them out as well.

Some similar projects are Pyston , PyPy and Rapids.ai

Pyston I was able to get a 15% increase in speed out of the box. PyPy I could not get to work with my project (numpy/pandas/ta-lib), I finally got everything to compile but it blew through my 32 gigs in 10 seconds and crashed.

Pyccel, Cython, Numba and Rapids I have yet to try but am interested in anyones experience.

If you are going to rewrite, you may want to consider Julia. It will give you close to C speed, but will be as high level (or higher level) than Python.

numba is pretty great, and i saw pretty extreme speedups (i can't remember exactly now, but they were orders of magnitude faster) when i used it in a project. however, to make full use of it, you pretty much have to isolate the numerical code that you want it to compile and be careful to ensure that it is easily translatable. (that is, avoid using libraries or python objects). essentially, for full effect, you need to turn on "nopython" mode which then severely limits what you can do in the accelerated code.

ultimately, i'd probably look at julia or maybe rust if i was building something from scratch these days though. i'll also note that adding numba to a project is a pretty heavyweight proposition (you are, afterall, adding llvm to your project).

Cython is so great. The two easiest ways I know to speed up python are 1) can I write this arraywise with enough cleverness (it's much more frequently doable than I always expect, and often comes with clarity gains) and just use numpy, and 2) just use cython. If cython wasn't such a pain in the ass to debug, maybe we'd all be writing cython instead of python today.

That is exactly my experience with cython. It’s a huge pain in the ass to debug. I found pybind11 to be way less of a pain in the ass than cython.

In most cases figuring out how to do it with nutty numpy is the least time consuming and is fast enough.

Some other similar projects are Pythran, Transonic, and mypyc.

Reply on news.ycombinator.com