Hacker News Reader

Comment by Aardwolf | Hacker News Reader

I found that python is very slow to use when analyzing billions of entries, if you do a function call per entry, because the overhead of a function call is so large in python (and may be much slower than what's actually inside the function). Even JavaScript can do this much faster.

Is there any way around that?

View on HN

Use the scientific stack in python for that type of analysis.

Yes, anything with one function call per item is going to be slow. That, along with memory inefficiency of lists, is the reason why numpy exists.

Indeed, numpy for numerical calculations. For strings, we have our own data structure based on Apache Arrow, but we plan/hope to move to Apache Arrow (in combination with numpy), since that's kind of the numpy++ for data science work.

For simply-vectorizable analysis, sure, but in my work I often have to apply a nontrivial transformation to the data en masse and I'd rather define a single function which defines the transformation on each row to trying to wrestle the problem into one of matrix multiplications and additions.

Does numba help? In the article they use it to calculate distance taking into account the curvature of the earth (lol).

Indeed, numba, Pythran or cupy can be really useful. In the example of the article, it is 'simple' vectorized math. But in general any function can be added in Vaex. Those that go through numba or Pythran usually release the GIL and can get you an enormous speed boost.

Reply on news.ycombinator.com