Hacker News Reader

Comment by wodenokoto | Hacker News Reader

Did the author beat pandas group an aggregate by using standard Python lists?

The main optimization at that stage seems to be preallocating the weights. I don't know pandas but such a thing would have been possible without dropping any of the linalg libraries I do know how to use.

I doubt the author's C++ implementations beat BLAS/LAPACK, but since they're not shown I can only guess.

I've done stuff like this before but the tooling is really no fun, somewhere between 2 and 3 I'd just write it all in C++.

Changing the interface just to get parallelism out seems not great - give it to the user for free if the array is long enough - but maybe it was more reasonable for the non-trivial real problem.

Most likely a missued of Pandas. DF are heavy to create, but calculations on them are fast if you stay in the numpy world and stay vectorized.

"It's fast so long as you don't use any of the many parts that aren't fast!"

This isn't great.

That's true for everything in computing.

Don't use a hammer as a screwdriver.

I'm not even implying they shouldn't have used pandas for this, I'm suggesting they probably wrote the wrong pandas code for this.

Pandas is typically 3 times faster than raw Python, not 10 times slower.

No, I think it is fair to call out mediocrity, even when it tries to pull the "disclaim exactly the set of specific applications it gets called out on" trick.

Sure, pandas often beats raw python by a bit, but come on, there's so much mediocrity between the two that I doubt they even had to cheat to find a situation the other way around.

I wish for myself to create any project in my life that reach twice this level of mediocrity then.

I used to be a hardcore functional programming weenie, but over time I realized that to do high-performance, systems programming in an FP language means writing a bunch of non-idiomatic code, to the point that it's worth considering C (or C++ for STL only, but not that OOP stuff) instead unless you have a good reason (which you might) for a nonstandard language.

The problem isn't Python itself. Python has come a long way from where it started. The problem is people using Python for modules where they actually end up needing, say, manual memory management or heterogeneous high performance (e.g. Monte Carlo algorithms).

People create accidentally quadratic code all the time. It's even easier in pandas because the feature set is so huge and finding the right way to do it takes some experience (see stackoverflow for a lot of plain loops over pandas dataframes).

Yeah, I'm a bit suspicious that they made two simultaneous changes:

  1. remove pandas
  2. externalize WEIGHTS and don't generate it every run

Point 2 is likely a huge portion of the runtime.

Yes. Apparently they have used Python lists to beat highly optimized library which builds upon numpy, in C. Yeah right.

Note that I'm not saying that their second version of the code wasn't faster, just that this has nothing to do with python vs. pandas.

Every time I've used tools that used Pandas things are slow. I totally believe this.

You would think that, wouldn't you? But every time I've worked on a Python code base I have torn out Pandas and replaced it with simple procedural code, getting at least an order of magnitude.

Pandas is spectacularly slow. I don't understand how or why, but it is.

the reason is here:

> I don't understand how or why

Reply on news.ycombinator.com