Hacker News Reader

Comment by nerdponx | Hacker News Reader

Polars is less ergonomic than Pandas in my experience for basic data analysis, and has less functionality for time series work. But it's better than Pandas for working with larger datasets and it's now my preferred choice for batch data processing in Python eg in ETL jobs. They both have their own places in the world.

View on HN

Yes, and with both supporting arrow, you can zero-copy share data from polars into pandas once you’ve completed your initial data load+filtering.

Does zero-copy work currently, or is that still theoretical/todo?

I would also be curious about Numpy, since I know you can transparently map data to a Numpy array, but that's just "raw" fixed-width binary data and not something more structured like Arrow.

Once this is fixed: https://github.com/pola-rs/polars/pull/7084

I meant to ask about the other way, Polars to Pandas, but this is also useful to know.

It looks like it has already been merged, or did I miss something?

Polars is more verbose, but simpler. It's core operations "fit the brain". I've used pandas for years, but I still have to regularly google basic indexing stuff.

I don’t think it’s so black and white. Depends on what you’re doing. If you’re doing lots of cross dataset operations like econometric and physical system modeling polars can get a bit too verbose. Eg how would you do the following two pandas operations in polars?

    capacity_df - outage_df

    prices.loc['2023-01'] *= 1

There’s many workflows and models that do thousands of these types of operations.

   prices.loc['2023-01'] *= 1

You can always do df.to_pandas() ... prices.loc['2023-01'] *= 1 ... from_pandas() :)

More seriously, you are right, this is a tough one to do in polars. Polars seems to want to work with whole columns at a time, it doesn't give you write access to row sets.

Reply on news.ycombinator.com