Hacker News Reader

Comment by loveparade | Hacker News Reader

There is also Polars [0], which is backed by arrow and a great alternative to pandas.

Polars is less ergonomic than Pandas in my experience for basic data analysis, and has less functionality for time series work. But it's better than Pandas for working with larger datasets and it's now my preferred choice for batch data processing in Python eg in ETL jobs. They both have their own places in the world.

Yes, and with both supporting arrow, you can zero-copy share data from polars into pandas once you’ve completed your initial data load+filtering.

Does zero-copy work currently, or is that still theoretical/todo?

I would also be curious about Numpy, since I know you can transparently map data to a Numpy array, but that's just "raw" fixed-width binary data and not something more structured like Arrow.

Once this is fixed: https://github.com/pola-rs/polars/pull/7084

I meant to ask about the other way, Polars to Pandas, but this is also useful to know.

It looks like it has already been merged, or did I miss something?

Polars is more verbose, but simpler. It's core operations "fit the brain". I've used pandas for years, but I still have to regularly google basic indexing stuff.

I don’t think it’s so black and white. Depends on what you’re doing. If you’re doing lots of cross dataset operations like econometric and physical system modeling polars can get a bit too verbose. Eg how would you do the following two pandas operations in polars?

    capacity_df - outage_df

    prices.loc['2023-01'] *= 1

There’s many workflows and models that do thousands of these types of operations.

   prices.loc['2023-01'] *= 1

You can always do df.to_pandas() ... prices.loc['2023-01'] *= 1 ... from_pandas() :)

More seriously, you are right, this is a tough one to do in polars. Polars seems to want to work with whole columns at a time, it doesn't give you write access to row sets.

There's also SQL, when paired with optimized column-store databases that can run embedded or local (duckdb or clickhouse), you can avoid the extra hop into python libraries entirely.

Do you have an opinion on N5? My second hand (somewhat dated) info is that Pandas are not a good fit for unstructured (CV) data or ML.

Yes, Marc talks extensively about the new interop options between Pandas and Polars enabled by the pyarrow backend.

Reply on news.ycombinator.com