Hacker News Reader

Comment by spenczar5 | Hacker News Reader

Polars is cool, but man, I really have come to think that dataframes are disastrous for software. The mess of internal state and confusion of writing functions that take “df” and manipulate it - its all so hard to clean up once you’re deep in the mess.

Quivr (https://github.com/spenczar/quivr) is an alternative approach that has been working for me. Maybe types are good!

View on HN

Totally agree with the critique, though it bears mentioning that one way that polars differentiates itself from pandas is the expressions API. See [1] for an example.

1: https://kevinheavey.github.io/modern-polars/method_chaining....

Polars is a lot better than pandas at maintaining valid state.

Because you ideally describe everything in terms of lazy operations, it actually internally keeps track of all your data types at every step until materialization and execution of the query plan. Because of that, you're not going to have the same kind of data type issues you might have in pandas. There are also libraries based off pydantic built for polars dataframes validation (see patito, though it's not mature yet).

Yes, the dataframe is very tricky to get right function signature wise. I used to write a lot of pandas heavy software and converged on writing most functions to take a dataframe or series and when possible just return a series, then put that series back into the df at the function call site. Handing dfs off to functions that mutate them gets gnarly very quickly.

Reply on news.ycombinator.com