Hacker News Reader

Comment by imgabe | Hacker News Reader

There must be a corollary to Greenspun's Tenth Rule (https://en.wikipedia.org/wiki/Greenspun's_tenth_rule) that any sufficiently complicated data analysis library contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of SQL.

I use Pandas from time to time and I'll probably try this out, but I always find myself wishing I'd just started with shoving whatever data I'm working with into Postgres.

It's not like I'm some database expert either, I'm far more comfortable with Python, but the facilities for selecting, sorting, filtering, joining, etc tabular data are just way better in SQL.

View on HN

I recommend you look at DuckDB and the duckdb-prql extension.

DuckDB allows you to work on your Polars and Pandas (and any data on Arrow format) directly using SQL without needing any data copying or duplication.

The duckdb-prql allows you to use PRQL (prql-lang.org) which gives you all the power and universality of SQL with the ergonomics and DX of Polars or Pandas (IMHO).

Disclaimer: I'm a PRQL contributor.

Thank you for suggesting DuckDB. Looks brilliant!

You could do that, but it would likely both perform significantly worse (if you're doing "analytical" kinds of queries) and be a lot less flexible and expressive.

But you may want to look into DuckDB, which has a sql implementation that is not ad hoc, bug ridden, slow, or incomplete (though I honestly don't know about the formality of its specification). And it is compatible with polars :)

Imo lazy dataframe syntax is a far superior frontend to a query engine. Polars also has SQL support, but really the frontend isn't generally where bugs come from but instead come from the query engine.

Postgres would be an order of magnitude slower than OLAP query engines for the types of queries that people do with them.

the creator of pandas created Ibis, which has a Postgres backend, for reasons like this: https://ibis-project.org/backends/postgresql

this is a better approach to Python dataframes and SQL

Reply on news.ycombinator.com