Hacker News Reader

I'm curious - what were you doing that polars was leaving a 40-80x speedup on the table? I've been happy with it's speed when held correctly, but it's certainly easy to hold it incorrectly and kill your perf if you're not careful

View on HN

20 year old BI app. Columnar DBs weren't really a thing. (MonetDB was brand new but not super stable. I committed the SQLAlchemy interface to it.)

KDB v1 is from sometime in the late 1990’s (I met v2 in 2002; but v1 was internal use only at some investment bank).

But that follows A and A+ which were extremely column oriented and date to early 1990s or even late 1980s ; and to various APL implementations going back to the 1960’s

Columnar DBs were very much a thing among APL users (finance and operations research) but weren’t really known outside those fields - and even in those fields, there was a period of amnesia in the late ‘90s/early 2000’s

Might be tangential but in my recent experience polars kept crashing the python server with OOM errors whenever I tried to stream data from and into large parquet files with some basic grouping and aggregation.

Claude suggested to just use DuckDB instead and indeed, it made short work of it.

Reply on news.ycombinator.com