Hacker News Reader

Vertical is the new horizontal, because of stuff like Epycs with 96 cores/192 threads, AWS instances with 24TB of RAM, M2 MacBook Pros with 96GB RAM, 12 cores, ludicrous SOC bandwidth.

What used to be 'big data' fits on an instance, you only need shards/clusters for 'ludicrous data'.

View on HN

Also, it would make me so happy if someone did a proper benchmark of

- pandas

- polars

- modin with various backends

- DuckDB and Postgres SQL from Jupyter/python with the usual SQL dataframe magic

- R

- on medium and 'pretty big data' i.e. 10GB dataframes with millions of rows.

- maybe on consumer Intel, M2, and Epyc with lots of cores

Reply on news.ycombinator.com