Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by hoerzu | Hacker News Reader

Full thread

hoerzu·

Can you give an example if I have 5gig (2 million rows)

How will it be created differently for columnar access?

View on HN

We ran some benchmarks (TPC-H, designed for OLAP) with ~10M records https://github.com/BemiHQ/BemiDB#benchmark

The BemiDB storage layer produced ~300MB columnar Parquet files (with ZSTD compression) vs 1.6GB of data in Postgres.

Does TPC-H SF1 really take _one and a half hours_ for you on regular Postgres? Last time I tried (in the form of DBT-3), it was 22 queries and most of them ran in a couple seconds.

Interesting. I haven't used the DBT-3 kit, does it add any indexes? I manually added these Postgres indexes https://github.com/BemiHQ/BemiDB/blob/main/benchmark/data/cr... to reduce the main bottlenecks on SF0.1 and reduce the total time from 1h23m13s to 1.5s. But SF1 still took more than 1h

It adds a bunch of indexes, yes. I don't think anyone really runs TPC-H unindexed unless they are using a database that plain doesn't support it; it wouldn't really give much meaningful information.

Edit: I seemingly don't have these benchmarks anymore, and I'm not going to re-run them now, but I found a very (_very_) roughly similar SF10 run clocking in around seven minutes total. So that's the order of magnitude I would be expecting, given ten times as much data.

Got it, thanks for sharing it! We'll try to look into DBT-3 and the indexes it creates to test with SF10

Reply on news.ycombinator.com