Hacker News Reader

DuckDB's file format is OLAP-based but supports CRUD operations. I've created DBs with it with 10s of millions of records and gotten amazing query speeds.

Efficient querying using indices over S3 for their format isn't something I've looked into just yet.

View on HN

DuckDB is amazing, I'm using it every day.

It would be marginally useful here though. I really just want to get the binary compressed blob. Fun fact: MS SQL Server has "DECOMPRESS" and DuckDB does not.

Wouldn't that blob not be human readable?

no, but it's one step away with gzip.decompress or similar; if we know the exact position, we can retrieve it with range reads, roughly like this:

aws s3api get-object --bucket mybucket --key partially_human_friendly_blob --range bytes=start-end | cut or awk | gunzip

Reply on news.ycombinator.com