Hacker News Reader

Comment by didip | Hacker News Reader

I would love to learn from database experts here:

Why some databases choose to use segment files as persistence and why some other databases choose to use key-value as persistence?

From my perspective, using key-value is very convenient because you can then outsource persistence to a distributed, fault tolerant, key-value storage.

View on HN

Many query plans (e.g. sorts, joins, aggregation) involve scanning large swaths of data. The importance of sequential access for such operations cannot be overstated. Storage, processors, and networks are designed to make sequential access performant.

Some SQL databases are backed by key-values stores, but they generally store large segments in the values rather than individual records. (e.g. Snowflake, Athena)

Segments also allow you to store some info about segment statistics, for example min and max for each numeric column which allow optimiser to skip entire segments when filtering. More tricks can be found in Snowflake paper.

There was another article posted today about LSM trees, and how they use segment files. I think at the end of the day the "distributed, fault tolerant, key-value storage" is using segment files too. By having your database use segment files directly you're just skipping a middleman.

Adding for (future) reference

https://news.ycombinator.com/item?id=30269286

There are no conflict: key-value is an interface, segment files is an implementation. You can implement a key-value using segment files too.

Not sure that I qualify as an expert, but in the case of Postgres the choice was probably influenced by the fact that readily available, distributed, fault tolerant, key-value storage wouldn't exist for at least another decade.

It's all very well until you discover the distributed, fault-tolerant, key-value storage is implemented using a database.

But what if that database is using a key value store?

it's turtles all the way down

it's tuples all the way down

perfect, thank you

Reply on news.ycombinator.com