Hacker News Reader

Both Iceberg and Delta Lake support 'time travel' so you can query data as it was at a certain date.

SELECT count * FROM my_table AS OF "2025-01-01"

https://delta.io/blog/2023-02-01-delta-lake-time-travel/

https://iceberg.apache.org/docs/latest/spark-queries/#spark-...

up to a limited number of snapshots. it's not arbitrary times, it's when there was an atomic snapshot created in the catalog.

Don't know about other systems, but in Ducklake every committed transaction creates a new snapshot.

https://ducklake.select/docs/stable/duckdb/usage/snapshots

because they are making an extra step: instead of the snapshot being actually an object manifest, they store the snapshot metadata in a relation database.

I have a use case where we do 100k changes a second. I bet you "their unlimited snapshots because we are cool and using dbs for metadata" will fail in such a use case.

Every commit in Delta is time travel-able. Defaults in Delta clean up logs in 30 days (and data in 7 days but requires a VACUUM be issued.) You can set those retention periods to any arbitrary time period, but it's typically advised against due to the cost to store all the historical data.

Reply on news.ycombinator.com