Hacker News Reader

Comment by mey | Hacker News Reader

Immutability is an interesting property, but reality is messy. Depending on the use case you envision for the database, data that needs to be scrubbed will accidentally be added to the system. This could be an important private key, or something legally dubious. In an append only/sync system there isn't really a good way to scrub that data. (Replay the entire history, filter bad data, then delete all remotes and resync from clean copy is not good). Again depends entirely on the use case of the database you envision, but if you wish to make it general purpose it is something to consider.

View on HN

Temporal databases - where SQL:2011 left off. I think the concepts of bitemporality and tritemporality are what the post author is looking for.

Something the original post reminds me of a little is Crux [1].

[1] https://opencrux.com/main/index.html

This article[1] from 2015 has stuck with me for years, talking about the same approach. The interesting thing about transactional/bitemporal databases is really the effect on how we consume them with our applications. The schema can be separated from the transaction data, schema migrations become easy, tables become cached materialized views (with the cache updated concurrently at write time), reduced need for application-level cache, no cache cold start, and pub/sub becomes the norm.

[1] https://www.confluent.io/blog/turning-the-database-inside-ou...

(Nice to see it's still relevant – the article is linked at the very top of the crux about page: https://opencrux.com/about/what-is-crux.html)

I’m the same regarding that article. I always thought of that article as a kind of “deconstruction” or “decoupling” and “rearrangement” of traditional DBMS architecture.

The end result you speak of is a more flexible data platform, which can handle more use cases, including applying decoupled parts to different things.

A good example, in my mind, is Apache Spark - which is like taking a compute and SQL query engine, making it distributed and decoupling it from the underlying storage. Now, suddenly SQL or code can be used on anything you can fit into a data frame, even streams - wow!

Pulling out the transaction log - and wow, suddenly we have durable, potentially immutable queues we can use for even long term storage.

I was so excited by the line of thinking in that article. :)

> Immutability is an interesting property, but reality is messy.

Words have fuzzy boundaries - immutable obviously doesn't mean actually immutable. It means immutable like how git is immutable - you're not meant to change history. It'll fight like a tiger to stop you easily changing history. It'll give you two different ways to get what you want while preserving history. But if you want to change history you can.

SQL is said to be declarative but if you get hit dealing with the wrong sort of workload in SQL then the implementation details can start to matter extremely quickly.

In practice, the author probably only means 'much more immutable' rather than 'illegal to use with European data' level immutability.

Git's primary advantage over other VCS is how it makes it trivially easy to change history, though.

True. And what makes it easy is it's immutability. When you change history in Git, you rewrite it (rebase). But re-writing is only possible if you know every bit of the journey up to now. When you have actual updates, history is lost.

the way git changes history is a stretch on the word change, rebasing creates a parallel history and then promotes it to be the true one. the reflog still remembers the previous history.

It’s interesting because it reflects the reality of modern computing — for many use cases storage capacity is not a meaningful constraint.

I ran pretty important systems as a DBA years ago that we’re significant but would never, ever require a meaningful constraint on growth. (Performance, different story)

Sometimes looking at a problem like a database with a new set of constraints leads to interesting and novel things.

I've been working on personal and professional projects where growth isn't a truly restricting constraint but is still an actual constraint. What do you mean by "meaningful"? Once I get into the 10's of terabytes there's no storage system where copying a TiB of data is "instant" or ignorable.

Professionaly I work on a financial reporting system where there are around 1.5 million accounts with trading activity. Things like moving a month of data around and checksumming still take on the order of 1 hour. If the size of that monthly dataset doubled, we'd be OKAY but it it grew 10x we'd have to start restructuring things both on the software side and the hardware side.

For personal work I am working with searching video frame substrings regardless of codec, resolution etc within videos in a ~50 TiB video set. Having to re-encode hundrends of GiB of video at a time is some thing that takes a huge amount of time and nothing something I can just "throw into an S3 bucket". The growth of said working set is not truly constrained but is actually constrained. I can only afford so many 8-10 TiB HDDs at a time! They also fail and have to be mirrored/backedup with rsycn at the very least which adds to the cost for me.

I mean that it’s not a technical constraint or limit. Not that its trivial to do! If there is an ROI in keeping stuff forever, you can.

Think about what you’re saying about your personal project. What you’re talking about wasn’t financially achievable for TV companies just 20 years ago. Hell, anything older than a few days would be stored offline in some tape silo, or in a shelf. (And 10-12 people to attend to it!) Now you as an individual could afford to do all sorts of stuff with that video data, albeit for some significant $.

Sure... but needing to purge sensitive data also reflects the reality of modern computing.

unsure whether to laugh or cry

Let me disagree. I'm working on a transactional OS [1] with an immutable filesystem, it's primarily a website builder. Every request creates a new snapshot of your website. It's append-only for the duration of the transaction. That's the only way to provide robust rollback and also provides version snapshots for free - but it doesn't mean snapshot's can't later be removed.

My solution was to provide an undo-chain of a fixed given length. Versions older than that simply "fall off the chain" and will be picked up by GC eventually. You can also make snapshots long-lived snapshots if you want.

(Of course removing a snapshot is an append-only operation :P but GC will go after non-referenced snapshots eventually.)

[1] https://boomla.com/

The problem of deleting stuff from a history of changes is challenging. But I think Darcs, the distributed version control system written in Haskell, does this rather neatly.

Darcs is different from git in that a "commit" (called a "patch" in Darcs lingo) does not depend upon the entire history of what has come before. Rather it has specified dependencies, which Darcs can infer itself – or you can specify if you want more semantic content in the patch structure.

A repository is a collection of patches, which are merged using something called patch algebra. So, if you want to delete a patch, you just issue the command "darcs obliterate" and it will remove the patch, and anything which depends on it. Then working directory is reconstructed by merging together any loose ends using patch algebra.

With careful administration of dependencies, this approach makes deleting specific things very slick.

Yeah, immutability doesn’t seem desirable in a world with GDPR and the right to be forgotten, which requires you to support deletion.

Database immutability does not mean it's impossible to delete content.

It means you separate two use cases: retracting information (I remove an item from my shopping cart because I want another item) and purging data (I delete my account and all my shopping history because I don't want the shop to store my data anymore).

You can still purge data easily in Datomic, for example.

I really don’t get why GDPR gets dragged up every time someone mentions immutable data stores; was everyone really building append-only systems before 2018?

It's because GDPR made things a law, not just a best practice. It does need to be considered, and it's worth making sure people remember it.

It has been a legal requirement in the UK that databases must be able to delete personal data since the data protection act 1984 - https://www.legislation.gov.uk/ukpga/1984/35/enacted - in particular see the data protection principles in schedule 1: the 6th principle requires that data is retained no longer than necessary. There are provisions elsewhere in the act that can require a data holder to delete data.

Most of GDPR is not really new. There was a significant amount of teaching about data protection when I was at university in 1994-1997 and it wasn’t even new then.

> Git repositories can only grow in size, and that is not only an acceptable condition, but also one of the reasons to use it.

That is so backwards I can't take the guy seriously.

Reply on news.ycombinator.com