Hacker News Reader

Comment by anamax | Hacker News Reader

> In my view, direct access to data decoupled from application code is a bug, not a feature. With multiple code bases touching the same data, schema improvements become nearly impossible.

It's unclear that multiple applications with direct access to said data make schema improvements any easier.

The obvious solution, copying the data for applications that are using the new schema, pretty much guarantees that one or more of the copies are wrong.

> I also think data integrity is easier to maintain with a system like this. SQL constraints don't allow me to express nearly as much about data integrity as I can in code. Sure, I could use stored procedures, but if I'm going to write code somewhere, I'd rather it be in my app.

How do you guarantee that all of the apps that touch that data use the current version of said code?

Code normalization is as important as data normalization.

View on HN

Very well put. It is not access type (direct vs indirect) that make schema improvements hard, but access preservation: availability. And indeed copying the data often leads to one or more of the copies being wrong, unless special measures are taken in that direction.

> How do you guarantee that all of the apps that touch that data use the current version of said code?

An approach that may be worth considering is to not require all apps to use the current version: allow multiple versions. For some cases this would work, say if the semantics of the newer version are backwards compatible with the semantics of the older version. If the data semantics are preservable, transforming a schema could happen while each data access request to the schema is actively transformed.

But it clearly wouldn't work in all cases. More work to handle that would be needed.

> Code normalization is as important as data normalization.

True, this is the near show-stopper really. In that case, the best one can hope for is preparing the state of the new version (new data in new schema) and carefully coordinating a quick restart of the old version for the new version.

I would love to hear your thoughts on this. We have been working towards that direction with ChronicDB (http://chronicdb.com) and would welcome feedback.

> I would love to hear your thoughts on this.

An e-mail address in your profile would have made that possible. (Chronicdb looks interesting and complements something that I've been thinking about. I suspect that you've implemented many of the relevant mechanisms.)

> How do you guarantee that all of the apps that touch that data use the current version of said code?

In a NoDB app? It's very easy: only one code base ever directly touches the data, because the data lives in the RAM allocated to the app. You give external access via an API, so integrity is very easy to enforce.

Congratulations, you have reinvented the integrity constraint! Except IBM have been working on this for 40 years, making it reliable and performant. I have yet to see anyone roll their own data integrity layer that comes anywhere close to the major vendors.

Good point. With databases distributing data in-memory across machines, shared memory becomes the database. Don't be surprised if Arc runs HackerNews on distributed memory some day...

But one has to wonder, what happens when you need to upgrade the app? Shutting down the process and destroying the memory image doesn't seem like the best option:

- First, it disrupts connected applications since the process is killed, introducing downtime.

- Second, when starting up again in say version 2, the data that will be loaded in memory still needs to be transformed in the format expected by version 2. This transformation can take time on large data, introducing further downtime.

The challenge would be to eliminate this downtime by combining a solution for both client disruption and state transfer. A data abstraction like a database using SQL can simplify such a solution.

Reply on news.ycombinator.com