Hacker News Reader

Comment by trjordan | Hacker News Reader

I want ACID in my databases:

  - Atomic -

Yep, but done with a global lock.

  - Consistency -

Yep. Since it's NoSQL, there's no referential checks, so this is easy. Git itself probably wants index / commit tree has to be readable, and all attributes.json files be valid. I don't see any reason it wouldn't be.

  - Isolated -

Sure, works fine, due to that global lock.

  - Durable -

Check. Once it's written, it's written.

So, git doesn't get you any of this for free, and getting Atomic / Isolated is done by introducing a big ugly global lock. I guess I don't see the appeal of using git (or even a model like git) because it doesn't get you any interesting necessary features for free. I see the backup and history thing as a secondary feature -- neat, but not worth sacrificing ACID for.

That said, cool idea -- there's not a lot of code you need to get to a reasonably small codebase.

View on HN

I'm pretty sure I can fix that global lock, but I'm focusing on getting the basics solid before focusing on performance.

My idea was that if there are concurrent transactions, then you just need a merge when the second one commits.

This is "optimistic locking": since they're probably updating different subdirectories it would be a trivial merge. If they update the same attribute of the same entity (i.e. they both change the same line of the same attributes.json file) then there's a conflict and the second commit fails.

This still requires a lock but only a tiny one, just at the moment of actually updating refs/heads/<branchname> (whereas right now it's around the whole transaction which obviously sucks).

Are there any flaws in that?

The reason I haven't done it yet even though it sounds pretty trivial is that with Grit (the Ruby Git library) the merge will need to use the working directory and that's a problem for concurrency (and also because I want the working directory to be useable by a human). I was thinking that with a bit of extra work I could make it happen in memory somehow.

Paul

Maybe you should just think of it as the NoSQL version of sqlite3 but with added version control.

What does that even mean?

I think "easy to set up" combined with "less crappy than expected". Although SQLite isn't very crappy at all...

Sqlite is often used as a quick and dirty database by projects that don't want to burden their developers with the setup overhead of mysql or postgres.

It's a tiny, rock-solid executable that stores data in a single binary file and supports sql access. Access to the file is globally locked. It's good enough for testing out a project on a single developer's machine but the global lock makes it hard to use in a multi-user production scenario.

In a similar vein I could see git as a widely available, rock-solid, "quick and dirty" way for a project to store and retrieve documents.

Because sqlite stores everything in a flatfile, so your are just doing version control on flatfile changes.

> - Consistency -

> Since it's NoSQL, there's no referential checks, so this is easy.

Consistency is quite easily achieved by putting the consistency checks into appropriate Git hooks. That way, Git would not allow any commit/pull of inconsistent data. A "rollback" is then trivial, as this only has to be done in the local working directory. So regarding consistency, Git provides everything that SQL databases provide, not just the limited NoSQL interpretation of "consistency".

Also note that this already happens in the traditional use of Git: There, consistency checks are called "building the application and running the test suite". It is a vital part of continuous integration systems. So consistency checks are not only theoretically possible, but already actively practiced with Git.

I feel it is similair to the motivation when one attempts to install Linux on some random device.

I expect to see a lot more 'interesting' projects that create NoSQL datastores in creative ways.

Wouldn't atomicity come from doing things on per-commit basis?

Also, you're not taking into account the things that Git does give you that typical databases don't, like branching and merges

Git doesn't give you merges, it gives you merge conflicts. I find it similar to how Riak uses vector clocks (http://blog.basho.com/2010/01/29/why-vector-clocks-are-easy/). It knows enough to tell you there's a conflict, but requires you (or the app) to tell it which version is the right one.

Right, it's either a trivial merge when you commit the transaction, or it's a conflict (i.e. two transactions updated the same attribute of the same entity).

There's no automatically fixing a conflict so the second transaction fails. Since there's full isolation that's pretty standard optimistic locking: http://en.wikipedia.org/wiki/Optimistic_locking

In most cases the transaction would succeed if it was automatically retried.

What global lock? Are you talking about the index? There are options for writing the index to separate directories. You can write each blob/tree separately, build an index in a unique temporary directory, and then the commit.

Is there something else I'm missing?

GitModel itself has a global lock (for now), here: https://github.com/pauldowman/gitmodel/blob/master/lib/gitmo...

I think it's easily fixed with a merge, as I described here: http://news.ycombinator.com/item?id=1929691

Concurrent updates to the same object.

You could get around some of the limitations by creating a branch for every session (though probably not on a high volume site), but then you will have lots of interesting problems once a session ends and you have to merge all changes back into master.

Ahh, yes this would be a problem on GitModel. I was thinking of Git at a lower level, since you can't update objects without also changing the SHA.

Reply on news.ycombinator.com