Gkv – Git as a key-value store | Hacker News Reader

Gkv – Git as a key-value store

github.com

43 pointsrhgraysonii26 comments

My only shock here is that this wasn't done sooner.

I have kind of seen Git as two tightly coupled systems (and IIRC, other VCS's have modeled their design with a similar separation of concerns): a content addressable store, and a tagging/tracking system for metadata.

I actually kind of expected that someone would eventually try out replacing one of the two with their own "better" system to see how that worked out, but obviously the more interesting opportunity is to see if one or both of the components might be useful for an entirely different purpose.

> My only shock here is that this wasn't done sooner.

This was my initial reasoning for making it. However when I began looking into it on Twitter @steveklabnik informed me that Rust's crates.io uses a similar system [0]. I found this especially intriguing, but have not gotten in touch with someone who knows the codebase better to give me some more insight.

[0] https://twitter.com/steveklabnik/status/608778665825083392

OctopusDeploy played with this to store configurations. That was supposed to allow you to have full history with branching possibilities and a straight forward way to backups. https://github.com/paulstovell/OctoDB

Also ...

Graydon Hoare, one of the people behind Rust is the one of the people behind Monotone. Which was an inspiration for Git.

Personally, I'm surprised that git hasn't been factored into two separate sets of libraries/tools: one to build and manipulate generic content-addressable stores, and then another than just consumes that library, adds a bit of metadata, and exposes the git toolchain.

If the API of the content-addressible-store library were standardized, one could even replace it with other backends (Plan9's Venti, or an object-store like S3 or Riak CS, or a DHT.) I could see it becoming sort of a libvirt thing.

Author here. Would be happy to answer any questions or clarify things. Its very much moreso a 'fun/useful toy' than 'production software' but there are some interesting principles that tie into it.

Cool idea and thanks for sharing! I see that you are shelling out to run the git commands. Any reason for not using a git binding instead? rugged[0] is a pretty well-used library for interfacing with git directly.

0 - https://github.com/libgit2/rugged

This is definitely a plan in the future. My initial use case was operating off of the assumption that whatever server I would be running this on would have git pre-installed, so it was a simple way to get it up and running quickly. Thanks for the recommendation, its definitely towards the top of my todo list for 0.3 :)

Have you tried comparing how this serves as a KV store as compared to other engines?

I've done no benchmarking yet. It is a piece that evolved initially from a project I was doing to just explore git's innards a bit more. I eventually gave a talk on what became v0.1 of this (only string store, no typing) at the Columbus Ruby Brigade, and have just been hacking on it a bit in the last couple of weeks because I've found it intriguing.

Someone added a PR with some benchmarks. Here they are on my system doing 1000 set/gets each time in 4 blocks [core i5, 8gb ram, ubuntu 15.04]

    user     system      total        real
    set:      0.150000   0.080000   0.230000 (  1.093421)
    hash set: 0.110000   0.120000   0.230000 (  1.085441)
    get:      0.150000   0.040000   0.190000 (  1.091698)
    hash get: 0.110000   0.060000   0.170000 (  1.075470)

The type is inferred from when you initially set the value. Note that saving the string '1' will return the integer 1 due to the naive nature of the implementation. Hashes, arrays and booleans behave as expected when saved.

Can you elaborate upon this in the readme? It wasn't clear to me whether git, ruby or the gkv has the naive implementation (or if the conversion only happens with the string '1').

Also is it possible to enumerate the versions of key-values that have been saved?

I'm sure I have several more questions but I'll leave it at that. Thanks for making this!

I just saw this now, sorry for being 5 days late if you even end up noticing it.

The typing is governed by YAML standards. This is why symbols are left out. Since it uses `YAML.dump` and `YAML.load` with the set/get, it will automatically assume the string '1' is an integer, '1.0' a float, and so on.

> is it possible to enumerate the versions of key-values that have been saved?

Yes, using the `get_version` method, you could do something like:

    all_items = $ITEMS[key].map { |hash| db.get(hash) }

I should add this as a normal function to the API. Its now on my todo list.

Just added the all past versions feature to the API now.

Just wanted to say thanks for all the input, pull requests, and ideas to develop this further with. :) Really appreciate it HN!

Nice! I made a "joke" key value store that uses HFS+ xattr file system attributes https://github.com/radiofreejohn/xattrkv

In the process I found a bug in one of the Darwin syscalls. Still not fixed :(

That's awesome (though frustrating w.r.t the last tidbit). I always find working on 'hacky' or 'pointless' things often yields the most new knowledge and insight. I'm reminded of Jim Weirich's 'Y Not: Adventures in Functional Programming' talk.

You should make an npm module and have it conform to this: https://www.npmjs.com/package/abstract-blob-store

That would definitely be interesting! My JavaScript is rusty, it would be a good excuse to do a refresher. Bookmarked and added to my todo notes :)

I made a Haskell library that uses Git as a JSON store: https://github.com/myfreeweb/gitson

Used in my website engine: https://github.com/myfreeweb/sweetroll

I just took a quick look at the code. This is non-persistent as it relies on the global `$ITEMS`, right? What benefit would this bring over simply storing directly into the variable?

> This is non-persistent as it relies on the global `$ITEMS`, right?

In a sense. $ITEMS is a hash ruby-object whose keys are what is set as the key by the input, but its values are the hashes that are stored in the git filesystem. There are about 10,000,000 improvements this could have and by no means would I consider it production quality software. Any ideas on a better means of persistance knowing this? Could back up the dict itself as a string in the git filesystem and have it be restorable on instantiation, possibly...

You should take a look at the garbage collector (git gc) of git as well. I'm pretty sure that it'll clean up your KV-store, as the objects are not referenced anywhere. Choose a storage model that'll survive `git gc`. Maybe take a look at the tree objects git itself uses?

This would hopefully play well with the potential of use of a more robust git tooling like rubiquity had mentioned in another comment. I had thought about the gc aspect but didn't dive into it too much since its so immature still. Definitely will be considering all this as I plan 0.3. Thanks.

Bobby!

Hunna Fresh! I miss you, man. Hope life is good! :)

PS I emailed you the other day.

Reply on news.ycombinator.com