Git as a NoSql Database (2016)
kenneth-truyers.net
kenneth-truyers.net
My current preferred strategy for dealing with this (at least for any table smaller than a few GBs) is to dump the entire table contents to a git repository on a schedule.
I've run this for a few toy projects and it seems to work really well. I'm ready to try it with something production-scale the next time the opportunity presents itself.
I guess you can't just plug this into a common system of rotating logs easily, as there might be several changes beteeen the log rotation.
Also, I guess you'd need a user friendlier interface to actually display who made the change from the git repo.
Anyway, interesting solution.
How does that solve the problem? All this would do is give you snapshots of the state of the DB at particular points in time.
The only way to know who changed what when is to keep track of those changes as they are made, and the best way to do that is to stick the changes in an eventlog table.
As the senior dev it was my job to provide info as needed and subtly assess the new, potential rockstar code ninjas.
Anyway, this new guy says "I store all my dates in sql as text in iso format so I can easily sort them".
If he had said anything other than "in sql" it would be the correct answer.
"I store all dates in spreadsheets in ISO format"
"I prepend ISO dates to marketing campaign names so I can easily sort them in the horrible web frontends that doesn't have a sort by date option"
"I start all my blog entries with ISO dates..."
I mean, just... so close!
EDIT: also I wouldn’t consider this egregious. If the Senior explains and the person was happy to learn something then that’s a good outcome. If they are stubborn about it, then that wouldn’t be great.
I've seen far worse things than this done to a SQL database though…
Just have a look at the list from Wikipedia [1] and notice how many repeats there are.
As an example AMT could be either UTC+4 or UTC-4.
And this doesn't even includes the various local acronyms that people use around the world.
[1] https://en.wikipedia.org/wiki/List_of_time_zone_abbreviation...
IANA time zone identifiers are fully-qualified and unambiguous like America/Los_Angeles or Asia/Kolkata.
If you don't use these, you will inevitably be somewhat wrong due to leap issues, political changes in time, or the like.
If you are trying to store dates and not timestamps, that’s what you want, otherwise equality tests might not work right.
> (The integers stored are usually POSIX timestamps.)
For date types, if they exist distinctly, usually not.
Strong SQLite vibes... (SQLite doesn't have a date format, it has builtin functions that accept ISO format text or unix timestamps)
https://stackoverflow.com/questions/1732348/regex-match-open...
:)
Just point him in the right direction and move on.
I am way less judgemental now and a lot of the comments here are making me rethink what I thought about him saying this. He had a lot of other issues and resigned after a couple of days so I guess we'll never know what he was capable of.
Among the option of storing (date field, millisecond field) and (date as iso8601 text field with millisecond without time zone) I chose the latter, and I can’t tell for sure but it was almost surely the better choice.
I saw no benefit in storing two copies. That’s recipe for problems - e.g. “update” code that doesn’t update both, or somehow updates them differently (which one is right?). You can probably have t-sql or some “check” constraint to enforce equivalence, but ... why?
For most practical uses, an ISO8601 textual representation is as good, though a little less space efficient.
At risk of extreme overkill the other way, something like Debezium [1] doing change monitoring dumping into S3 might be a viable industrial strength approach. I haven't used it in production, but have been looking for appropriate time to try it.
Sally makes a change to column 1 of record 1.
Billy makes a change to column 2 of record 1 a nanosecond later.
You commit to Git again.
Your boss wants to know who changed column 1 of record 1.
You report it was Billy.
Billy is fired.
Your boss decides to fire people as a solution
Your boss should be fired
You should be taking care of this, not Billy
For example, for audit logs: https://www.enterpriseready.io/features/audit-log/
It’s not, though. History tables don’t provide only 111% of the value of this, and don’t take 20× the effort. They have a deterministic relation to base tables (so generating the schema from that of base tables, and the code necesaary for maintenance, is a scriptable mechanical transform if you have the base table schema.)
OTOH, lots of places I've seen approach this by haphazard addition of some of what I think of as “audit theater” columns to base tables (usually some or all of “created”, “created_at”, “last_updated”, “updated_by”), so its also not the worst approach I’ve seen.
One of the larger relational DBs beating the OG NoSQL will always fill me with amusement.
1. Ed. looks like this is still the case, but please remember this is MongoDB specifically. AFAICT MongoDB is mostly a dead branch at this point and a comparison against Redis or something else that's had continued investment might be more fair. I'm just happy chilling in my SQL world - I'm not super up-to-date on the NoSQL market.
Also, here's SiSense sensationalizing it for easy digest: https://www.sisense.com/blog/postgres-vs-mongodb-for-storing...
That's not true. The assumption that you are storing text files is very much built in to the design of git, and specifically, into the design of its diff and merge algorithm. Those algorithms treat line breaks as privileged markers of the structure of the underlying data. This is the reason git does not play well with large binary files.
In git world this distinction is referred to as the "porcelain" vs "plumbing", the plumbing being the underlying structures and porcelain being the stuff most people actually use (diffs, merges, rebases, etc...)
> one or more parent commits identified by their hash
The reason that matters is because you need that information to do merges. So merges are integral to git. They are not "porcelain". They are, arguably, the whole point.
The plumbing cares only about one thing, that you have a valid data structure, a data structure that among many things tracks "what was/were the previous state/s before this commit?" as a pointer to the previous commit/s, nothing more. I can construct a git tree by hand / shell script just piping in full text/binary files, and git's porcelain will happily spit out "diffs" of the structure.
Your porcelain on top of that decides what to do with that information, standard builds admittedly just provide diff tooling that focuses on , more expansive UIs can diff some binary data (e.g. GitHub shows diffs of images using a UI more suited for that data, with side by side, swiping, or fading to allow someone to examine the changes)
> The assumption that you are storing text files is very much built in to the design of git
I haven't dived deep in a while to git's source code, but the last time I did, this argument would only really hold true for packfiles / where delta encoding is used to compact the .git directory's contents as that does indeed seem tuned for generic text content instead of binary data.
That isn't related to the diff/merge functionality of git though; that is a feature used to reduce disk space by reducing loose objects (full copies of the previous versions of files), trading increased compute time (rebuilding a previous version of a file from a set of diffs) for reduced disk space.
Does anyone happen to know of an implementation or tests for delta encoding I could consult that is available under an MIT-like license? (BSD, Apache v2, etc.)
http://shithub.us/ori/git9/724c516a6eda0063439457a6701ef0d7e...
http://shithub.us/ori/git9/724c516a6eda0063439457a6701ef0d7e...
as far as I'm aware, the OpenBSD implemention of git, Game of trees, is adapting this code too.
Maybe I’ll implement this tonight.
https://git-annex.branchable.com/
I like it a lot so far, and I think it could be used for "cloud" stuff, not just backups.
I'd like to see Debian/Docker/PyPI/npm repositories in git annex, etc.
It has lazy checkouts which fits that use case. By default you just sync the metadata with 'git annex sync', and then you can get content with 'git annex get FILE', or git annex sync --content.
Can anyone see a reason why not? Those kinds of repositories all seem to have weird custom protocols. I'd rather just sync the metadata and do the query/package resolution locally. It might be a bigger that way, but you only have fully sync it once, and the rest are incremental.
If you've ever tried that, then git will start to choke around a few gigabytes (I think it's the packing/diffing algorithms). Github recommends that you keep repos less than 1 GB and definitely less than 5 GB, and they probably have a hard limit.
So what git annex does is simply store symlinks to big files inside .git/annex, and then it has algorithms for managing and syncing the big files. I don't love symlinks and neither does the author, but it seems to work fine. I just do ls -L -l instead of ls -l to follow the symlinks.
I think package repos are something like 300 GB, which should be easily manageable by git annex. And again you don't have to check out everything eagerly. I'm also pretty certain that git annex could support 3TB or 30TB repos if the file system has enough space.
For container images, I think you could simple store layers as files which will save some space for many versions.
There's also git LFS, which github supports, but git annex seems more truly distributed, which I like.
I find git-annex to become a bit unwieldy at around 20k to 30k files, at least on modest hardware like a Raspberry Pi or a core i3.
(This hasn't been a problem for my use case, I've just split things up into a couple of annex repos)
Oh well there goes my plan to use it for the 500 million files we have in cloud storage!
This person said they put over a million files in a single git repo, and pushed it to Github, and that was in 2015.
https://www.monperrus.net/martin/one-million-files-on-git-an...
I'm using git annex for a repository with 100K+ files and it seems totally fine.
If you're running on a Raspberry Pi, YMMV, but IME Raspberry Pi's are extremely slow at tasks like compiling CPython, so it wouldn't surprise me if they're also slow at running git.
I remember measuring and a Rasperry Pi with 5x lower clock rate than an Intel CPU (700 Mhhz vs. 3.5 Ghz) was more like fifty times slower, not 5 times slower.
---
That said, 500 million is probably too many for one repo. But I would guess not all files meed a globally consistent version, so you can have multiple repos.
Also df --inodes on my 4T and 8T drives shows 240 million, so you would most likely have to format a single drive in a special way. But it's not out of the ballpark. I think the sync algorithms would probably get slow at that number.
It's definitely not a cloud storage replacement now, but I guess my goal is to avoid cloud storage :) Although git annex is complementary to the cloud and has S3 and Glacier back ends, among many others.
We could certainly shard our usage e.g. by customer - they're enterprise customers so there aren't that many of them. We wouldn't be putting the files themselves into git anyway - using a cloud storage backend would be fine.
We currently export directory listings to BigQuery to allow us to analyze usage and generate lists of items to delete. We used to use bucket versioning but found that made it harder to manage - we now manage versioning ourselves. git-annex could potentially help manage the versioning, at least, and could also provide an easier way to browse and do simple queries on the file listings.
The basic idea is that each file targeted by `git annex add` gets replaced by a symlink pointing to its content. The content is managed by git annex and lives as a checksum-addressable blob in .git/annex. The symlink is staged in git to be committed and tracked by the usual git mechanisms. Git annex keeps a log of which host has (had) which file in a branch named "git annex". (There is an alternate non-symlink mechanism for Windows that I don't use and know little about.)
I use git annex in the git LFS-like fashion to store experimental data (microscope images, etc.) in the same repository as the code used to analyze it. The main downside is that you have to remember to sync (push) the git annex branch _and_ copy the annexed content, as well as pushing your main branch. It can take a very long time to sync content when the other repository is not guaranteed to have all the content it's supposed to have, since in that scenario the existence and checksum of each annexed file has to be checked. (You can skip this check if you're feeling lucky.) Also, because partial content syncs are allowed, you do need to run `git annex fsck` periodically and pay attention to the number of verified file copies across repos.
There was a project of backing up the internet archive by using git-annex (https://wiki.archiveteam.org/index.php?title=INTERNETARCHIVE...). Basically the source project would create repositories of files and users like you and I would be remote repositories; we would get content and claim that we have it, so that everyone would know this repository has a valid copy on our server.
I’m not seeing a way to do anything similar with dolthub.
Some pointers to related ideas:
https://en.m.wikipedia.org/wiki/Persistent_data_structure - https://docs.datomic.com/cloud/index.html - https://opencrux.com/ - https://github.com/attic-labs/noms - https://researcher.watson.ibm.com/researcher/files/us-leejin...
If you're already comfortable with git, the Git Internals section of the book may be better: https://git-scm.com/book/en/v2/Git-Internals-Git-Objects
Isn't the point of a database the ability to query? Why else would you want "Git as a NoSql database?" If this is really what you're after, maybe you should be using Fossil, for which the repo is an sqlite database that you can query like any sqlite database.
I was thinking about this in terms of the new CloudFlare Durable Objects open beta: https://blog.cloudflare.com/durable-objects-open-beta/ Storing data is really only half the battle.
How could you reimplement the poorest of a poor man's SQL on top of a key/value store? The idea I came up with is: whatever fields you want to query by need an index as a separate key. Maybe this would allow JOINs? Probably not?
One of the main advantage of this approach is that replication becomes trivial, it's just a matter of running `git fetch` on all the repositories.
The main disadvantage is that NodeDB only has one implementation (Java), and no CLI to play with it.
[1]: https://gerrit-review.googlesource.com/Documentation/note-db...
This also reminds me of Dolt: https://github.com/dolthub/dolt which I believe has been on HN a couple times
> A distributed database built on the same principles as Git (and which can use git as a backend)
You mean "at most" two.