Hacker News Reader

Comment by GauntletWizard | Hacker News Reader

I've been seriously considering using Git for all sorts of oft-changing-but-rarely-majorly data. Lists of books in my bookshelves.

The other problem is that I sorta want transactional-database features on top of these things. Git does this well. I also want fast indexing on parts. Git does not do this well. I am considering writing a "standard" for the dumping of sqlite to git, so that I can just delegate this out; Any transaction can be expressed as a git commit, and I can run both at once for both the durability and the reasonable indexing; The sqlite database can be re-created and reindexed whenever, and it also sorta works for backups...

Definitely just spinning my wheels, though. We'll see where databases take us next.

View on HN

> oft-changing-but-rarely-majorly data

I think you're referring to SCDs, and there are plenty of well-defined ways to track these within relational databases:

https://en.wikipedia.org/wiki/Slowly_changing_dimension

Why git?

Would fossil fit this bill?

Somewhat git-compatible, based on sqlite3.

Specifically the fossil repo is a sqlite database.

I have a cron script stashing whatever is currently in an obsidian vault into a fossil repo. There's a fossil addremove command that makes that very easy. Thus distributed backups of said vault.

What does "somewhat git-compatible" mean? Can it sometimes use existing git repos, or is the mental model close enough to pick up without learning much?

> is the mental model close enough to pick up without learning much?

I would say git core concepts are pretty similar to fossil concepts, but actual plumbing implementation details are pretty distinct.

The major difference that I remember from a day-to-day "git porcelain" perspective is that rebases and other types of history rewriting are very discouraged.

For a Rosetta stone of somewhat comparable commands: https://fossil-scm.org/home/doc/trunk/www/gitusers.md

> Can it sometimes use existing git repos

You can, but it's kinda a lot of really slow busywork. And you lose some of the not-file-dvcs features of Fossil, but it is possible.

This page explains how: https://www.fossil-scm.org/home/doc/trunk/www/mirrortogithub...

In your git & sqlite setup, I'm not sure which way round you are thinking - which is the SSOT (Single Source Of Truth) and which is the handy cache.

I've been working on a tool that treats the git repo as the SSOT then lets you dump out all kinds of formats for data work including a sqlite DB. I haven't had as much time as I would like on it but it's at https://pypi.org/project/DataTig/

If anyone is coming in late, I have discovered https://immudb.io/ , which claims to be a lot of what I'm looking for. Transactional SQL database, indexed, yet immutable data and verifiable chains of transactions.

Maybe this https://github.com/dolthub/dolt

Dolt is really close and yet just doesn't feel right; I don't want to "commit" between transactions, I want every transaction to be a commit.

You can do this with a setting:

https://docs.dolthub.com/sql-reference/version-control/dolt-...

That's awesome. I will have to try that out.

Reply on news.ycombinator.com