Hacker News Reader

Comment by baby | Hacker News Reader

So using git instead of a database when a database would be more suited?

Git is a database. It's designed for storing the changes made to files over time.

But hey, if there's a different kind of database that you think would be better here, I'd be fascinated to see it. Just make sure that it can

1. handle major structural changes to the files being tracked (e.g. JSON schema revisions)

2. store and retrieve these changes as efficiently as git

3. do the above with as little work required by the user

Good luck!

the point is that this isnt particularly useful except to just see a raw diff of the files. the example shows some interesting data points about fires changing periodically but there's basically nothing you can do with that information unless you put it in a real database.

Sure - that's what I did with my PG&E outages project (https://simonwillison.net/2019/Oct/10/pge-outages/). I wrote a Python script that iterated through the git commits and used them to create a SQLite database so I could run queries.

Essentially I was using the commit log as the point of truth for the data, and building a database as an ephemeral asset derived from that data.

so full cycle, back to a database :)

What's different here is what you treat as the point of truth.

If the point of truth is the git repository and its history, then the SQLite database that you build from it is essentially a fancy caching layer - just like if you were to populate a memcached or redis instance or build an Elasticsearch index.

A decentralised revision control database revolution is brewing!

Is that sarcasm? Because again you're describing a database.

Not sure if databases really meet #3. It may be hard for a database to beat the simplicity of Git when all you want is add file revisions and look at diffs.

I actually think git is MUCH better suited to this than a traditional database.

I've built track-changes-over-times things in relational databases before and it's a real pain to implement!

I've actually started using a GitHub repository to back up the PostgreSQL database that powers my blog, because it's a really cheap way of getting a revision history of that database without having to write any extra code: https://github.com/simonw/simonwillisonblog-backup/commits/m...

I think you want temporal tables for historical dimensions. Postgres doesnt have good support as its a bolt on. MariaDB has good support for it now.

I'm not sure I understand the difficulty, your columns are the fields in your object, the rows are the different points in time

Didn't cha know, innovation in the tech industry is all about using things differently than intended and building new compositions that result in the same end goals as older less sophisticated compositions.

A database is probably better if the schema doesn't evolve and the present state of the data fits neatly into a relational schema, because then you are just extending it with versioning and the transformation to a version tracking database is fairly mechanical.

But if the data is already in a textual format where [or which can be canonicalized so that] each line is an atomic unit, Git + existing common diff tools get you a lot with very little work and are resilient to schema changes.

You could also reduce the data to EAV triples and do a change-tracking DB for that, wwhich would get you immunity to schema changes at the cost of losing all the value that a tracking DB has for known-schema data.

So, really, I'd say Git is the easy & general solution, though a DB might be worth the extra work if the schema was known to be fixed.

I’ve put a git-scraped history of 12 years of the FAA registration database into an EAVT database (Datomic) and it worked reasonably well.

Can you expand on why a database would be more well suited?

It seems to me that if the goal is to track changes over time, git is very well suited to that.

git is not suited to search through the different fields that have changed (it's content agnostic). A database allows you to plot and search efficiently.

Well you could say folders are like "tables" and files are like "fields" within those tables...

The main difference between git and a traditional DB is querying and relationships between fields

Reply on news.ycombinator.com