Hacker News Reader

Comment by gumby | Hacker News Reader

I don't want a github for data, I want a git for data. The one reason that POS Perforce is used in game development is that it can handle binary files, especially large ones. LFS is..OK.

And this company isn't really even aping GitHub given that they are generating the content too.

View on HN

Have you heard of Dolt? It's git for data.

https://github.com/liquidata-inc/dolt

What would you use a "git for data" for?

Data?

Although having said then when you say data to me, I do imagine some level of immutability. Logging scientific results using git would be pretty good if required, in the sense that there is a habit to just use folders and text files which is fine at the time but is really really hard to take over sometimes (Like code written by scientists with their single-letter variable names and hatred of functions, in my experience of Fortran - yuck)

I think I meant more in terms of use cases. So it sounds like one of your use cases is logging scientific results...?

We’ve released our branched versioning system for data at http://dvid.io. It’s entirely open source and uses a science HTTP API with pluggable data types and a versioned key-value backend. I’m currently developing a new backend for it: DAGStore, a lower-level ordered key-value store that will have an explicit immutable store and is tailored for distributed versioning of big data.

Recording outputs of analytical runs or deep learning models; the ability to say ‘this is the output of program X v <git hash> run on the output of program Y <git hash> whose source data was Z <data-git hash>” for all the reasons of repeatability and audit ability needed for actual science.

Reply on news.ycombinator.com