lichtenberger

Same here, was a backend software engineer, now even embedded ;-)

If anyone is up to building a new frontend, that would be awesome (of course, work could also be split between interested people) :-)

https://github.com/sirixdb/sirix/issues/627

Oh yes, I've heard about TerminusDB, but completely forgot, that it exists. Thanks :)

Thanks, I remember your PR, thanks again :-)

Can someone (an admin) maybe add the direct link to https://github.com/sirixdb/sirix in the URL field. Totally forgot, that it's possible to add both a text and a URL ;)

This could be very interesting, too (our query engine now with full JSON support -- using sophisticated set-oriented join and aggregate optimizations):

Separating Key Concerns in Query Processing - Set Orientation, Physical Data Independence, and Parallelism

http://wwwlgis.informatik.uni-kl.de/cms/fileadmin/publicatio...

I've also ported their indexing ideas regarding XML to JSON, such that we can easily index whole paths with typed values as described in the README :-) the indexes itself are also versioned, of course and always updated.

First AST index rewrite rules for the query engine Brackit have also been added (Brackit is regargetable, so other data stores can easily implement a couple of interfaces).

Thanks, Dolt it awesome. I think it's probably the only DBS with branching/merging capabilities as of now.

Sirix from the ground up was built having (bi)temporality and easy audits / append-only paradigm in mind.

The other very similar DBS in this regard seems to be Datomic (as it also uses a persistent index structure), but it doesn't version the pages itself.

Thank you, @BozeWolf for the clickable links!

I'd be honored if you take a closer look, of course and best of all obviously are contributions. Thanks!

I'm working on an append-only (immutable) (bi)temporal DBS[1] in my spare time, which transforms CRUD operations into an event store, automatically providing an audit log for each stored node, while the nodes are stored with immutable node-IDs, which never change. As the contents stored are based on a custom binary JSON format also a rolling hash can optionally be built, to check if a whole subtree has changed or not.

You can also add commit comments, revert to a specific revision (while preserving all revisions in-between)...

The system uses persistent index data structures to share unchanged pages between revisions.

The intermittant snapshots are omitted. Rather the snapshot is spread over several revisions, applying a sliding snapshot algorithm on the data pages (thus, avoiding write peaks, while at max a predefined number of page fragments has to be read in parallel to reconstruct a page in-memory).

[1] https://github.com/sirixdb/sirix | https://sirix.io | https://sirix.io/docs/concepts.html

Wouldn't it be possible to contribute the faster algorithm to the JDK (e.g. the Instant class)? Maybe you'd need some iterations, but do you see any downsides?

You can use Datomic for instance (mentioned already in your article IIRC!?) or SirixDB[1] on sich I'm working in my spare time.

The idea is an indexed append-only log-structure and to use a functional tree structure (sharing unchanged nodes between revisions) plus a novel algorithm to balance incremental and full dumps of database pages using a sliding window instead.

[1] https://sirix.io | https://github.com/sirixdb/sirix

Yes, I think what is always most valuable are the underlying concepts :-)

I wonder if we'll in the near future also see JVM based in-memory or hybrid OLAP database systems, which will make use of SIMD instructions and columnar storage layouts with the incubating Vector API.

It would be also interesting to see how we can process semi-structured data in a similar way.

Friends of mine also went to Nepal with Clowns without Borders. They even did a documentary :-)

https://anita.bertolami.net/project.html

https://www.youtube.com/watch?v=74qK0n8kA-w

No, it's a real world data set, but uncompressed.

Not sure if it's even possible on Github without Git LFS? But even then, the files seem to be too large.

Working again...

You can do this with a tree-based persistent/functional data-structure, too.

For instance the single writer (per resource) in the evolutionary, append-only database system[1], I'm working on in my spare time, can simply revert the whole resource (resource is like a table in the relational DB jargon) to a past revision. Once a new commit is issued a new revision gets appended and the whole history in-between is preserved. Thus, you can, for instance, retrieve individual changes and cherry pick these from the revisions in-between.

[1] https://github.com/sirixdb

Phil Bagwells work is amazing. Especially the Hash Array Mapped Trie (HAMT) for me was an eye opener, as we had implemented a simple arrays based trie more or less for 64bit integers before and I always wondered how to compress the null pointers to children, which are non-existent. Nowadays I'm using ideas from the HAMT and the Adaptive Radix Trie (ART) to compress the trie nodes. Cool thing of course is also that each previous version is preserved through structure sharing and copy-on-write plus path copying (functional data structure, again).

I guess Datomic is the most mature system, which uses a persistent in-memory index structure, mapped to a common on-disk storage system.

That said, nowadays it should probably be possible to store the persistent data structure in a log-structured file for instance and to make use of fast random-reads of enterprise PCIe SSDs without the indirection of another data store / index to traverse for each index page.

Ideally the temporal database system stores both the state after the modification as well as the events/changes which lead on to the new state.

I think this is a good (short) read about the relation: https://event-driven.io/en/temporal_tables_and_event_sourcin...

There's the valid time on the one hand when something is valid in the real world and the transaction time, the time at which it's known to the database system, when it's recorded/committed.

Fastutil is amazing. Specialized collection types for almost everything.

Really awesome work :-)

I've implemented the Fast Match / Simple Edit script algorithm almost 10 years ago for my Master's thesis[1] for my database project[1][2] in order to import revisions of files with a hopefully minimal edit number of edit operations between the stored revision and a new one (back then it was for XML databases).

The diffing was only one aspect for the visual analytics approach to compare the revisions (tree structures) visually [4]. Internally the nodes are addressed through dense, ascending 64bit ints stored in a special trie index. Furthermore, during the import optionally changes are tracked as well as a rolling hash is stored for each node optionally. After the import you can query the changes or execute time travel queries easily.

Technically, a tree of tries is mapped to an append-only data file using a persistent data structure (in the functional sense), COW with path copying and a novel sliding snapshot algorithm for the leaf data pages itself.

I always have the vision to implement different visualizations to compare the revisions in a web frontend, but I'm currently spending my time on improving the latency of both writes and reads.

Thus, if someone would like to help, that would be awesome :-)

Kind regards

Johannes

[1] https://github.com/JohannesLichtenberger/master-thesis/blob/...

[2] https://github.com/sirixdb/sirix

[3] https://github.com/sirixdb/sirix/tree/master/bundles/sirix-c...

[4] https://youtube.com/watch?v=l9CXXBkl5vI

Basically my research project[1] I'm working on in my spare time is all about versioning and efficiently storing small sized revisions of the data as well as allowing sophisticated time travel queries for audits and analysis.

Of course all secondary user-defined, typed indexes are also versioned.

Basically the technical idea is to map a huge tree of index tries (with revisions as indexed leave pages at the top-level and a document index as well as secondary indexes on the second level) to an append-only file. To reduce write amplification and to reduce the size of each snapshot data pages are first compressed and second versioned through a sliding snapshot algorithm. Thus, Sirix does not simply do a copy on write per page. Instead it writes nodes, which have been changed in the current revision plus nodes which fall out of the sliding window (therefore it needs a fast random-read drive).

[1] https://github.com/sirixdb/sirix

As far as I understand there's also Zeitgitter[1], to provide similar proof.

[1] https://github.com/zeitgitter/zeitgitterd

Thanks, will watch later on :-)

Yes, I think Loom is an amazing step forward and it's brilliant to just change the blocking methods and thus existing code might already benefit with minor changes.

The mentioned twitter discussion is also pretty interesting IMHO :-)