Great to see the advances made in Crux :-)

I'm working on a versioned database system[1] in my spare time which offers similar features and benefits. The core is Java based, whereas the Server is written in Kotlin using Vert.x. A Python client as well as currently a TypeScript based client for a non-blocking, asynchronous HTTP-Server exists.

The data store can easily be embedded into other Java/Kotlin based projects without having to use the Server indirection.

Lately I changed a lot of stuff regarding the set-oriented query engine which uses an XQuery derivate to process both XML and JSON as well as structured data. For now I've integrated rule based index rewriting stuff to answer path queries and path queries with one predicate through the index, filtering false positives if necessary. Next, I'll add a twig based operator and AST rewrite rules to answer multiple path queries with a single scan over smaller subtrees.

Furthermore, I want to use the new Java Foreign Memory API to memory map the storage file and change the tuple-at-a time iterator model to fetch batches of tuples at once for better SIMD support through the JVM.

I've also thought about replacing the Kafka backend with SirixDB itself, as SirixDB doesn't need a WAL for consistency... so SirixDB can hopefully horizontally scale at some point in the future.

Some of the futures so far:

    - storage engine written from scratch
    - completely isolated read-only transactions and one read/write transaction concurrently with a single lock to guard the writer. Readers will never be blocked by the single read/write transaction and execute without any latches/locks.
    - variable sized pages
    - lightweight buffer management with a "kind of" pointer swizzling
    - dropping the need for a write ahead log due to atomic switching of an UberPage
    - rolling merkle hash tree of all nodes built during updates optionally
    - ID-based diff-algorithm to determine differences between revisions taking the (secure) hashes optionally into account
    - serialization of edit operations for changed subtree roots to make comparisons between consecutive revisions or subtrees thereof incredibly fast (no-diff computation needed at all)
    - non-blocking REST-API, which also takes the hashes into account to throw an error if a subtree has been modified in the meantime concurrently during updates
    - versioning through a huge persistent and durable, variable sized page tree using copy-on-write
    - storing delta page-fragments using a patented sliding snapshot algorithm
    - using a special trie, which is especially good for storing records sith numerical dense, monotonically increasing 64 Bit integer IDs. We make heavy use of bit shifting to calculate the path to fetch a record
    - time or modification counter based auto commit
    - versioned, user-defined secondary index structures
    - a versioned path summary
    - indexing every revision, such that a timestamp is only stored once in a RevisionRootPage. The resources stored in SirixDB are based on a huge, persistent (functional) and durable tree 
    - sophisticated time travel queries

As I'm spending a lot of my spare time on the project and would love to spend even more time, give it a try :-) Any help is more than welcome.

Kind regards Johannes

[1] https://sirix.io and https://github.com/sirixdb/sirix