lichtenberger

(first since a few years ago), but thanks so much :)

Oh BTW: I'm also no native english speaker, so, any hint on stuff, which is wrong is also greatly appreciated :-) just got my first spelling error correction pull request. You guys are really awesome, thanks :)

Wow, great summary, thanks from me, too :)

And I also have an idea how to save up even more space on disk/on the flash drive :-)

Oh and I think with pointer swizzling (Java object references instead of references to the page-IDs in the buffer manager) we can gain quiet some speed :-)

And maybe adding memory mapped files in the future.

By the way, would it make sense to work on these visualizations again for comparing revisions?

https://youtu.be/l9CXXBkl5vI

Or would anyone be interested to port the processing.org stuff to the web? Sadly I'm a backend developer and feel a bit overwhelmed by all the front-end frameworks JavaScript stuff), but I think the visualizations for comparing revisions are really helpful.

More information: https://github.com/JohannesLichtenberger/master-thesis/raw/m...

By the way, if you check it out and find bugs, please let me know :-) It's still at an early stage and for sure has some rough edges.

Thanks and have a great day :)

Oh and maybe another interesting article: https://hackernoon.com/sirix-io-why-copy-on-write-semantics-...

Hey, I think best explanation is this:

https://stackoverflow.com/questions/51533143/temporal-vs-tim...

:-)

Will have to sleep now... but yes, the ideas and the first prototype emerged already in I think 2006 from Marc Kramis (back then it internally was named Idefix, then Treetank... but I think Treetank is pretty strange ;-))

My latest article released a few days ago can be found here:

https://hackernoon.com/asynchronous-temporal-rest-with-vert-...

Some of the use cases can be found here:

https://sirix.io/documentation.html

The main distinctive features are our main document store index, features inherited by ZFS (checksums in parent pointers, the main index, a trie, compression and hopefully soon encryption of page-fragments, always consistent on-disk, log-structured without the need of a WAL...), versioned user-defined indexes, highly concurrent data structures (every transaction has access to one snapshot and we only allow one read/write transaction, parallelization has to be done by the client code) and record-level versioning (also a novel sliding window algorithm -- whereas a slightly other implementation is patented by the founder Marc Kramis).

I might implement pointer swizzling for the upcoming 1.0.0 release, should speedup Sirix considerably :-)

https://sirix.io/features.html

I think XQuery is great for querying JSON data. That said in the future I'd also implement something based on Spark to distribute queries...

No, I had to "fork" it, when it still was on Google Code, because of temporal extensions and lately JSON stuff. Sebastian sadly doesn't seem to be on Github a lot. Would love to contribute my changes, but I also have to have the ability to change/fix stuff on my own. That said I really love the whole idea of Brackit, a flexible query compiler framework with proven rules, to rewrite the AST usable by any data store... and I want to plugin rewrite rules for index accesses as well as operators in the first place and so on (the indexes can be manually created and queried as of now). Sirix basically as of now has to use raw document scans, but I'd love to work on rewrite rules and then on a cost based optimizer (that said it would be great to find someone who is experienced and can help a bit). His Ph.D. thesis can be found here: http://wwwlgis.informatik.uni-kl.de/cms/fileadmin/publicatio...

And Sebastian worked under supervision from Dr. Theo Härder :-)

By the way I've added some documentation lately on https://sirix.io :-)

Okay thanks, would need to change the example and add it to the token checker before inserting elements/attributes...

Wow, thank you so much for posting/mentioning :-)

Have a great evening Johannes

Wow, crazy :-) thanks for posting

Hi all,

posted already yesterday, but maybe it's of some value and not so many people have seen it. Disclaimer: I'm the author :)

Kind regards and best weekend Johannes

Hi all,

I want to release version 1.0.0 soon, but would love some input, as I added a complete native JSON storage about three weeks ago (several layers) :-)

I've just published an article called Asynchronous, Temporal REST with Vert.x, Keykloak and Kotlin Coroutines: https://hackernoon.com/asynchronous-temporal-rest-with-vert-...

And another one about some use cases and internals: https://hackernoon.com/sirix-io-why-copy-on-write-semantics-...

I've also added a bunch of documentation on the website :-)

Have a great weekend :)

Kind regards Johannes

BTW: What's your take on random read speed? I think with SSDs now and maybe NVRAM in the future we can address physical storage more fine granular than with HDDs and for instance with http://sirix.io data is only clustered during writes (batched and synced to the flash drive) and it versions variable length in-memory and physical pages such that not every record has to be copied and written again. For instance to reconstruct a page we have to read from random places (maybe in parallel) to reconstruct a page in-memory. Thus Sirix relies on fast random read speed and on find granular access to the physical stored records.

I'm not entirely sure, but I'll apply the technique for my open source storage system. I have reference-instances with a log-key, persistent-key (file offset for instance) and will (re-add) the Java object references. In order to set them to null when the page is going to be evicted from the buffer cache I'll add a simple in-memory map with the proposed child => parent mapping, such that I can set the in-memory page reference to null again.

Can we access the publication for free? Someone changed the link, originally it was:

https://scibib.dbvis.de/uploadedFiles/MotionRugsPreprint.pdf

"For the Z-Order Curve we found that it is prone toproduce “Phantom-Splits”, visual artifacts creating the impression of a physical split of the entities when there is none. The Hilbert Curve insteaddisplays the spatial dynamic better than the Point-QuadTree and shows fewer artifacts than the Z-Order."

Thanks for the suggestion/tip :-)

I'm familiar with merkle trees :-)

So much stuff you can learn in the computer science world. For me the blockchain still is more or less a black box, should read a book about this stuff, as it seems to be hyped everywhere ;-)

Thanks :)

Thanks, great use case and I have to say I have to read about genomics... :-)

Hm, where else are Lowest Common Ancestors in tree-structures useful? Storing for instance ORDPATH/DeweyIDs allows to simply check for a common prefix (they are hierarchical node labels). I think maybe for locking in a storage system with multiple read/write transactions or to determine which of two given nodes is the firs one in preorder, which is useful for XQuery/XPath processing (to determine the so caslled document order). Can anyone think of other usages? Or for having hierarchical node labels in trees?

Ha, Rick and Morty, best show ever ;-)

Good catch :-) right, but still merging/compaction work has to be done. Maybe too much, if you just need to fetch a value by its key and thus just an equality scan is needed (no range scans or other comparisons). For the latter case I've implemented an AVL-tree, which is also versioned and stored in our record pages and best read fully in-memory (but doesn't have to). For sure there are plenty of optimizations and for instance also spatio-temporal indexes or full-text indexes possible, but I guess first looking into cost-based rewrite rules for the query compiler and replication/partitioning for horizontal scaling. Too many ideas I guess ;-) but the best would be to have a great open source community :-)

Haven't heard of, but the storage system is heavily inspired by ZFS and by putting some of the ideas (plus adding our own obviously ;)) to the sub-file level: https://kops.uni-konstanz.de/bitstream/handle/123456789/2769...

Basically, I'd like to provide the I/O layer with memory mapped file regions, such that it's simply a configuration option if you use the RandomAccessFile implementation or maybe one based on memory mapped files. I think for the JVM there's Chronicle. Thanks so much for asking :-) and by the way any contribution would be the best I can hope for