whilo

Hey. Hybrid in which sense?

I have integrated Stratum's columnar indices as a secondary index in the new query engine of https://github.com/replikativ/datahike itself, so for numerical data you will be able to use Datalog/SQL to have combined (OLTP, OLAP, ...) processing. Same for proximum (persistent HNSW vector index) and scriptum (persistent Lucene).

Stratum already can be copy-on-write updated online with better write throughput than purely columnar alternatives (Stratum uses a persistent B-tree over column chunks) as far as I tested. I have not compared it in benchmarks yet though, DuckDB recommends to not update it online for instance. But it depends on the workload, if you do random access writes the columnar layout overhead will still be a slow-down compared to OLTP/Datahike's row/entity-wise indices. Also storing fully variable strings in a column is inefficient, for this you want the entity-wise indices.

Hi HN - I’m the author of Stratum.

The headline benchmark result is that on 10M rows, Stratum is faster than DuckDB on 35 of 46 single-threaded analytical queries, despite running entirely on the JVM.

But the main idea is actually branchable tables: you can fork a table in O(1), keep copy-on-write snapshots, and query different branches through SQL.

It speaks the PostgreSQL wire protocol, so psql/JDBC/DBeaver work out of the box.

Benchmarks, methodology, and repo are linked from the page. Happy to answer questions.

Hi HN - I’m the author of Stratum.

Stratum is a columnar SQL engine built on the JVM exploring two ideas:

• SIMD execution using the Java Vector API

• copy-on-write branching for tables

It speaks the PostgreSQL wire protocol, so tools like psql, JDBC, and DBeaver work out of the box.

The engine is pure JVM (no JNI). In benchmarks on 10M rows it performs competitively with DuckDB on many analytical queries.

GitHub repo: https://github.com/replikativ/stratum

Benchmarks and methodology are described in the docs.

Happy to answer questions - feedback very welcome.

No, he lives in Switzerland now (Lugano as far as I know). Switzerland is a big magnet for European talent, as is Germany to a lesser degree. My point though was that Germany was not lacking the thinkers to drive a technological revolution, it is rather the society that does not really have the mindset for radical changes and forward thinking.

As a German coming from Mannheim, and now living on the Canadian West coast, I have to say that this is exactly the mindset that makes it so difficult to innovate in Germany. While people have a top education and know everything they need to, they don't have a "digital mindset". I think of myself now as computing process, and see myself as a cyberneticist in the German philosophical tradition of Hegel, Marx, Hilbert, Gödel and Bloch, but even I more often than necessary mistrust innovations.

Almost all the science fiction and cybernetic work of the last 75 years came either out of the Eastern block, or out of the US. There is basically no German sci fi vision, and there is extreme reluctance to speculative, big picture thinking as it is pursued in Silicon Valley. Software companies grow quickly and need a very different approach. Silicon Valley mostly understands this scaling aspect and winner take all markets really well. German investors are cheap and extremely risk averse, I tried to build a software company in Germany and it is hard.

While a lot of the current AI work was also done in Germany, e.g. by Schmidhuber, Germans are stuck in their business model. I recommend Münchau's book "Kaput" (or one of his podcast interviews) on how poorly Germans have adapted to the non-industrial aspects of a modern economy (read: "services"). I really hope that more tech founder thinking like Benz, Bosch or Siemens returns to Germany in a modern form. But I don't see it yet, and Germans are still super reactive and conservative to larger changes. The Greens tried to think a bit out of the box, and were heavily punished for it. In general there is basically no political representation for building a new successful economy. At best there is this nice little narrative about the long established "Mittelstand", which has produce almost zero software companies. The first step right now would be to own the idea of the EU, and wanting to win instead of complaining.

I see, so your problem was that you wanted to scan over all Datoms for one entity over a time period and you would have needed to have an EVAT index? In Datahike it would be fairly simple to add new indices like this.

Yes, access management must not incur a large overhead, that is why many systems have a separate restricted way to express and track rules. My hunch is that it still would be better to keep it in Datalog and specialize the query engine that it is fast on these (potentially restricted) rules and relations.

Interesting, honestly speaking we have not thought about time series data a lot yet, but I think we should be able to provide custom indices and extend Datalog with more efficient query primitives, if this is necessary. Can you elaborate a bit? I have used HDF5 binary blobs for tensors of experimental recordings (parameter evolution in spiking neural networks) in Datomic a few years ago and it is definitely possible to integrate external index data structures, but eventually the query engine will need to be aware of how to join them efficiently.

W.r.t. security, our current approach is to shard access rights and encryption on a database level and just provide many databases, one for each user. This is obviously not the most space efficient, but the most general approach. If users can share access keys and data we can also do structural sharing between these instances and factorize further. We envision doing joins potentially over dozens of distributed Datahike instances in a global address space during single queries. Since the indices are amortized data structures it does not make too much sense to encrypt chunks of them for different users as this defeats the optimality guarantees of B+-trees, i.e. you could have very bad scan behaviour over huge ranges of encrypted Datoms. How have you tried to partition the data? This is an interesting problem.

We can also expose the datahike-server query endpoint directly and you can write static checks for access right restrictions. We only do this so far to limit the usage of builtin functions to safe ones, but you could also go ahead and do the same for more complex access controls. Some work in this direction for Datahike has also been done here: https://github.com/theronic/eacl Doing this openly on the internet will also require a resource model to fend of denial of service attacks, fortunately Datalog engines can have powerful query planners and we can restrict our runtime to limited budgets as well.

Hey, one of the core architects of Datahike here. As a Clojure company we are also super happy that Nubank made Clojure and Datomic much more credible with this move. While Datomic is obviously much more mature, it is important to understand that we have a different scope in our goals than Datomic. Datomic is mostly built as a more convenient backend database for corporate environments and is highly tailored towards AWS and a business environment where the costs of operating and depending on these cloud services is acceptable, which is only the most profitable, but small, slice of the whole market. Even when Datomic gets open-sourced it will not be automatically built with other than these design goals in mind.

Our goals on the other hand go even beyond thinking about this backend market; we want to use Datalog really as a distributed systems environment and extend Datahike to all endpoints including the browser and IOT development:

https://www.youtube.com/watch?v=A2CZwOHOb6U

Our main intention was never to just build an open-source Datomic. But it made too much sense not to do it as a first step. In fact we also really hope that Datomic will be open-sourced such that we can merge our efforts. But given the current governance model of Clojure and Datomic we do not yet foresee that open-sourcing Datomic alone would address a large section of our plans. We are ahead of Datomic already in a few areas:

We have funded development for ClojureScript support going for instance and in comparison to Datomic all our efforts where from the beginning aimed at this, we in fact provide more a set of libraries and abstractions that can stand on their own and that you can compose in different ways than having a top-down design that we then unbundle into libraries. This made it much easier for us to evolve and reuse our stack despite the pivot we did from replikativ to Datahike.

Regarding maturity we have worked hard during quarantine to address some of our pain points:

1. We significantly improved our write throughput and Datomic performance is in reach (close to release), https://github.com/replikativ/datahike/pull/201 2. We have a first version of our server API available and will extend this in the next months to provide Datomic-style local querying https://github.com/replikativ/datahike-server/ 3. We recently provided Java bindings https://lambdaforge.io/2020/05/25/java-api.html

Over the last year we also built a cooperative that has more than 5 people working on this full-time and we aim to grow even faster next year and really bring Datalog to the community and the masses. If Datomic gets properly open-sourced, we will get there even faster.