Hacker News Reader

Comment by Conlectus | Hacker News Reader

Though I have only a recreational interest in datastores, I would be pretty wary of a service that claims to strap a Consistant-Unavailable database (Postgres) to an Inconsistent-Available database (ElasticSearch) in a useful way.

Doing so would require ElasticSearch to reach consensus on every read/write, which would remove most of the point of a distributed cluster. Despite this, ZomboDB's documentation says "complex aggregate queries can be answered in parallel across your ElasticSearch cluster".

They also claim that transactions will abort if ElasticSearch runs into network trouble, but the ElasticSearch documentation notes that writes during network partitions don't wait for confirmation of success[1], so I'm not sure how they would be able to detect that.

In short: I'll wait for the Jepsen analysis.

[1] https://www.elastic.co/blog/tracking-in-sync-shard-copies#:~...

View on HN

> Doing so would require ElasticSearch to reach consensus on every read/write

ZomboDB only requires that ES have a view of its index that's consistent with the active Postgres transaction snapshot. ZDB handles this by ensuring that the ES index is fully refreshed after writes.

This doesn't necessarily make ZDB great for high-update loads, but that's not ZDB's target usage.

> They also claim that transactions will abort if ElasticSearch runs into network trouble...

I had to search my own repo to see where I make this claim. I don't. I do note that network failures between PG & ES will cause the active Postgres xact to abort. On top of that, any error that ES is capable of reporting back to the client will cause the PG xact to abort -- ensuring consistency between the two.

Because the ES index is properly refreshed as it relates to the active Postgres transaction, all of ES' aggregate search functions are capable of providing proper MVCC-correct results, using the parallelism provided by the ES cluster.

I don't have the time to detail everything that ZDB does to project Postgres xact snapshots on top of ES, but the above two points are the easy ones to solve.

ZomboDB scratches an itch in a way I find fascinating, though I have yet to do more with it than shoehorn it into a prototype that was a bit square-peg-in-a-round-hole. It is in my catalogue of technologies I hope exploit someday. And I hope you're enjoying the work and making some $$$ too.

While I'm here might I ask, are you finding the hosted PostgreSQL services (AWS, Azure, etc.) growing or shrinking your market opportunities? Also, does it play nice with Citus?

I don’t pay any attention to what the cloud providers are doing. I have no control over them so….. eh.

ZDB is still alive and well but I have a real job now too.

I suppose I must have misinterpreted this line

> As such, any sort of failure either with ZomboDB itself, between Postgres and Elasticsearch (network layer), or within Elasticsearch will cause the operating Postgres transaction to ABORT. [1]

In my defense, there is a fairly important distinction between "any error that ES is capable of reporting back" and "any sort of failure within Elasticsearch".

That said, I and trust that you're more familiar with consistency levels than me, so I'll bow out here.

[1] https://www.zombodb.com/documentation/things-to-know/

I can definitely reword that if it's confusing.

Reply on news.ycombinator.com