Hacker News Reader

Comment by munk-a | Hacker News Reader

In fact - if you're building a very large distributed system the goal is usually to shrink that centralized component to the smallest and most robust surface you can. If the system is well designed it is amazing just how much consistency power you can get from a tiny component of centralization.

There are always tradeoffs of course, but building a truly decentralized system requires some really difficult compromises to correctness. The two general's problem is a great piece of reading on this topic - distribution always requires compromises in general, but to fully remove an authority on truth gets quite tricky.

View on HN

I think Ducklake[1] is a terrific example of this. They said "look, let's build a lake house over S3, but for the bit that needs strong consistency (the manifest of which S3 blobs are in play), let's use Postgres". Postgres as a metadata catalog or control plane is brilliant for this, since you get strong consistency and the scaling story around a metadata catalog is far different than the volume of data you need to store. Use S3 for volume, Postgres for consistent metadata.

A similar pattern has spilled out of projects like Warpstream[2], which I suspect is using Postgres behind the scenes of their control plane.

[1]: https://ducklake.select

[2]: https://www.warpstream.com/

I have built and maintain a system that uses a very similar system - we register artifacts with UUIDs into S3 in a specifically write-once, never edit, never remove approach and then store those UUIDs in a postgres system. We simply juggle around the connection of other model objects to UUIDs as needed allowing us to achieve safe guarantees without burdening the centralized system with the massive volume (these artifacts are often 50MB+ PDFs). I will mention that I am quite fond of this approach but it's good to be aware that introducing levels of abstraction like this do necessarily widen some fail points on the storage side - if your service uses multiple persistence stores each additional store exposes yet another point where inconsistency could be introduced and/or a message could be lost. Still, fragmenting your data over multiple stores that are particularly well suited for their specialized usages can be huge for performance and cost.

If you use hashes of the content itself for your UUIDs, you'll (a) get deduplication and data consistency checking for free and (b) have basically implemented (a subset of) git that uses S3 backing instead of a local filesystem directory :)

for smallish databases, where uuid is surogate id, i would trust dbos/postgres replication. replication is just too hard to role your own.

tangentially, for dbs with large blobs, lot's of easy tricks when uuids are immutable digests.

syncing, say, two blob stores, A and B, boils down to jaccard metric, as a first order approximation

   |(A ∩ B)| / |(A ∪ B)|

diffing the two digest sets at point in time is second order approximation.

and don't forget logical replication ...

> The two general's problem is a great piece of reading on this topic

It is!

And the solution is to add an extra general on the left side. Let's call him Outus Boxus. The two generals on the left side can communicate in perfect lockstep. Then if you need the general on the right to find out about something, you can send a few workers to tell him or something...

More seriously though, you can have a DS for two reasons: tech or political.

Tech means scaling or reliability. So clients can be serviced by any of the nodes.

Political means different actors don't have a central authority. You can't stick two banks into one db.

This technique doesn't seem to address either aspect.

Reply on news.ycombinator.com