Databases = Frameworks for Distributed Systems | Hacker News Reader

Databases = Frameworks for Distributed Systems

laplab.me

91 pointslaplab20 comments

I was watching a video by one of Amazon’s distinguished engineers a while ago and for the life of me I can’t find it now but the thing that stuck with me from it is,

“There are only three modes for a distributed system - it implements paxos/raft itself, it relies on a data store that implements paxos/raft or it’s wrong.”

Obviously a bit of an over simplification since there have been some advancements in that area but not far off.

This article introduces an approach that falls under the second mode and I think it’s a good way to build a distributed system without having to re invent the wheel especially if you can just rely on the data store for the consensus and make your brokers and nodes relatively dumb.

Look at k8s and etcd for a very widely used example of this approach as well.

I’m also currently building a distributed search engine using etcd for the service registry, broker peer announcements, and worker queue and it’s been a good experience so far.

Well in some situations you can also get away with CRDTs or Gossip (SWIM), but yeah in the general sense Paxos (Raft is a Paxos-family algo). There are lots of paxos family algos to choose from[0].

> I’m also currently building a distributed search engine using etcd for the service registry, broker peer announcements, and worker queue and it’s been a good experience so far.

As a random stranger on the internet, please build an abstraction layer around etcd. Even if there's only one implementation, I've found that so many distributed projects that just never reach the scale that etcd is built for would benefit from the option of writing their own drivers (you don't need a full plugin system just a regular abstraction layer).

One example is Kine[0] for k8s -- if k8s had a built in option for writing/reading from something like Postgres from the beginning it would have been a better project for it, IMO.

If you do build a plugin system though, you can pass off the work of maintaining the other implementations!

[0]: https://vadosware.io/post/paxosmon-gotta-concensus-them-all

[1]: https://github.com/k3s-io/kine

I think that's a great idea, I already have the beginnings of a plugin system since local non distributed is a mode you can run in and I had to abstract etcd away to do that. Will add it to the list!

Thanks! Also I’m pretty curious where I can find your project…

Hey at the moment it’s not public but I’m getting to the point where it will be. Happy to ping you when it is, also planning to do a show HN post here as well with it.

Please do! email in bio and also I'll try to look out for the Show HN!

> “There are only three modes for a distributed system - it implements paxos/raft itself, it relies on a data store that implements paxos/raft or it’s wrong.”

That sounds like something Al Vermeulen would say. Though my recollection of the quote is more along the lines of:

"There are three types of consistent distributed systems: paxos, broken protocols, and single points of failures."

I'm guessing the talk it came from is Under The Covers: Core Distributed Systems Primitives that Power Our Platform[0]. That presentation is full of great insight into how AWS thinks about distributed systems. The other presenter, Swami, is the VP of AWS Databases now but started off getting Amazon's initial Paxos implementation going. I was lucky enough to work on those core building blocks and it was an incredible experience.

0 - https://www.youtube.com/watch?v=QVvFVwyElLY

Yup that's the one! Thank you for digging it up.

That work must have been really gratifying especially since it underpins so much now.

We've been designing TigerBeetle [1], a distributed financial accounting database, from the beginning with the ultimate goal that the whole architecture of:

  * global consensus layer (Viewstamped Replication) [2], and the
  * local storage engine (a high-performance storage fault-tolerant LSM-tree over io_uring) [3],
  * which can all be deterministically simulated and fuzzed,

can eventually be used as a distributed library that gives you not only consensus but storage, all in one, kind of like “a DistSys Iron Man suit”—you put in your own state machine and business logic [4] inside it, and get all the replication and fault-tolerance around that, out of the box.

TigerBeetle is based on some of the latest research around storage faults, how to recover safely and maximize high availability, and we've been heavily inspired by the deterministic simulation testing of FoundationDB [5].

[1] https://github.com/coilhq/tigerbeetle

[2] https://github.com/coilhq/tigerbeetle/blob/main/src/vsr/repl...

[3] https://www.hytradboi.com/2022/tigerbeetles-lsm-forest

[4] https://github.com/coilhq/tigerbeetle/blob/main/src/state_ma...

[5] https://www.youtube.com/watch?v=OJb8A6h9jQQ — “How I Learned to Stop Worrying and Trust the Database”

While databases having been converging on composability for a long time, the model presented here is not the correct one and causes problems for many important and emerging use cases.

As an elementary observation, isolating the designs of storage, compute, and networking as separate modules is a reliable way to achieve poor performance and resource efficiency. Some of the most important scalability and performance optimizations in modern database architecture are schedule-driven, which requires tight coupling across the design of storage, compute, and networking. Ignoring this class of optimizations produces very large reductions in real-world workload performance, and feeds a narrative that the software is wasteful and not environmentally friendly.

Heterogeneous and specialized resource collections with complex topologies are becoming more common for a diverse set of reasons. This breaks most horizontal resource abstractions. The practice of treating every thread as a complete monolithic database kernel connected together with protocols that are good at equilibrium finding lends itself to this environment very well, and the edge (for which database technology is broadly broken), in addition to being ideal for the boring data center case where the topology is highly regular and every node has uniform resources.

Most of our distributed systems are modeled as giant distributed file systems, and that fits okay with the idea of horizontally partitioning compute, storage, and network. This is easy for developers to reason about because they are familiar with file systems. However, it is difficult to express important database-y operations in this model. For example, if you want to do ad hoc joins in a scale-out environment, you need mechanics like decentralized parallel orchestration which isn't really a concept in a "distributed file system" model of "distributed".

Databases have been slowly converging on latency-hiding HPC software architectures (which has no implication of HPC hardware), which look very different than a distributed file system, can express more powerful capabilities, and are generally more scalable. To the extent there is a general principle and framework behind databases, it is more this than traditional distributed systems.

Some of academia’s preeminent database researchers have made a similar hypothesis: DBOS: A DBMS-oriented Operating System, https://vldb.org/pvldb/vol15/p21-skiadopoulos.pdf and https://dbos-project.github.io/

I would say the author just tried to redefined "distributed", and ignored those real distributed systems, e.g. Paxos/Raft-based ones.

You are thinking of fully decentralised systems.

A system where there is a centralised leader is still a distributed system so long as at least some of the processing takes place on other nodes.

As per Wikipedia, a distributed system is: "a system whose components are located on different networked computers, which communicate and coordinate their actions by passing messages to one another from any system."

You're describing consensus algorithms.

A few years back, a colleague and me built a distributed system toolkit in an effort to understand the blocks/pieces of a distributed system and also to make it easier to build one for our needs. This article seems to hit on the same set of notes that we had in mind when we started out building suuchi.

https://github.com/ashwanthkumar/suuchi.

At My first programming job, as an intern we went with his approach, using rethinkDB amd python “microservices” that were all separate python processes reading from their own rethinkDB change feed lambda - the system was essentially built as a distributed reactive microservice pipeline, with rethinkDB as the source of truth for every other process - which existed on 100-1000 machines depending!

This article was originally posted on dev.to, but it turns out that HackerNews banned this website, so I decided to create my own.

HN "blocks" dev.to ? Why ?

Dang answered that here: https://news.ycombinator.com/item?id=27110093

Interesting. It never occurred to me that some domains may stuff a lot of spam/adverts in between good articles.

Sadly, it does happen.

This post is on to an important dynamic.

We've scaled databases then broken them up and our applications are distributed databases now. The tools mentioned seem a bit random but... There does seem to be space for database tooling that melds into our applications but everyone wants to bring you into their bubble instead.

Reply on news.ycombinator.com