Liftbridge: Lightweight, fault-tolerant message streams

We'd got tired of trying to hack around SQS. After a boat load of testing, we settled on NATS. It was our big "innovation token" for that project. It has paid off wonderfully.

The server clusters very well, and is very simple to peer. We've joined two clusters across two regions, and we've not had any issues.

If you need a fast, low latentcy, non persistant, non guarenteed message delevery system NATS is for you.

If you like to store messages in your queues for later consumption, or need _every_ message, NATS isn't for you.

I have an app in production that publishes real-time events via NATS at high volume, and it has never failed. You do have to understand its design -- NATS can drop messages, and your application has to deal with this.

One lesser-known feature of NATS is that it's embeddable -- if you're writing a Go app, you can use it as a library. You end up with a single statically linked binary that can do everything out of the box.

Liftbridge looks like a neat way to add a layer of statefulness to NATS. I have an app that will need a log abstraction, and have been hoping to avoid integrating any oversized guns like Kafka (whose dependency on ZooKeeper makes it operationally more complex) and Pulsar for what is a fairly simple system.

Thank you very much for your attention!

"NATS Streaming provides a similar log-based messaging solution. However, it is an entirely separate protocol built on top of NATS. NATS is an implementation detail—the transport—for NATS Streaming. This means the two systems have separate messaging namespaces—messages published to NATS are not accessible from NATS Streaming and vice versa. Of course, it’s a bit more nuanced than this because, in reality, NATS Streaming is using NATS subjects underneath; technically messages can be accessed, but they are serialized protobufs. These nuances often get confounded by first–time users as it’s not always clear that NATS and NATS Streaming are completely separate systems. NATS Streaming also does not support wildcard subscriptions, which sometimes surprises users since it’s a major feature of NATS.

As a result, Liftbridge was built to augment NATS with durability rather than providing a completely separate system. To be clear, it’s still a separate server, but it merely acts as a write-ahead log for NATS subjects. NATS Streaming provides a broader set of features such as durable subscriptions, queue groups, pluggable storage backends, and multiple fault-tolerance modes. Liftbridge aims to have a relatively small API surface area.

The key features that differentiate Liftbridge are the shared message namespace, wildcards, log compaction, and horizontal scalability. NATS Streaming replicates channels to the entire cluster through a single Raft group, so adding servers does not help with scalability and actually creates a head-of-line bottleneck since everything is replicated through a single consensus group (n.b. NATS Streaming does have a partitioning mechanism, but it cannot be used in conjunction with clustering). Liftbridge allows replicating to a subset of the cluster, and each stream is replicated independently in parallel. This allows the cluster to scale horizontally and partition workloads more easily within a single, multi-tenant cluster."

[1] https://bravenewgeek.com/introducing-liftbridge-lightweight-...

NATS Streaming (aka STAN) was an answer to persistence by basically building acks and storage on top of the base NATS protocol, resulting in another protocol on top that are stored as files or backed by a database. STAN has limitations on topic/subs and is designed as single master with multiple failovers but works well if you need a simpler self-contained alternative to Kafka. Later a Raft-based clustering option was added but isn't a great architecture and has a major performance impact. The docs recommend you use the failover model instead.

Liftbridge offers a better version that uses the standard NATS protocol and basically functions as invisible subscribers to the topics and just logs all the messages using multiple Raft groups, which lets you scale out by using more topics. NATS itself has recently been revamped with 2.0 release that adds some major new features and they're working on a better answer to the persisted/distributed log offering.

If you need fast pub/sub for ephemeral messaging then NATS is great. If you need persistence and a single-server is enough throughput then use NATS Streaming (with failover if you need it). If you need more persistence throughput then use Liftbridge. Although once you start getting to these scales I would recommend looking at Kafka (which also has gotten much better with v2.0+) or something like Apache Pulsar which is more advanced and scalable then both. There are also commercial options like Solace with a free tier that are worth checking out.

If you require fanout to your stream consumers then this is supported simply by having multiple consumers subscribe to each stream. If you require a competing consumers model then this isn't directly supported, but you can have load-balance groups which you could use to achieve something similar, but with a static configuration - that is, your consumer population can't resize dynamically, but you can distribute messages to a fixed set of streams, each of which has a consumer.

liftbridge doesn't yet support durable consumer-offset checkpointing, although this is advertised as being on the roadmap. In the meantime, if you wanted the consumer offset to survive consumer restarts, then the consumer itself would need to take care of persisting its current offset.

NATS on its own won't cut it, since it has no persistence and is entirely real-time (new clients receive messages published after they connect, not old messages).

I've not used Liftbridge yet, but plan to. While NATS Streaming is solid, its distributed design seems a bit ramshackle, relying on a combination of Raft and a shared state (a distributed file system or an SQL database) for failover. Liftbridge seems like a better design to me.