tylertreat

Author here. I started Liftbridge after working as a core committer on NATS and NATS Streaming (as well as my experience using Kafka in the past). In particular, implementing replication in NATS Streaming and drawing from that experience/lessons learned. Happy to answer any questions!

Yes, for more on that see my reply here: https://news.ycombinator.com/item?id=21946939

It's getting very close. I had wanted to make a production-ready 1.0 release before the end of the year, but we're in the process of switching from protobuf to flatbuffers. Once that is complete, a stable release will be made.

ZeroMQ is not a message queue, it's a networking library.

Check out Liftbridge (https://liftbridge.io) as a way to add these capabilities to NATS.

Also check out Liftbridge (https://liftbridge.io), which is a Kafka-like API on top of NATS.

Disclaimer: I'm the author and former core contributor of NATS and NATS Streaming.

This is correct. Also, I'm planning to introduce a mode that runs NATS "embedded" within the Liftbridge server, such that NATS basically becomes an implementation detail of Liftbridge.

Re: consumer-offset checkpointing, this will be implemented in the form of consumer groups, similar to how Kafka works where a consumer group checkpoints its position in the log by writing it to the log itself.

It has at-least-once delivery guarantees currently. I have plans for some lightweight transactional-style publishing with an idempotent producer though.

Just to clarify, Liftbridge relies on a single Raft group used purely for metadata replication, i.e. control plane, not data plane. Streams themselves are replicated using a protocol very similar to Kafka (ISR-based with followers fetching messages from the leader's log).

Basically the same types of uses cases as Kafka and similar systems: event sourcing, CQRS, stream processing (e.g. processing click stream events), data pipelines (e.g. log or metric aggregation used to feed multiple backends), or as a replicated transaction commit log (e.g. use it to create materialized views, populate caches or indexes, etc.). Log compaction means it can do this efficiently by just retaining the current state and discarding older transactions.

https://kafka.apache.org/uses

Author here. Happy to answer any questions.

My experience with graph DBs: schemas usually become implicit and ad hoc, minimal tooling support and even less mature tooling, difficult to explore and understand your data, performance is often poor, memory usage can be problematic because they usually pull the graph into RAM, and there are few people with real experience (comparatively to traditional DBs) which can create a negative feedback loop.

I have yet to run into a use case where a graph provided more value than a relational model. I'm sure they exist, but I haven't found them yet.

There are several things wrong with this IMO.

First, branching/PRs are not automatically process-heavy. If it is for your company, there's probably something else wrong.

Second, this process—if done right—dramatically INCREASES code quality and DECREASES defects. This can be backed up with data. Code reviews, if they're not rubber stamps, are extremely valuable.

Third, the trunk-based model described in the article often leads to big-bang release branches. Master should be gold. This facilitates rapid and small releases, which directly impacts the business. This can also be backed up by data (e.g. see the Accelerate book).

Fourth, this model (again if done right), plays an instrumental role in compliance. The article doesn't even mention compliance, which is a HUGE thing bogging down a lot of companies trying to adopt DevOps practices right now. It has nothing to do with discipline or trust.

Check out OKRs for tying back longer-term planning into shorter-term deliverables: https://blog.realkinetic.com/okr-process-489891e6b6a8

To be fair, part one of the series explains this:

> we will focus on what it takes to build something like this using Kafka and NATS Streaming as case studies of sorts—Kafka because of its ubiquity, NATS Streaming because it’s something with which I have personal experience.

Performance mainly, I think. Benchmarking against UUID v4 shows a ~175% speedup.

Curious how this compares to gnatsd: https://github.com/nats-io/gnatsd

A latency/throughput benchmark comparing the two is effectively comparing the latency/throughput of a socket from A to B. The same benchmark for NATS would be a socket from A to NATS and NATS to B. Obviously the latency of going over the wire once is going to be lower than going over the wire twice.

A proper comparison would need to include the forwarding device for ZeroMQ.

> * clustering - what kinds of clusters are supported? How do they scale? Do they favor availability or consistency?

There is no persistence so consistency is irrelevant. NATS does everything in the name of availability to the point of dropping connections for slow consumers.

> * security - is there any authentication? authorization for certain resources? TLS?

TLS is planned, not sure on timeline. It sounds like Apcera has plans for some form of authentication and multitenancy as well.

> * guarantuees for delivery? Transactional modes?

At-most-once delivery. No transactions. There may be some form of more reliable delivery coming in the future, but I'm not sure of the details.

> * what does the wire protocol look like? Support for meta data?

Very simple text-based protocol. I don't think it's formally documented anywhere currently though.

Well, for one, there is no streaming socket API, just HTTP. I suspect the 5 reads/sec limit on shards is mainly due to multitenancy, but I could see this increasing in the future. Amazon has already improved the latency of Kinesis a fair amount, but it still has a ways to go before it matches Kafka.

We also have cases where we need many consumers reading a stream, so having that effectively limited to 5 concurrent consumers is a pretty tough limitation. Their solution to this is to just have consumers sleep on a failed poll, but that doesn't really help if you want to scale out your ingestion. You don't have that problem with Kafka.

In my experience, Kinesis' performance makes it a nonstarter for low-latency systems and use cases with many consumers. Shards are limited to 5 reads/sec, so consumers are heavily throttled. Producers also have significant put latency. Kafka's latency and throughput are OOM better but of course come with the operational overhead. Depends on what your needs are, but Kafka is a better choice for "real-time" systems.

Not a fair comparison. ZeroMQ is a socket library. NATS is a message broker.

NATS is at-most-once delivery, which is partly why it's so fast. Sounds like there are plans to introduce stronger guarantees in the future though.

There seems to be a bit of confusion over just what exactly I'm trying to get at here. I'll be the first to admit, the article might not do it justice. This is a comment I posted on it which hopefully helps clarify:

The point is there are certain realities inherent in distributed systems which can’t be papered over, and those realities often manifest themselves at all levels in the stack. Sure, we can try to build abstractions which hide those problems, but they often leak or simply eschew the problems in dangerously subtle ways.

Frontend and UX folks need to understand what they are building on. They need to understand the potential pitfalls of these abstractions and why they are as such. Sometimes there is a good business case for apologizing, sometimes there’s not. These things are never black and white, but if everyone can understand how systems work, we can build better ones and make better choices.

I think the main point is that Elasticsearch should not be used as a primary store. It seems obvious (or maybe not?), but some people do use it as such. Just like Redis should only be used as a cache.

Consensus and atomic broadcast. Paxos, raft, zab. Formal methods.

"As many Python insiders know well, the Castro brothers, Fidel and Raúl, have long been fans of FLOSS and of Python in particular."

Good one :)

You cannot have it if the network is unreliable and systems fail. The network is unreliable and systems fail.

The point is that temporary network partitions happen and you can't guarantee delivery without expensive coordination. Sure, exactly-once is possible in environments where network partitions don't occur. Unfortunately, we don't live in that world.

There's a hidden message in there somewhere. Just keep looking.