Hacker News Reader

Comment by bsaul | Hacker News Reader

Sidenote question :

Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack.

It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit too high. Is that a valid concern ?

Also, i've seen that kafka is working on removing the dependency to zookeeper, is pulsar going to take the same road ?

View on HN

This sound about right. Apart from maybe original Apache HTTP server most of the Apache projects are in Java.

Looking at codebase of Pulsar it looks like typical Apache style sprawling Java project with more than thousand directories, many thousand files and more than hundred dependencies. As comparison NATS which is in Go has few hundred files, less than hundred directories and about a dozen or so dependencies.

NATS is an amazing project, I just wanted to take the opportunity to highlight it for those first hearing about it in this comment. It's so brilliantly simple, yet changed the way I design distributed systems. I handle almost anything in regards to the standard messaging guarantees that a Kafka-like system offers at the endpoints now. As a result, systems are much simpler, and diagnosability of bugs or edge cases are much more straightforward.

NATS is amazing but note that it makes different promises than Pulsar. NATS doesn't offer true durability (in exchange for amazing performance and great simplicity) whereas Pulsar and similar are meant to survive certain partition or failure situations and not lose data.

It's not one or the other, they're just different tools.

There is nats-streaming-server as well which offers true durability (via file or SQL store) and a streaming model very similar to Kafka and Pulsar. It can also run as a raft cluster or in fault tolerance mode. It still has very good performance and is very simple to deploy and operate (I use it for event sourcing for real time IoT data at my day job).

NATS Streaming has major scalability problems even if it's simple to deploy. It's only high-availability unless you the Raft clustering but that has been bolted on to the original project and isn't really well-designed.

The team is working on an entirely new system called Jetstream to eventually replace it.

This sounds interesting, what exactly do you mean by 'endpoint' in this scenario? I looked into a few alternatives before settling for pulsar, and disregarded nats because it didn't seem to support message persistence. I didn't look into it too deeply though, maybe i should have. How do you guarantee no message is lost with NATS?

In my thinking, I think of an endpoint as something at either end of the communication channel (NATS in this case) where it is effectively terminal. Usually this is where the application logic lies. Dereck Collison (creator of NATS) brings this up in many of his talks about NATS, but I think the source of his thinking might come from “End-to-End Arguments in System Design” by Saltzer, Reed, & Clark.

The core of it is this point:

"Functions placed at low levels of a system may be redundant or of little value when compared with the cost of providing them at that low level."

That is, in order get that message redundancy or exactly once delivery, or message persistence, you pay a high cost, and you may be better off delegating to the endpoints.

This blog provides a good overview

https://blog.acolyer.org/2014/11/14/end-to-end-arguments-in-...

Here is the original paper

http://web.mit.edu/Saltzer/www/publications/endtoend/endtoen...

Thanks, much appreciated!

Did you check this?

https://docs.nats.io/nats-streaming-concepts/intro

"..Message/event persistence - NATS Streaming offers configurable message persistence: in-memory, flat files or database. The storage subsystem uses a public interface that allows contributors to develop their own custom implementations."

and

"At-least-once-delivery - NATS Streaming offers message acknowledgements between publisher and server (for publish operations) and between subscriber and server (to confirm message delivery). Messages are persisted by the server in memory or secondary storage (or other external storage) and will be redelivered to eligible subscribing clients as needed."

Also check out Liftbridge (https://liftbridge.io), which is a Kafka-like API on top of NATS.

Disclaimer: I'm the author and former core contributor of NATS and NATS Streaming.

I looked at Liftbridge when choosing a streaming platform for event sourcing, but the FAQ says it's not production ready. Is that still accurate?

Yes, for more on that see my reply here: https://news.ycombinator.com/item?id=21946939

No, i missed that. I think ive seen 'nats streaming', but didn't realize that it is its own distinct thing. All this makes more sense now to me, thanks!

Outbox pattern and NATS streaming

Additionally, you could always have reliable delivery over NATS using the request/response pattern using acking and retries.

Is it me or does NATS looks like it's aimed at an actor-based style of distributed system ?

It not necessarily aiming at that.

Its more than one project in the repo.

It's an interesting observation.

I think that the modern approach to distributed systems is moving towards golang style microservices and lightweight / simple system design with RPC communication, reconcile type loops for state reconciliation, and backing CP databases. I think this is the influence of k8s (and maybe google's approach to distributed systems).

I will almost certainly get downvoted for this (as I always seem to when I criticize the JVM), but Apache/JVM style architecture feels REALLY long in the tooth to me. I think you are committing to an outdated and very expensive approach to building software if you use anything running on the JVM, especially Apache based anything. Cassandra is a great example of this - out of the box it's a terribly performing database that is extremely expensive to run and tune. Throw enough resources and time at it and you can get it to acceptable scalability - but running on the JVM which is a huge memory hog will always make it expensive to run (and even then, you will always get terrible latency distributions with the JVM's awful GC).

If I was building a business I would run far far away from any JVM based solution. The only thing it has going for it is momentum. If you need to hire 100s of engineers off the street for a large project, then a JVM based stack is about your only option unfortunately.

> the JVM's awful GC

This just makes it seem like you are trolling. JVM devs have done more to advance state of art in this area than any other language. The problem is that most JVM apps just produce too much garbage, not necessarily that the algo itself is awful.

Either way, there's no such thing as an optimal GC algorithm, just different trade-offs depending on your use case. Not everyone cares about latency.

> The problem is that most JVM apps just produce too much garbage,

This is strange way of saying that Java the language and most frameworks around it force apps to generate this much garbage.

No, it's a way of saying that Java != the JVM. The JVM's GC is not "awful" precisely because so many Java frameworks are awful.

I stand by that - I really think the JVM has a bad GC, and it has cost businesses billions probably. There is no trade off here - the JVM & Java have incredibly high memory requirements due to poor design - and this leads to all kinds of issues (like the bad GC latencies). No other language has this kind of problem as badly as the JVM does, even other GC languages. And we shouldn't accept this anymore - look at how much better golang's GC is for real world usage.

I understand your architecture criticism, and think it has merit, but I'm not sure why Apache gets dragged into that. Apache Airflow is in Python. Apache Arrow is in C. CouchDB is Erlang.

There's a ton of projects Apache Foundation hosts that fit your description but it's a mistake, I think, to confuse individual projects with Apache in general. Bad enough that people confuse the license with the foundation.

Sorry, you are right. It would probably be more accurate to use Spring actually. I was thinking of Cassandra/Zookeeper specifically when I said Apache.

For what its worth I have a java microservice running on 13mb of ram.

Without knowing what that service does, how many users/request it serves per second/day, I can only assume it just an http listener up on some port that returns "Hello {username}" when someone sends a GET request.

Nah, its full Java EE. Its fully possible to run low profile java.

https://quarkus.io/

Though criticizing the JVM is fair game, this sounds too absolutist. I like and dislike some things about C++, Java, Scala, Python, Bash, SQL, etc. I bet you do too, no?

Nowadays Java based micro-services can compile to native thanks to Graal native-image and frameworks like: https://quarkus.io/.

I can't wait for projects to ditch ZooKeeper. Apache Bookkeeper, which Apache Pulsar uses for its state, already supports Etcd as a consensus store (though I believe this is still alpha? beta? quality). Pulsar is also working on supporting Etcd.

And I believe Kafka has a KIP in progress to remove their reliance on Zookeeper altogether but having the brokers communicate with each other directly.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A...

It seems to be the trend. There would be more and more equivalents in Go in a foreseeable future. There would be a split.

I generally don't use Java or Go in my applications, but Go components are usually more lightweight and easier to use if you're not majorly working with those runtimes. JVM has a big overhead, and Java applications usually make things worse if they use things like dependency injection.

It depends. We have a ton of Java apps running atop of kubernetes. All of them use zk, but every team operates their own mini zk cluster deployed on k8s. It’s worked fine except for certain hard to debug problems that hit a few teams occasionally.

I guess my point is that k8s let’s you shift the operational burden to dev teams if they need it. If you have a centralized operations team running a giant, common zk/ etcd, yeah this would be additional operational burden.

> is pulsar going to take the same road ?

Yes, it's in the works

It doesn't look like it's going to be ready anytime soon though.

Just to confirm, Pulsar has on it's roadmap to remove it's dependency on Zookeeper. Is that correct?

That's correct, we're moving to have a pluggable metadata store and coordination service.

Note that this is a different goal from the KIP referenced above, which is to entirely remove any dependency on an external configuration service. The idea of “pluggable consensus” is explicitly rejected in this KIP.

Zookeeper sux. If you are in the Java world you can often roll something better out using Hazelcast.

Reply on news.ycombinator.com