RocketMQ – Open source distributed messaging and streaming data platform | Hacker News Reader

RocketMQ – Open source distributed messaging and streaming data platform

rocketmq.incubator.apache.org

133 pointsmanigandham29 comments

There's quite a lot of conflicting information here.

On the comparison of queues, wrt ordering, it states:

    Ensure strict ordering of messages, have no hot spot problem,and can scale out gracefully

And slates Kafka with:

    Ensure ordering of messages within a partition

But then on the "core concept" page it states:

    Topic is partitioned into one or more sub-topics, “message queues”.

And:

    Consuming messages orderly means messages are consumed the same order they are sent by producers for each message queue. If you are dealing with scenario that global order is mandatory, make sure the topic you use has only one message queue.

Which amounts to exactly the same as Kafka, albeit with "partitions" renamed "message queues".

A Kafka topic is more synonymous with a "message queue" and a partition is a division among hardware for parallelism.

So you can have one topic, partitioned to 5 consumers, and each consumer storing the received events. Each consumer will see their own messages in-order, but if you merge the 5 it is not guaranteed to be ordered.

I'd guess this is what RocketMQ means by "scale out gracefully"

In Kafka, multiple consumers can recieve multiple partitions. The only thing, from a consumer's perspective, that's special about partitions is that it's the level at which message ordering is guaranteed.

That's exactly the same semantics provided by RocketMQ's "message queues", from what I can tell - except perhaps that these message queues are explicitly subscribed to? If that's the case, you can do that in Kafka too, but it's generally discouraged, as it can lead to performance problems.

Hi,

There is nothing about `rename`, RocketMQ has different storage architecture than Kafka. see https://rocketmq.incubator.apache.org/rocketmq/how-to-suppor...

I thought it was a problem to navigate through all AWS services. Then I discovered (surely not) all the Apache projects.

This is by no means a criticism, rather amusement when looking at all the projects within the epic Apache ecosystem.

Apache is lately semi-dumping ground of Java server side/big data types of projects. This rocketmq is developed by Alibaba. Not sure if they are still fully involved or planning to move on to next thing.

The model is dump the core on Apache and try to make money by value added services like management, monitoring, clustering etc if it is created by software vendor or try to defray the maintenance cost if it is done by non-software company.

For many years, Apache was the Boost of Java programmers.

Meaning, where all cool libraries were being developed.

What's the difference between this and activemq (and apollo, if anything came out of that project...)?

Edit: Found something https://rocketmq.incubator.apache.org/docs/motivation/

But is this a re-implementation or something else? Wasn't apollo also a re-implementation?

As far as I can see it looks like there is a possibility that a new version of ActiveMQ will be based on Apache Artemis in the future sometime, as in later on, maybe... Artemis claims to be really fast and solves some of the problems outlined in the RocketMQ motivation page you listed. It uses an append only journal file similar to Kafka and the clustering and failover story looked simple and reliable to me though I haven't tested it.

Whether or not Artemis becomes the new version of ActiveMQ, RedHat have recently released a new version of their AMQ and it is based on Artemis. It supports AMQP 1.0, MQTT, STOMP, OpenWire, and HornetQ Core.

https://developers.redhat.com/products/amq/overview/

I'm still not sure I understand what the consumer queues are adding? Is it to allow non-ordered reading to be fast but also allow ordered reading?

In Java again? I wish such systems would use Rust.

I love Rust, but it still has a long road to walk until it can match the IDEs, debuggers, available set of supported OSes and hardware devices, build speed (cargo always builds from source) and server monitoring tools that the JVM enjoys.

Plus for the majority of distributed applications on the enterprise level, using a programming language with GC support isn't an issue.

Finally, there are plenty of languages available on the JVM, straight Java isn't the only option, just like (luckily) C isn't the only option on UNIX.

This. I am falling more and more in love with Rust but i'm having issues properly using it without proper IDE and debugger support.

It's already a quite different language for someone as me coming from a C89 and Python background, I need as many helpers as I can to ease in eehehe. intellij-rust edits code, but does not debug (clion does as beta, but clion is paid). vscode+rust debugs code, but the whole integration feels fragile. I have yet to try RustDT but this whole time I am trying different toolkits I could be building code....

So yes, Rust guys please make this happen as fast as you can, we are thirsty to dive in!

I'm going to try to get involved in the Rust community, translate docs or work shit they need and that I can help. "x.py" is a joke right now.

> but the whole integration feels fragile.

Could you say more about this?

> "x.py" is a joke right now.

This too.

Java is great for large engineering teams. It also has a great Standard Library.

I get why a startup might not like Java, but for large scale software projects its not a bad choice.

* Its stable

* Highly Supported

* Easy to find developers

* Fast (compared to other popular programming languages)

* Has a great package library

* Old code will just `work` for years if not decades. New java versions are generally non breaking

+= * There are bridges to Java for most popular languages (ex. PySpark uses Py4J to run on top of the Apache Spark Scala version on the JVM)

This is apache project. 99% of those are on Java/JVM. Also if someone wants to use Rust they can always do so, perhaps, outside Apache umbrella.

Why is Apache foundation supposed to be Java-centric?

It is not supposed to be java, but java is dominant language there. As of now, there is not a single Rust project https://projects.apache.org/projects.html?language

A project can become Apache project if it follows their governance rules - they define management, decision process, minimum number of organisations and such.

You are welcome to write one in whatever language you like

How is this different from kafka?

There's a comparison table on the site though it's not super helpful.

https://rocketmq.incubator.apache.org/docs/motivation/

Kafka is usually used in log collection and big data processing, which care less about latency. RocketMQ is quite friendly for latency-sensitive applications. And also with retry and track mechanism， it is used by finance or trading systems, such as Alibaba and other bank systems.

Does it use AMQP?

No, according to [1] it uses JMS or TCP (correction: JMS over TCP?).

[1] https://rocketmq.incubator.apache.org/docs/motivation/#rocke...

JMS is an API (over-simplification, but at it's core it's an API). It doesn't specify a wire protocol

I don't see MQTT support either.

There is a mention of MQTT support in roadmap section as planned feature.

https://rocketmq.incubator.apache.org/docs/roadmap/

Yes, it's our roadmap, and we will support MQTT soon.

Reply on news.ycombinator.com