Hacker News Reader

Comment by mrkeen | Hacker News Reader

I jumped onto https://mqtt.org/ to try to answer my usual use-case question about non-Kafka messaging, which is: "Do the messages get saved anywhere so you can come back and read them later?" Still not entirely sure about it.

But I did see:

    This is why MQTT has 3 defined quality of service levels: 0 - at most once, 1- at least once, 2 - exactly once

I'm a big fan of advertising the impossible on the front page.

View on HN

MQTT isn’t designed as a persistent log, but can fulfil some of what you might want to use one for.

Each message has a couple of flags, the first being Quality of Service, which as you quoted above determines deliver guarantees. 0 is fire and forget, with potential loss of messages. 1 will queue messages for delivery to offline clients that are subscribed to a topic (within reason, all brokers set limits on that), and 2 is often described as “exactly once”, but is in fact just a more involved dance to acknowledge messages.

The other flag is a Retain flag, which instructs the broker to associate that message with the topic it was sent to, and send it on to any newly subscribing clients when they subscribe. This is good for use cases like remote device configuration - you can send it to a topic, setting the retain flag, and then when a device comes online it’ll immediately receive new configuration.

MQTT is great as a message queue for remote devices, mostly because it’s so lightweight anything with an IP stack can integrate with it, but I’m not sure why anyone would attempt to make it a piece of core infrastructure.

> advertising the impossible

eh if you read the finer print it's just a deduplication id appended to every message. blog doesn't go into detail on what happen when two client pust a message with the same it, or what happens if there is more than one failure (i.e. client fails to detect a service outage and during the service outage the message is consumed by the broker but persisting fails) but in general the usage of a at least once + a deduplication id is not something revolutionary.

With mqtt it depends on the broker, eg. Emitter.io can save them for a week etc. Offset for a client is usually stored on the broker, so if a client reconnects, all of the messages it has missed are forwarded to it.

As mentioned in other answers, service levels have a defined meaning, which is different to the absolute theoretical meaning, and really is to do with the message aks.

Really, as the article mentions, kafka and mqtt are for different purposes, with some overlap. Kafka is all about the log, whereas mqtt is about uncertain connections. A better comparison which I've yet to see, is comparing mqtt to nats.

Lastly, kafka is much easier to administer using redpanda, which doesn't have zookeeper, combines the registry and kafka connect (see WASM runners) with the runtime, and has a very nice console for debugging.

Similarly, Emitter.io does a great job with clustering for mqtt.

I'd like to see an open source kafka-mqtt bridge that worked in both directions as they all seem to go mqtt->kafka only.

nats +1

> I'm a big fan of advertising the impossible on the front page.

Do you mean like Confluent do? https://www.confluent.io/blog/exactly-once-semantics-are-pos...

The impossibility of "exactly once" is a theorem, not an opinion.

Knowing that, the article you linked is funny. It begins with maybe a thousand filler words complaining that all of this is "poorly understood", and it's not "impossible", just "very hard". Then it gets to the meat: Yeah well, you know, it's not quite "exactly once delivery", just "exactly once semantics", and to achieve that, messages need to be idempotent so that duplicates don't matter.

We all know that. It's called "at least once".

> The impossibility of "exactly once" is a theorem, not an opinion.

It's quite likely that your definition of what "exactly once" means differs from the one followed by MQTT. As this issue was documented years ago, I doubt this is a relevant argument to have, unless we want to feel smart by criticising others.

https://www.eejournal.com/2015/05/28/is-exactly-once-deliver...

That's the Kafka Streams API. "Exactly-once semantics" has a very specific meaning in the context of that particular API, which the article could probably do a better job of clarifying upfront. (Otherwise it is an excellent overview of the problem and solution provided by the Streams API.)

MQTT.org can't answer that as it's a web page for for a protocol. I've worked on platforms that do have a historian feature but it will vary from broker to broker.

(disclosure, I work on Eclipse Amlen and it does not - but people often rig it up to a subscriber that funnels (some/all) messages into databases

Reply on news.ycombinator.com