MQTT: A Conceptual Deep-Dive
ably.io
ably.io
Also, the spec is actually pretty readable: http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-o...
Note: I'm not affiliated with HiveMQ.
Each message in MQTT carries a 16bit message ID. There protocol allows for retries of message delivers for QoS1/2, but in revised specifications limits to allow retries only if the session got disconnected.
Now this opens the question what a client should actually do when it does not get a ACK in a certain timeframe? Since retries are not allowed, an alternative is to close the connection. That allows for resending this message, but opens up other problems. First of all reconnecting when only a single message timed out introduces a huge overhead, which is not what we want for small connected devices.
In addition to that there exists an issue with the persistent session feature, which requires the client and server to track all message IDs across connection attempts. This has the implication that if a client does not get an ACK for a certain message ID it can never reuse the ID - even not with reconnects (assuming clean_session=false).
The tiny 16bit message ID space also requires clients to remember sent messages and prevents it from easily dropping/cancelling pending transmissions if they are outdated or superseded. The server might still respond to those IDs, and if we send a new message with the same ID there exists ambiguity. So technically the client would need to track all used message IDs until they are ACKd. This might be never, and is not reasonable for a small IoT device. Common clients just ignore the problem and reuse message IDs whenever convenient, which means they are actually not reliable. A bigger address space would have made the problem going away by being able to utilize unique IDs. But with the current spec it seems hard to really provide reliability on non-ambiguity utilizing the MQTT QoS guarantees.
So thereby the conclusion was that there is no way to provide real reliability with the defined QoS classes on MQTT, since that would not allow for ambiguity.
From my point of view the best way to add reliability on top of MQTT is to add a custom reliability layer on the application layer and just use QoS0 transmissions. Those work somewhat better.
Depending on the application a different protocol (fully custom, HTTP long polling with a persistent connection, Thrift, grpc, etc) might also be a reasonable choice.
Before you jump down my throat, I've used all three protocols to a fair degree. MQTT was the most painful, and without strong justification, what's the point? I always read a bandwidth usage justification, but if that's the case, someone should be able to tell me the number of overhead bytes saved using MQTT over WebSockets.
I believe MQTT implementations tend to be smaller, which is why it's popular for IoT devices. Both HTTP and WebSockets are much more diverse specs, so a generally available implementation might have a lot of features the device never uses but still occupy disk and memory (multiple kinds of headers and their semantics, data encodings, etc).
All of that is very useful in my use case (networking and IoT equipment in the middle of nowhere in rough conditions that can go down a fair amount).
The alternative to MQTT wouldn't be HTTP or WebSockets, but actual sockets.
MQTT is simple to implements and doesn't require much performance. Many of its implementations don't support its QoS feature that retries because the embedded processor doesn't have enough memory.
A HTTP client can be implemented by hand with just a few examples in a few hours. I was unable to do the same with MQTT.
MQTT is always on and it's bidirectional. The client can both listen to the server and talk to it. You sure can do that using some kind of push/pull over HTTP but that's going to be costly. You can sure do HTTP long polling, but that's most probably contains code smell.
MQTT can be used simply to broadcast something, which is quite useful in the IoT world. You got a data that has changed, another client can easily get it and do something with it (either log it, show it, or change its behaviour based on it). Again something that you'll have to implements over HTTP.
> A HTTP client can be implemented by hand with just a few examples in a few hours.
If an implementation for it already exists (and most likely than not it does) you are better of with using it than rewriting it.
Sure you can't do it in hours, but MQTT is simple enough that there's already an implementation for you somewhere. Though seeing this source code, I'm pretty sure it can be pulled off in hours still [1].
[1] https://github.com/knolleary/pubsubclient/blob/master/src/Pu...
For a persistent connection, you might want to move from HTTP to WebSockets, which also brings the bidirectional quality and some more of the pub/sub qualities. Though WebSockets' complexity compared to just HTTP is similar to MQTT.
Being able to implement the protocol yourself is less something I would do in production and more a comment on how easy a protocol might be to grok.
I wouldn't expect to implement my own WebSockets client. So this and the relative overlap of features puts MQTT and WebSockets in the same bucket for me (far more so than MQTT and a raw socket).
Summarizing my views: MQTT seems as opaque as WebSockets without the benefits of being built on a very common protocol (HTTP) and being used in industries beyond just IoT. The main benefits proponents of MQTT argue for (low bandwidth, small libraries) don't seem particularly true in comparison to HTTP and WebSockets.
For use case beyond IoT, sure use everything else, but MQTT is great specifically for IoT. It's like saying x86 is better than others 32 bits architectures used by microcontroller because it's used in industries beyond embedded. What is used in embedded make sense for embedded only. If your needs are beyond that, go beyond.
> I wouldn't expect to implement my own WebSockets client. So this and the relative overlap of features puts MQTT and WebSockets in the same bucket for me (far more so than MQTT and a raw socket).
I wouldn't consider even implementing a WebSockets client. I would consider porting the 600 lines of that library I linked though. Would you consider porting that librarie instead [1].
> don't seem particularly true in comparison to HTTP and WebSockets.
I use MQTT over my ESP8266 devices. That thing has 96 KB of RAM and 4 MB of flash memory. Sure Websocket can works on it, sure you can write an HTTP server on it, but in comparison, the library I linked is 600 lines of code, is simple to use and does everything that is needed of IoT really simply.
Even if you were to try to replicate that in WebSocket, you would still have to add a way to broadcast that information easily instead of simply having another client that subscribes to the same server and listen to the topic you publish to.
[1] https://github.com/Links2004/arduinoWebSockets/tree/master/s...
Certainly, with that attitude. Personally, I fight that mentality when I can. The reason to lean on technology that is used in multiple disciplines is to increase the pool of viable programmers. If it's the wrong tool, then it's the wrong tool. But otherwise use the tool that the most can use.
On porting libraries...well, the WebSockets client I play with is only 60 lines. So, I don't know, you tell me: https://github.com/danni/uwebsockets/blob/esp8266/uwebsocket...
And yeah, that's micropython (because I love inviting the wrath of embedded developers). But that will run on your ESP8266 just fine. I use the D1 mini :)
You're definitely right on having to implement more of the pub/sub on your WebSockets server. I misspoke in saying WebSockets gives you those qualities.
Which is my point, the right tool in that case is one that fit on the memory footprint of an embedded microcontroler.
> On porting libraries...well, the WebSockets client I play with is only 60 lines.
Theses 60 lines does almost nothing. You ignore the 240 lines inside protocol.py just beside. You also ignore the 600 KB binary required for MicroPython. Already you simply CAN'T run on an ESP8266 that's only embedded with 512 KB (so can't be run on any of my ESP-01 module).
That does make me think though that the Websocket protocol is quite simpler than I though, but if I were to use it, I would plainly do a socket connection instead which would be even more simple.
> I use the D1 mini :)
Sure with one big enough, but we are talking about embedded, where resources are scarce. Running Micropython is a luxury for many embedded developers.
Micropython and this WebSocket client DO run on the ESP8266. As noted before, I'm using these libraries on a D1 mini. I don't know the specifics of the memory footprint as I stay away from applications where I'm up against the wall of my device's capabilities. But the point stands that there's plenty enough space on the ESP8266 for the glorious luxury of micropython.
In this Dzone article we explain a bit more about adaptive throttling: https://dzone.com/articles/mqtt-throttling-data
> 3.3.2.1 Topic Name
> The Topic Name identifies the information channel to which Payload data is published.
Isn't "at most once" a misnomer ? By definition if it is at most once it is not duplicated.
Exactly once message delivery cannot be guaranteed from a theoretical point of view (if it does exist then the Two Generals Problem can be solved, which has been proven to not have a solution). I don't know how MQTT can get around that, especially in an environment where transient network issues are expected.
/s
What it doesn't really guarantee is the order, which requires even more clever programming and more network exchanges.
MQTT 5 standardizes this by adding a message expiry interval field, though even there it's opt-in by the publisher and defaults to forever.
In other words, MQTT's exactly once delivery conceptually works as if the server would store a big hashmap of all message IDs it has seen once and the rejecting any duplicates using that hashmap. The client the simply keeps retrying with the same message ID forever. This guarantees that the message is received by the server "exactly once". The actual protocol is more involved, but the details are basically just an optimization to prevent the server from having to keep around the "received messages" hashmap forever.
The problem that you're thinking of that MQTT of course doesn't solve is that it is impossible to perform any externally visible action (such as moving an actuator, storing a record into another database system or detonating a bomb) "exactly once" in the general case when you take into account the possibility of failures in between the two steps of performing the action and storing the information that you have performed it. You can either first commit and then perform the action, which means you can not safely retry after having failed between the commit and performing the action, because you might have already performed it ("at most once"). Or you can first perform the action and then commit, potentially redoing the action after a failed commit ("at least once"). There is, conceptually, no way around this unless we get help from the environment, such as an atomic perform+commit primitive. To see how this relates to two generals, think about the exchange of the "performed" bit between the external system and the storage as an exchange of messages that can be lossy. We can not come up with any algorithm that guarantees that the external system and the storage always agree on the state of the performed bit after any finite number of steps.
Anyone care to expound on this part:
>>> and without any way to distinguish a deliberate disconnection from a transient network issue
Last will is given to the broker at connect time. It doesn’t tell the broker that a disconnect is / isn’t deliberate. Last will has nothing to do with the reasoning for disconnect. It tells the broker what to do in case of a disconnect.
Edit: Also, it's worth noting that MQTT 5 greatly expands on the error reasons that the server can send to the client if it wants to indicate precisely why it's dropping the client.
https://dzone.com/articles/protocols-and-standards-behind-io...