XMPP: Swiss Army Knife for the Internet of Things | Hacker News Reader

XMPP: Swiss Army Knife for the Internet of Things

blog.securitycompass.com

For IoT XMPP is utterly wrong. Yes, some planes use it for a message bus. But that was silly then as it is now.

most IoT application have to worry about both power and network efficiency. XMPP is neither light nor concise.

For example, I'm building a sensor network that needs to report a range of sensors every n seconds. I could use XMPP, but first I'd have to figure out how to make a decent XML parser in < 2kb or ram. Let alone a full IP stack. The cheap radio I have has an MTU of 60 bytes.

None of this is going to fly, for what is effectively: Node Address: 1: 56.456 2: -6704 3: 80

If we ignore the <node><sensor><value> boiler plate that will instantly blow our packet budget, lets look at how the values are represented as: ascii. so sensor 1 which was a 4byte float is now a 6 byte varchar.

We are of course ignoring the address which is going to be at least 16 bytes.

We haven't even touched on the server requirements either. Can you name a decent XMPP server? I can't they are all finicky and nasty.

So, no XMPP is not the swiss army knife of IoT, because its too large, too power hungry and just a massive pain in the arse.

Anything that uses ascii representations of numbers is probably a bad fit for IoT

Your argument only shows how wide the field of "IoT" is. For embedded sensor networks, your arguments apply without restriction.

However, for things like "smart lightbulbs" or other devices connected to the cloud (not that I would endorse such things), where you need a scalable infrastructure for secure bidirectional communication between nodes deployed somewhere and your frontend application, XMPP provides a solid base.

Besides, embedded nodes are getting more and more powerful (consider the ESP8266, which runs IPv4 and TLS and could easily handle XMPP), and it provides huge benefits to have a common end-to-end protocol that is not based on periodic polling like HTTP.

The ESP* sucks power, which means bigger PSU, or in the case of lightbulbs, no config support when the light switch is off. Making a PSU that is both small and efficient for both low current draw and high is rather hard.

Protocol design is trivial for something like a light bulb, compared to the cost of designing the device, and making the scalable backend (and no XMPP is no shortcut for that.

I know how wide the IoT field is. I also know the sheer volume of shite that is being produced. Most of the Wifi products will die a horrid death. replaced by something better and more suitable to low power usage.

Failing that something more secure, after a rash of home fires caused by a terribly insecure system that was able to be turned off and on again repeatedly until it caught fire.

Would you suppose it will be something like bluetooth keyboards getting replaced by a usb dongle that talks to your keyboard using some other radio protocol that isn't bluetooth?

There is nothing inherently wrong with bluetooth. So long as you are prepared to put work in over the top to handle security.

Bluetooth LE has a reasonable power profile, and the advantage of being about to interop with lots of other things.

As someone who started a company around IoT before people were really calling it that, I agree with you that the IoT market is being polluted with crap. But I think there is room for multiple service tiers for IoT devices.

For example, some systems and devices need frequent, low-latency updates (such as lights that quickly and smoothly change colors in response to events), others need low latency but only a few packets per second (a light that only switches on and off), etc. So unless there is a low-power protocol that meets all those needs, I don't think we'll eliminate all of them but one.

I wouldn't touch XMPP with a 10' pole even on desktop PCs. That's considering the fact that parsing XML is at least two orders of magnitude faster on PC than anything else, simply because of larger CPU caches.

XML has to be the heaviest data format I know of to parse for CPUs, and the most cumbersome to read for humans. Its the worst of both worlds.

Its not because we bloated our desktop PCs and our mobile phones' software that we need to do the same for IoT devices.

Glad mine wasn't the only "oh hell no!" reaction.

Anyone who's ever dug into XMPP ought to know it doesn't belong anywhere near low-power, possibly battery-dependent devices.

It's a terrible protocol in general. Has all the hallmarks of old thinking, a million people on the panel with a million needs, no consolidation, and no clean minimal implementation that gets you 99% of what you want.

Thanks for saving me the trouble of having to write this.

The problem XMPP addresses has been solved far more efficiently for IoT by MQTT.

Sort of - if the devices are push only: ie, device decides when to push to server.

MQTT might not be what you want if the server would like to converse with a device instantly. Examples include a server commanding a garage door to open, or a server querying a device state.

I'm curious why you say that.

I'm unashamedly a fan of CoAP and have bashed on MQTT in the past. But isn't that the one thing that MQTT is good at? It is a pubsub protocol after all.

I'm not so sure, I use MQTT to control lighting and various other things via NodeMCU and Arduino. It's not as quick as a direct link to the devices, but it's perfectly fine. Far more robust than something like IRC for example.

More and more IoT moves from these very very micro machines to something like raspberry pis. And in that range you probably already have your IP stack and your xml parser since you are using Linux anyways.

And what a disaster for security that is. 95% of the IoT crap that's being made today will never get an update. When you have something as complicated as a linux box the only thing that can mean is in ~5 years we'll have botnets that are an order of magnitude larger.

Yeah probably. But as a species we probably need to make that experience to learn to set up the corresponding systems. Think about how long it took to get some decent server security into the world. And even today there are more than enough computers hooked up to the net that shouldn't be.

Are you using CoAP or MQTT for your sensor network? CoAP fits the bill in terms of message overhead.

I only have 2k of RAM so sadly I don't have the ability to run MQTT (no IP) and not enough ram for CoAP

We do CoAP in 2K on a custom embedded device I work on. What makes you think it's not enough? CoAP is dead simple (though DTLS complicates things if you want to run that).

In that case, I'll have a proper look.... :)

> Can you name a decent XMPP server? I can't they are all finicky and nasty.

I've run a couple of different ones but Prosody has always been a pleasure to run. It's easy, simple to configure and very easily extensible.

Not to say that makes XMPP a good fit for IoT, but it's a very pleasant and capable XMPP server and has a friendly set of developers behind it.

Now I love XMPP and before that I was even building website support apps with IRC. But it seems strange now in 2016 to be looking at XMPP for the new IoT wave because there are plenty of other interesting protocols available.

AVS from Wire Swiss for example.

Websockets that is being used by slack, signal and maybe more.

Maybe fresh eyes, fresh code is a positive thing if we're going to move forward. Instead of using XML. ;)

What's wrong with XML? It's more human-writable than json, more flexible, less quirky than yaml (which I even cannot write without constantly looking into specification). Binary protocols are not better too, in age of lightbulbs running linux.

I like XMPP but I reckon XML introduced some complexity.

For example, many clients (like mcabber) had lots of features broken (e.g. typing notifications) when interacting with GTalk because they couldn't parse XML namespaces properly.

If you can't parse XML namespaces at least mostly properly, you can't write an XMPP client.

Unfortunately, this disqualifies a lot of the less maintained and/or less well-written parsers. It also sometimes disqualifies "bindings" to things like libxml2, when the bindings are written by someone who doesn't understand namespaces and end up wrecking them even though the underlying library supports them fine.

XML namespaces are strange to me. I think they're the simplest example of a standard I consider somewhat simple that to a first approximation nobody understands. It really goes to show that there aren't really all that many people who understand XML qua XML. There's a lot of people who understand "there's angle brackets and attributes and sometimes these entities" and not much more, but there's substantially more meat to correctly understanding XML. Unfortunately (and I mean this fully, not sarcastically or with preaching), to use and implement XMPP requires a somewhat more full understanding of XML than most people have. Namespaces are the biggest sticking point, but XMPP is also a bit quirky in that it tries to have an XML stream, which is sensible enough at the protocol level but often translates strangely up to the API level. (The optimum library for XMPP permits you to use a SAX-like parser for the stream opening, then presents a DOM-like view of each individual element, and there aren't very many libraries that allow both sides of that.)

The funny thing is, XML namespaces are not that complicated. They're the solution I would come up with given the same task. They're defined simply thus:

-- Data model --

1. Every tag and attribute is optionally namespaced by a URI. (Tags usually are, attributes usually are not.)

2. The meaning of a non-namespaced attribute is defined by the tag to which it's applied.

3. The meaning of a non-namespaced tag is defined by the application.

-- Serialization --

4. To save space in the textual representation of XML, namespaces are (necessarily) referenced by user-defined aliases.

5. To save even more space, a default namespace may be (and usually is) defined for use by tags (but not attributes).

6. Alias definitions live in opening tags, and are scoped to the tag itself and all its children. They are of the form xmlns:{alias}="{namespace}", or xmlns="{namespace}" for the default namespace.

7. Tag and attribute names are prefixed with an alias to indicate that they live in that namespace: {alias}:{tag/attribute name}

8. If a tag name is not prefixed with an alias, it lives in the default namespace (if any).

9. If an attribute name is not prefixed with an alias, it is not namespaced.

10. The aliases generally have no semantic meaning, but there are wacky exceptions (e.g. XML Schema).

EDIT, forgot: 11. The "xml" namespace is reserved and predefined as 'http://www.w3.org/XML/1998/namespace'.

I realized I ought to write it up too, so, I promise you I started before I saw your post :)

http://www.jerf.org/iri/post/2950

I'm sorry but this list only provides more evidence that XML is too complicated.

First, XML namespaces are an optional add-on, which is definitely not always used.

Secondly, when you need namespaces, they're a clean, simple solution to the core problem. When you don't need them, don't use them. Fortunately, unlike some of the DTD stuff which is hard to ignore correctly, it's really easy to not use namespaces. If the complexity comes from the problem, trying to solve it with a too-simple solution is not a virtue. You'll just reinvent the same thing, only worse, and with less library support.

For proof... see the endless poorly-supported attempts to do anything even remotely similar for JSON. The good news is that JSON is very simple. The bad news is that when it's too simple, you typically end up with a mess of poorly-designed stuff on top.

OK, propose a simpler mechanism for globally namespacing identifiers.

Given that you find three semantic rules to be "too complicated", I suspect you'll have a difficult time doing better.

I agree, XML is too complicated, but I also don't know of a better tool for the job (and XMPP's only requires a simplified profile of XML). What else provides an ordered collection of anonymous values that can be streamed (eg. you can read just the start tag and have enough information to parse/route the rest of the element), or has 20 years of very fast parsers built for it? I know the whole web has an obsession with JSON lately, and it's great when you need an unordered list of named values, but it's not so great for anything where you need a SAX-style parser, or where you need to be able to rapidly stream data.

SAX-style / streaming parsers do exist for json. http://oboejs.com/ http://rapidjson.org/md_doc_sax.html

JSON has no notion of object "type". So if you have a list of things, whose processing is determined by their type, you may need to parse the entire object before finding the key/value pair that distinguishes its type.

Because XML puts the "type" of an object syntactically first, it supports this use case.

Pity they don't use it for npm, which cannot create a local search index because of this: it tries to load a large JSON file into memory and fails. I never was able to do this on my VM despite trying to give it more memory (up to 2 GB, at which point I gave up).

The explanation is complicated. XML has no namespaces, the real namespace of XML is flat and all names are globally unique. The things they call namespaces are long prefixes that can be shortened; also, one prefix can be made default at any point in the tree. By default the prefix is empty.

[Edit: rephrase.]

If you're lucky you can in often get by looking for xmlns attribute on the IQ or message child element and operate on the assumption that you know exactly what namespaces you expect to see a priori.

But yes, it's an uphill battle if you don't have a namespace aware XML parser.

I think the uphill's not too steep - keeping track of the aliases on your own is quite reasonable.

The thing that does _not_ work is not using namespaces to decide on the meaning of tags/attributes when processing XMPP :)

XML namespaces are pretty simple: they're just really long names with two parts: a long prefix and a small name plus a convenient notation to shorten the long prefixes. Otherwise the pool of names is flat and the names are globally unique. I think it's good.

It's definitely not in any way more human-writable than JSON. Nor is it more human-readable. It's a waste of space and coal used to parse it, and it causes unnecessary strain on the eyes of the poor souls that have to read it.

SGML derivatives make sense for markup, not structured data representation.

>>> What's wrong with XML? It's more human-writable than json

wait, what?

XML is more readable to people that learned XML first (like me). The younger crowd reads JSON better then XML. Just a matter of perspective. The only argument for XML is that you see the closing tags to know where the element ends. However, I avoid using XML for anything these days.

comments... json needs comments

They're not the same.

Both YAML and JSON map to data structures in your program; scalars, sequences and mappings.

XML on the other hand maps to.. a DOM! You've still got to normalize that into what your program expects. SAX is not any better.

Everytime I land in a project using XML I know I'll spend orders of magnitude more time dealing with data than if the project had used YAML or JSON.

> Both YAML and JSON map to data structures in your program; scalars, sequences and mappings.

> XML on the other hand maps to.. a DOM! You've still got to normalize that into what your program expects. SAX is not any better.

Of course, your program probably operates on more than just scalars, sequences & maps, which means you still need to convert your JSON structures into application data structures — validating along the way, too.

My application data structures ARE the JSON structures (or the YAML ones, they both map to generic data types in any language). That's the beauty of it.

If you have distinct application structures that means you're reinventing ad-hoc scalars, sequences and mappings for the sake of encapsulation. You gain absolutely nothing yet you now have mountains of rigid code to maintain and you can't use generic data functions anymore - more ad-hoc stuff to write!

Even working in statically typed languages you can get away with HashMap<String, Object> or IDictionary<string, object> and call it a day. Very few things require the performance of custom types and its a complete myth that types makes you safer - you have so much more code that bugs are more likely to crawl in.

> If you have distinct application structures that means you're reinventing ad-hoc scalars, sequences and mappings for the sake of encapsulation. You gain absolutely nothing yet you now have mountains of rigid code to maintain and you can't use generic data functions anymore - more ad-hoc stuff to write!

No, it means that one has types. What does '{"length": 3}' mean? 3 items? 3 inches? 3 kilometres? For that matter, is '{"length": "abc"}' incorrect?

> Very few things require the performance of custom types and its a complete myth that types makes you safer - you have so much more code that bugs are more likely to crawl in.

Having caught many, many bugs from strong types, I feel comfortable saying that you're incorrect here. Yes, there are even more sources of bugs than just type errors. There are also more bugs than memory-management errors. But just as manual management of memory very rarely makes sense, manual management of types very rarely makes sense.

If you don't have a solid sanitisation/marshalling/untainting/whatever-you-call-it layer, you're going to have security and other correctness errors.

> No, it means that one has types.

{length: 3} has a very specific type. Shoving that into a struct Foo { double length; } doesn't make it any safer. A simple validator and properly naming fields to mention units makes you safer than types.

> For that matter, is '{"length": "abc"}' incorrect?

That's a non-issue with even the simplest of validators.

> Having caught many, many bugs from strong types, I feel comfortable saying that you're incorrect here.

Yet you still need a test suite to handle the huge holes left by the type-system. These tests will catch all bugs the strong types would along the way. Unless you're having a Hindley-Milner type system, chances are you're working in C++, Java or C# and bugs caught by these type-systems are very superficial.

> If you don't have a solid sanitisation/marshalling/untainting/whatever-you-call-it layer, you're going to have security and other correctness errors.

Totally agree, but you don't need types or XML for that.

Having worked as much with statically typed languages and dynamic ones, I'll chose the later any day unless I have very strict performance/memory requirements which is not happening for 99.9% of projects.

Types might catch the easy and superficial bugs, but my ability to reason prevents the hard ones. Having orders of magnitude less code to think about is exactly what I want to achieve it.

> > For that matter, is '{"length": "abc"}' incorrect?

> That's a non-issue with even the simplest of validators.

That's the point: the user-marshalled data must pass through the validator, and once it's passed through, it's no longer the same type.

Perl figured this out with its taint mode 20 years ago. We shouldn't be still exposing ourselves to these sorts of bugs.

> Unless you're having a Hindley-Milner type system, chances are you're working in C++, Java or C# and bugs caught by these type-systems are very superficial.

If a better tool is available, why use a lesser?

> Totally agree, but you don't need types or XML for that.

I'm not arguing for XML. XML is almost as horrendous an embarrassment for the profession of software development as is JavaScript. If someone is looking for a data-interchange format and settles on XML, something is deeply wrong with either him or his situation. XML's just terrible.

But passing around JSON objects within one's code is insane. JSON is a (decent) data-transfer format, an completely unsuitable for computation.

> Types might catch the easy and superficial bugs, but my ability to reason prevents the hard ones. Having orders of magnitude less code to think about is exactly what I want to achieve it.

I completely agree, which is why I don't use XML, JSON, JavaScript, C or Java. It's why I use a marshalling layer: I only have to think about user-marshalled data in that layer; I get to only think about my problem set everywhere else.

> and once it's passed through, it's no longer the same type.

How is that different from loading a JSON into custom data structures? You run the exact same validation step, have the exact same data for input, but your result is now unusable by the entirety of your generic data functions. You'd be better off with a hash-map than a class.

> If a better tool is available, why use a lesser?

Because no tool is better all across the board. Otherwise we'd have the holy grail of programming languages right there. However, I'll favor productivity and the ability to reason about code over performance or some weak validation that I didn't make a typo or sent a string where a boolean was expected.

Besides, I have already spent weeks trying to debug type errors in C++; its ridiculously easy to break the type system. I've also had countless cast exceptions in both Java and C#; how are types helping there? If anything they give you the illusion of correctness.

> But passing around JSON objects within one's code is insane.

That's not at all what you do. When you do JSON.parse the result is already using the language's data structures; there is no concept of "JSON objects". Even in C#/Java you can map these to dictionaries or use reflection to directly map these to ad-hoc structures/classes.

> I completely agree, which is why I don't use XML, JSON, JavaScript, C or Java.

Its rarely up to you. There aren't many endpoints supporting something other than JSON or XML. Anything running in the browser is automatically running JavaScript; anything embedded will go for C.

Once the data is past the validator, its no longer user-marshalled data. Its just data. Wrapping it around ad-hoc types doesn't help your problem domain, it just adds yet another level of indirection you could easily do without.

[edit] I should also add that I strongly favor YAML over JSON for two simple reasons; the format is both versioned and extensible. That fixes the problem in JSON of using string conventions for fancier types like dates. There I can load the data directly in exactly the shape I want it to.

> How is that different from loading a JSON into custom data structures?

That's what I'm arguing for doing. I thought that you were arguing for using JSON data structures rather than custom data structures.

> You run the exact same validation step, have the exact same data for input, but your result is now unusable by the entirety of your generic data functions.

What do you mean by 'generic data functions'? I can do whatever I like with whatever parts of my language I wish.

> Besides, I have already spent weeks trying to debug type errors in C++; its ridiculously easy to break the type system. I've also had countless cast exceptions in both Java and C#; how are types helping there? If anything they give you the illusion of correctness.

They're not: C++, Java & C# are all examples of lesser tools. That's why I don't use them.

> When you do JSON.parse the result is already using the language's data structures

Which is what's insane. I don't want a dictionary; I want an instance.

> Once the data is past the validator, its no longer user-marshalled data. Its just data.

And we discovered decades ago that it's useful to talk about categories of data (types) and about data that carries along with it behaviour (classes). Why limit yourself to whacking things with a stick to collect pinecones when you can use a starship to explore the galaxy?

> Wrapping it around ad-hoc types doesn't help your problem domain, it just adds yet another level of indirection you could easily do without.

Programming is all about building intelligent abstractions which enable you to reason at a higher level. I don't want to think about the third item of an array which is the value of a property name 'foo'; I want to think about interest rates, organisational rules and categories.

I like YAML too; it's a decent format. Not perfect, but decent. Bit big though. I'd prefer something lighter weight like S-expressions though: just enough structure to make it easy to turn into native objects.

> I thought that you were arguing for using JSON data structures rather than custom data structures.

I very rarely go for custom data structures :) I won't destroy data by giving it an ad-hoc type, and unless I'm working in a statically typed language I won't have a "JSONValue" type to deal with in the first place; I'll get the same value from JSON.parse as if I wrote "const defaultCfg = {host: 80};".

But now I can do something like "const cfg = merge(defaultCfg, validateConfig(JSON.parse(cfgFile)));" where validateConfig takes a hash map and returns a hash map (or something behaving like a hash map).

[edit] by destroy I mean you lose the ability to work on unknown data, which is a requirement to build robust distributed systems. When you marshall into a custom type and lose 3 fields, the endpoint is not going to be happy.

> What do you mean by 'generic data functions'?

Functions like map, filter, reduce, take, drop, zip, split, join, get, nth, head, tail, and hundreds more operating on scalars, sequences and mappings.

> Which is what's insane. I don't want a dictionary; I want an instance.

A dictionary is an instance. Your custom class with getters/setters is nothing more than an ad-hoc dictionary.

The only time I'll go for a struct is when I get performance constraints and need to pack/align data and have continuous memory to batch operations and not trash the cache.

I don't want a custom type where I have to write boilerplate around for every single operation when I have hundreds available ready to be composed.

> And we discovered decades ago that it's useful to talk about categories of data (types) and about data that carries along with it behaviour (classes).

Data by definition has no behaviour, its plain information. You're talking about state and there I agree, I want my state close to its behaviour because they're tightly related. But I don't want the _declarations_ of state and behaviour to be coupled either. Haskell and Clojure both solve this problem nicely with type classes and protocols respectively.

You can make your types behave like sequences and mappings, but that's just extra boilerplate when talking about data and not state.

> Programming is all about building intelligent abstractions which enable you to reason at a higher level.

I agree, but you only need a handful of abstractions, most of the time what you're doing by wrapping data in a type is an indirection not an abstraction.

S-exprs by themselves are just data, they're the very same as JSON (not versioned, not extensible). YAML supports tags making it easy to plug your custom types directly into its reader.

> Your custom class with getters/setters is nothing more than an ad-hoc dictionary.

I'd say that a dictionary is an ad-hoc, typeless instance.

> I don't want a custom type where I have to write boilerplate around for every single operation

That's why I use a language which lets me abstract away boilerplate:-)

> S-exprs by themselves are just data, they're the very same as JSON (not versioned, not extensible). YAML supports tags making it easy to plug your custom types directly into its reader.

I prefer (no surprise!) the Lisp reader, which does offer ways to plug your own types directly in with read-macros.

> I'd say that a dictionary is an ad-hoc, typeless instance.

The dictionary will preserve the entirety of the data, that's already better than any custom type can ever dream to do. You're not going to want to upgrade your codebase every time a 3rd party adds fields to their documents.

Trying to make data "safe" by giving it strict types is only going to result in the destruction of data and no actual gains in safety.

> I prefer (no surprise!) the Lisp reader

The Lisp reader is not a data format nor is it available outside Lisp. That's no good on the wire unless you live in a sandbox and never talk to the real world.

You're going to receive data with fields/types your program do not understand (the endpoint could've pushed a new version without your knowledge) and there your ad-hoc types will become pain points.

Imagine if we tried to make HTTP "safe" by giving types to everything. The second a proxy sees a header it doesn't understand the request would fail. We'd have absolutely no internet. Distributed systems depend on the fact that you're able to handle unknown data and types.

What if you need DOM-like data, like formatted text? And "formatted" may mean something relatively simple, like a a basic math formula or even a subscript?

Store it as a plain string. Most GUI libraries I know of will accept that type only for their rich text controls anyways.

If I need to work on the data I'll parse it into a data structure. Then I get something like ["Hello " [:b "World!"]] where I can leverage the existing functions operating on data in order to manipulate it.

Younger crowd still uses HTML and it's very similar to XML.

No syntax for comments, mandatory quotes everywhere, no trailing comma requirement, deeper nesting due to lower expressivity, lack of context (XML elements has name, json nodes usually has implicit context, for example in "Open API Specification" aka Swagger often it's hard to tell what current object expresses).

It does a good job for markup, but really sucks for data serialisation. You have to come up with a schema, when you can translate your data structures 1:1 into a schema for e.g. json or protocol buffers.

With some formats like SVG you even need to parse the document into a DOM and work on it to get things right.

Lots of "features" that actually make things just harder to get right. Like XML entities can lead to a billion laughs [1] or sensitive files being included [2]. XML 1.0 files can contain null bytes, 1.1 files can't.

[1]: https://en.wikipedia.org/wiki/Billion_laughs [2]: https://www.owasp.org/index.php/XML_External_Entity_(XXE)_Pr...

These days you should be following standards, using namespaces, and of course xml libraries. It's not too bad. But it could be much better, that's my point.

It looks like you're just way more used to XML than other formats. Not knowing YAML doesn't make XML any less terrible.

It took me 10 minutes to learn the YAML format and I find it incredibly more expressive, powerful and readable than XML.

Have you noticed that YAML specs are noticeably longer than that of XML? (Of XML 1.0, in particular; and that spec describes both the serialization format and a simple grammar description language.)

XMPP and WebSockets are complementary, not mutually exclusive.

Indeed, these are two different layers of protocols. Websockets is a transport which can carry XMPP if you can't create a normal TCP connection for some reason (eg. browsers); see also RFC 7395: An Extensible Messaging and Presence Protocol (XMPP) Subprotocol for WebSocket (https://tools.ietf.org/html/rfc7395)

MQTT is a better setup for the IoT.

We use XMPP + OTR internally @Infoxchange and it's been really fantastic for us.

It's cross platform (although the Windows clients aren't that pretty to be honest, but we don't have many machines running Windows anymore), it's reliable, we can run it on site and ensure that secure chats are not logged (i.e. 1 on 1 chats aren't logged, rooms are just for history when someone joins), it integrates with monitoring / alerting and general operational notifications such as alerts or notifications from Nagios, GitLab CI builds, Application releases, Errors discovered in logs, Changes to oncall rosters, 'interesting' stats from load balancers, global notifications, secure file transfers and the list goes on.

One thing I keep meaning to get around to implementing for everyone is a web interface for say people that are volunteering / on a temporary device etc...

If you really want to figure out which messaging protocols are being used for "IoT" deployments, looks at the leading IoT device management services providers: AWS IoT and Azure IoT Hub.

AWS IoT uses MQTT and HTTP (over WebSocket).

Azure allows customers to choose between MQTT, AMQP, and HTTP, but suggests that its users choose AMQP as a good general-purpose protocol.

HTTP is usually used for devices/machines with external power supplies that are sending high volumes of relatively complex data. MQTT/AMQP are generally used for nodes/sensors with internal power supplies that are sending low volumes of telemetry data.

Neither service supports XMPP natively which should give you a good indication of how suitable it is for IoT applications.

What measure do you use to determine that AWS and Azure are the leading IoT providers? (Genuinely curious.) I know that they're large infrastructure players who are doing IoT, but are they also leaders in the IoT space?

Revenue share of the overall "IoT Services" market, which includes connectivity, remote monitoring & control, storage, and data analytics services.

Pub/sub protocols like XMPP, AMQP, MQTT are generally used in solutions that are sold into the remote monitoring & control services market.

There is not "IoT market" though, so it's impossible to say who is a leader in the IoT space in general. It depends on the technology category. Eg NXP/Freescale is a leader in the IoT processor space, Wind River/Green Hills are leaders in the IoT OS space (RTOS), Amazon/MS are leaders in the IoT services space.

I wondered how common XMPP was for IoT, my IR Hub (Hub for IR + RF remote) for my TV/Media center speaks XMPP and thanks to some people who reverse engineered it I was able to drop in a library and start sending it commands pretty easily.

It also looks like XMPP is used in some home alarm systems [0]. Unfortunaltey, the developers there did too many things wrong, not only security-wise but also in how they used XMPP [1].

[0] http://www.heise.de/ct/artikel/Home-security-systems-hacked-...

[1] http://geekplace.eu/flow/posts/drafts/xmpp-iot-antipatterns....

Getting sensors, data and actuators to work for ONE usecase has always been easy. just use what you need.

The problem is still how you get these different company systems (silos) to cooperate. "how can you securely give the inside temperature from your alarm's smoke detector and feed it to the heating system to act upon". These services are provided from two different IoT companies.

Here XMPP is the simple and perfect match. Any device in any company can talk to any other device in any other company just as easy as you send email to some one you know in another company.

These companies can solve and hide there internal legacy systems with x500,modbus,coap, binary junk or whatever they need but when you pass the company xmpp server you pack it in a nice letter for others to easily understand what you are saying. With encryption and certificates. And you don't talk to servers/endpoints you don't do business with.

How can NEST homes talk to openHAB? Zigbee legacy with new coap things? And how can they all participate in a smart city ?

And then everything sends a different payload in their XMPP messages, and everything is broken again. XMPP is a possible part of a solution, but not the solution alone.

Thats why the XMPP http://www.xmpp.org/extensions/ is there defining the payload. reading values http://www.xmpp.org/extensions/xep-0323.html writing values http://www.xmpp.org/extensions/xep-0325.html

No there will always be several solutions interacting for different usecases, XMPP fits very well in some.

Also, MQTT.

MQTT suffers from not having a real RPC system. Eventually with pub/sub you're going to end up with a "request ID" that is copied in replies, and at that point you've just invented a really shitty RPC system.

Better to use one that was designed from the start for RPC (and that also supports streaming/push/pubsub as you'll want that too). I recommend gRPC (if your IoT think is beefy enough to support HTTP2), or something lighter like CapnProto's RPC.

Any good Python XMPP library you guys may know ?

Twisted with Wokkel (https://github.com/ralphm/wokkel). It looks unmaintained, but a Python 3 port is on its way, and we are using it actively

SleekXMPP (http://sleekxmpp.com/)

and it's asynchronous fork Slixmpp (https://pypi.python.org/pypi/slixmpp/1.1)

+1 for SleekXMPP. A colleague and I actually contributed a bit to it some years back when we were using it in a commercial application (ARM/Linux hardware.)

I've used the XMPP support in Twisted Words pretty heavily; it's a bit outdated at this point, but Ralph is hard at work on an update that suppors Python 3, so it's worth looking into if you're using Python.

I use SleekXMPP for most of my XMPP work and is about to move in my support for the IoT Xeps into main track which is currently out in my branch on github https://github.com/joachimlindborg/SleekXMPP/tree/xep_0323_3...

Do you know about the TR-069 standard?

Use it right. It's not a message bus. XMPP is a chat protocol. Message delivery is not guaranteed.

Same old, same old: Core XMPP doesn't provide guaranteed delivery, but Extensions do, and today XMPP is more or less unusable without Extensions. I'm talking about XEP-0184 (https://xmpp.org/extensions/xep-0184.html) which is pretty much used in all maintained clients and servers. Note that XEP-0184 provides end-to-end delivery receipts; if you're only interested in hop-to-hop delivery, you're going want XEP-0198. I believe this one is a bit less widespread though.

For guaranteed delivery you really want XEP-0198 (http://xmpp.org/extensions/xep-0198.html) which effectively provides TCP like ack's and session management at the wire protocol level. It's fairly straight forward to implement, and works really well for both chat and message bus style applications where acks are required.

I think it really depends about what we mean by "guaranteed delivery". If you are more interested in the infrastructure and consider each piece as a different part then yeah, hop-to-hop reliability is important to you and you want XEP-0198. That's like "guaranted delivery of a packet". If however you are more interested in the application that sits on top of the infrastructure, then what you really care about is whether the recipient did receive your message: that's XEP-0184. That's more like "guaranted delivery of a message".

Of course both are complementary (and to me both should be implemented)

IQ messages are "guaranteed" at the protocol level. E.g. recipient is required by the protocol to reply, so pretty much all XMPP libraries make it easy to handle an IQ send with a reply timeout. This is not quite the same as "guaranteed delivery" but if you get a reply for your IQ you know it was received.

So XMPP gives you request/reply as well as one-way fire-and-forget semantics. This, in conjunction with presence (so I have a good idea if recipient is currently online) give fairly strong toolkit for reliable M2M messaging.

Reply on news.ycombinator.com