That said, I really don't see the value of whitespace and comments in our RPC traffic. I wish the industry were mature enough to successfully use ASN.1 for this stuff.
The use of text is the right thing; if you really care about performance at a level that naively compressed text formats can't provide, you frankly need to be defining your own wire protocols anyway.
Why? It's genuinely inefficient. This is not academic, as easily illustrated by serializing even a megabyte of binary data in a text-encoded document. Base64+GZIP is not a solution.
... if you really care about performance at a level that naively compressed text formats can't provide ...
Why are we optimizing a transfer serialization for your own readability and not the devices we're actually targeting?
I've wasted days trying to get barely acceptable XML parsing performance on the iPhone, and similar lower-end ARM devices are becoming much more prevalent these days.
... you frankly need to be defining your own wire protocols anyway.
Protocols like XMPP dictate the use of XML. There wouldn't be nearly as many performance issues if they'd used something akin to Thrift, Protobuf, or even the woefully misunderstood ASN.1 DER.
Since XMPP uses XML, they've had to define complex out-of-band protocols for transfering file data, instead of simply sending the data in-line as simple binary XMPP messages.
I could not more strongly disagree with your position. Protocol implementations should use well-defined, efficient binary serialization mechanisms (such as Protobuf or Thrift) wherever possible to maximize performance, and minimize both serialization overhead and implementation complexity.
Plenty of large data blocks occur without any binary file transfers, like disco items responses from large services.
The solution to this problem isn't a binary encoding of XMPP, it's multiple simultaneous streams. The problem exists regardless of the encoding. Some of the solutions for this set up internal streams (In-band bytestreams and in-band Jingle are two examples). Since well defined protocols already exist for voice/video and other such things, those are used when possible. This isn't a bad thing. No technology needs to do everything itself.
Also, the complexity of out-of-band protocols in XMPP has nothing at all to do with XML. It has to do with NATs and firewalls. It is not easy to set up peer-to-peer communication on the Internet that actually works. SIP and RTP and their ilk are quite complex, I agree. Jingle also has many simple out-of-band and in-band set ups you can use as well, but they aren't nearly as complete or flexible.
Having implemented both XMPP clients and servers, I don't particularly see your point. Exchanging small binary message packets would not be an issue.
How would binary stanzas be any different than other message-based multiplexing binary protocols?.
Also, the complexity of out-of-band protocols in XMPP has nothing at all to do with XML.
The fact that you have to go OOB at all introduces significant complexity.
How can no one hear ASN.1? Andy Tanenbaum's Computer Network mentions it. SSL uses part of it.
As a side note, I hope Google will finish real time search api soon and open the firehose :-)