Writing a video chat application from the ground up, part 1

https://news.ycombinator.com/item?id=11995441 Describes the premise. There is more detail around that comment in the thread that will address your question

Here's the paper - https://www.cs.jhu.edu/~cwright/oakland08.pdf

This is a variant of "should you compress or encrypt first?"

Compression relies on pattern matching, and compressed size will leak details about what you compressed, even if that result is encrypted. (Unless you then pad the encrypted size, but then what was the point of compressing? I can see some more or less secure ways to do this like establishing a compression ratio/bandwidth/entropy limit, then padding and achieving that constraint so each encrypted payload looks more or less the same, but latency sensitivity makes this difficult)

In the case of VOIP, the codec uses a lookup table for distinct parts of speech (tch, sp, buh, etc). Then "all it has to send" is table cell numbers around (certainly not all). On the receiving side, you just look in your speech table and reconstruct.

These values have distinct output patterns, particularly when compressed. If you can guess better than 70% of the time (I forget the exact number they achieved) what table value was used, then reconstruct it, you can listen in on what they're saying, without having to break the underlying encryption.

Voice codecs are also awful at encoding music which may explain why when you're on hold, the hold music may just be dropped and replaced with white noise because it's reached some bandwidth cap. C.f. video encoding and falling snow.

Yup, if your data is time-sensitive and newer data preempts older TCP is almost always the wrong protocol to use.

What are your thoughts on ENet?

I am not aware of a good obvious default answer. And even when there is a reasonable one (ie xmpp for text chat) it doesn't appear that the open flavor has won in the market place.

XMPP has the draft extension of Jingle, and there are Jingle extensions for things like file transfer. Similarly SIP/SIMPLE could be used, or WebRTC. Any of the three could be used instead of the mess we have now.

XMPP has the jingle extension for audio and video calls: https://en.wikipedia.org/wiki/Jingle_(protocol)

I have successfully used this for voice calls between the chat clients Gajim/Pidgin (I do not remember which) and Google Talk (the now-discontinued XMPP client by Google). I have also done video calls between two Nokia N900 phones, to see if it works. Voice and video via XMPP has worked since approximately 2009 and is just neglected by the companies for business reasons.

The claim that XMPP on mobile can only result in lots of bandwith and battery usage is false, according to Daniel Gultsch, developer of the Android XMPP client conversations: https://gultsch.de/xmpp_2016.html

Money quote: “XMPP is not suited for mobile devices. That’s a myth that has been around for ages. It is mostly spread by people who want to sell you their own proprietary instant messaging solution.”

From my own experience, high battery usage is the fault of the XMPP client. Regarding bandwith, I have been chatting over throttled 3G and regular 2G connections. The initial connection takes noticeably longer, otherwise everything except file transfers seems fine.

In the "last generation" of chat apps, there were certainly a lot of multiprotocol messengers. Trillium and Pidgin come to mind, and were way more popular with bitlbee.

The idea has been replicated in XMPP as so-called transports, which maps other protocols to XMPP. This arguably works better than bitlbee and other IRC gateways, since XMPP is a superset of many protocols, while IRC is a subset of many protocols. https://en.wikipedia.org/wiki/XMPP#Connecting_to_other_proto...

I can join chatrooms in my XMPP client by joining a multi-user chat, like maemo%irc.freenode.net@irc.netlab.cz. It may seem useless at first, but I have found out that when on a train and using IRC directly, I timeout much more often than if using XMPP in the same situation.

A great read indeed, thanks for that.

Minimize size? Maximize speed?

Ah, so you're counting video upload to texture and render of it as part of GUI cycle? Number makes sense then. You're basicaly redrawing whole gui each frame along with video texture and uploading/updating that texture at 60fps.

"Top-down" and "bottom-up" in compression usually refer to which direction you build the prefix tree. In Shannon-Fano coding (top-down), you start with the set of all symbols and their frequency distributions and then recursively divide them into roughly even-sized subsets, assigning a 0 as the prefix to the first and a 1 as the prefix to the second. In Huffman coding (bottom-up), you start with a priority queue of each individual symbol/frequency pair, and then merge the two lowest frequency nodes together, building the tree from the bottom up. In middle-out coding, presumably, the algorithm decides at runtime whether to merge two existing codes into a single prefix tree, or to split an existing prefix tree in some other way. There's some speculation [1] that this is done in a probabilistic way.

All of these algorithms require known frequency distributions, which requires that you have the full data available. In typical DEFLATE compressors (gzip, pkzip, zlib, etc.), this is handled by dividing the input stream into blocks and compressing each block individually.

A similar approach could be used for video - you could easily make the block size a single packet - but in practice, you'd rarely want to use a lossless compression algorithm for video chat anyway. Most video compression is lossy; you can drop a lot of detail before the human eye notices. That's how you can stream a full widescreen movie (which has an uncompressed size of 2 megapixel * 4 bytes/pixel * 30 frames/second = 240 MB/sec) over a typical 5 Mb/sec broadband connection.

[1] http://news.mlh.io/i-hacked-the-middle-out-compression-from-...