Bandwidth breakthrough promises to boost bandwidth tenfold

Protecting streaming data packets up to 1500 bytes in length, by using previous and later packets is something I've never even imagined. Kudos to the scientists behind it!

The magnitude of gains are interesting - perhaps partially due to lack of retransmissions at the MAC level, and perhaps due to TCP re-transmissions that would be triggered across the entire network.

They seem to be keeping the mechanism under wraps and licensing it out under NDA - that makes me a bit sad and skeptical about when we'll get to actually benefit from this - but these scientists have definitely earned their reward...

Wonder what encoding they're using, and if it reduces the error detection within a single packet.

Most of all, the article implies that the system bandwidth is increased with these techniques. In fact, individual client bandwidth can be increased but the system bandwidth is unchanged.

[1] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.24....

here is a paper on the principle at work: http://www.sciencedirect.com/science/article/pii/S0165168405...

Strictly speaking you cannot compress data by expressing the data as a solution a to system of linear equations, but if you are willing to make the system under-determined (too few equations and too many variables) you can compress data. All it looks like is going on is that they are compressing and storing the current packet's worth of data this way as well as half of each adjacent packet's worth. This is slightly less then double the original size because we are compressing the data. This way if we loose a packet we can re-construct it using the data from the adjacent packets.

It works out to be a reliability to memory usage/processing time trade off. For wireless sensor networks (the specialty of one of the labs at my college where I ran across this technique) it is a really good idea. If you lose more then 1 packet then there is not any gain, so any situation where packet loss is bursty and tends to last more then 1 packet then this will not be of any use, so it's application to wifi and cellular are iffy.

Besides the usual difficulties, traditional codes have a structure that precludes their composability. There are several issues with this. One is that they cannot be re-encoded without being decoded. The ability to re-encode blindly is crucial to achieving capacity. For a theoretical treatment of this, I would refer you to: http://202.114.89.42/resource/pdf/1131.pdf The article is rather mathematical but the abstract provides the gist of the results " We consider the use of random linear network coding in lossy packet networks. In particular, we consider the following simple strategy: nodes store the packets that they receive and, whenever they have a transmission opportunity, they send out coded packets formed from random linear combinations of stored packets. In such a strategy, intermediate nodes perform additional coding yet do not decode nor wait for a block of packets before sending out coded packets. Moreover, all coding and decoding operations have polynomial complexity. We show that, provided packet headers can be used to carry an amount of side-information that grows arbitrarily large (but independently of payload size), random linear network coding achieves packet-level capacity for both single unicast and single multicast connections and for both wireline and wireless networks. "

The benefit of this composability, which we had first shown theoretically, is illustrated in TCP in the following paper that recently appeared in Proceedings of IEEE

http://dandelion-patch.mit.edu/people/medard/papers2011/Netw...

The theory of network coding promises significant benefits in network performance, especially in lossy networks and in multicast and multipath scenarios. To realize these benefits in practice, we need to understand how coding across packets interacts with the acknowledgment (ACK)-based flow control mechanism that forms a central part of today’s Internet protocols such as transmission control protocol (TCP). Current approaches such as rateless codes and batch-based coding are not compatible with TCP’s retransmission and sliding-window mechanisms. In this paper, we propose a new mechanism called TCP/NC that incorporates network coding into TCP with only minor changes to the protocol stack, thereby allowing incremental deployment. In our scheme, the source transmits random linear combinations of packets currently in the con- gestion window. At the heart of our scheme is a new interpretation of ACKs the sink acknowledges every degree of freedom (i.e., a linear combination that reveals one unit of new information) even if it does not reveal an original packet immediately (...) An important feature of our solution is that it allows intermediate nodes to perform re-encoding of packets, which is known to provide significant throughput gains in lossy networks and multicast scenarios.

The exciting news that prompted the piece in TR is that we now had an implementation "in the wild", in a simple Amazon proxy, which required significant engineering and works remarkably well.

The article refers to the issue of composability by pointing out that there would be advantage if it were built "directly into transmitters and routers, she says". The more places you build this, the better. Dave Talbot also provided a link to work that our group has done with researchers at Alcatel-Lucent http://arxiv.org/pdf/1203.2841.pdf which touches on this point to show the very important energy savings that could be obtained by implementing network coding in different parts of the network.

My guess is that if you don't implement this carefully, it might actually cause congestion problems under certain circumstances. Regardless, getting a reliable connection for yourself by tunnelling and FEC is nice.

Between the tow half proxy use a hardened UDP communication from the packet lost point of view.

The idea is not original since I (at least) had it long ago but never bothered implementing it. maybe it is already implemented in IPsec. Things are easy if packets are lost randomly. If they are lost in bunches, then things may become more difficult.

I'm however surprized phone data networks or wifi would not have such functionality built in.