Mike Ash is a well known developer with a lot of great things to say about Apple technologies (I believe his day job is with Rogue Amoeba). He regularly breaks down notable parts of the Apple software stack and reimplements them as a learning exercise. His entire blog is worth a read.
AirFoil is also really neat, but I wish there was an AirFoil to AirFoil zero latency PCM mode that can plug into SoundFlower and just send my audio to my Windows desktop with no otherwise magic... AirFoil via the AirPlay protocol (or anything via the AirPlay protocol, even an Apple product to any other Apple product) has at least two seconds of lag, and also wastes time by encoding it (and is probably the source of the latency). Literally, the latency is the only thing preventing me from adopting the product.
Airfoil is limited by the decision to stick to Apple's protocol. Obviously, this is necessary to talk to Apple's devices. It's not strictly necessary when going from Airfoil to Airfoil Speakers, but it makes life much simpler to just use the same protocol for everything.
AirTunes (what Apple used to call the audio version of AirPlay, which is totally unrelated to the video version of AirPlay... confused yet?) isn't really suited to low latencies. The transmitter basically sends audio data, and then separately sends commands every so often saying, "At time T you should be playing sample number X in the audio stream." This allows sending to multiple outputs and having them all be synchronized. But for low latency you really just want "play this instantly," and to make sure you get audio data to it in a timely fashion. If it's fast enough, then you don't need to synchronize multiple outputs, since they'll be synchronized with "real time."
The major challenge is dealing with packet loss. A typical WiFi network might see 1% packet loss, which you need to recover from somehow. For non-realtime operations, it's easy. You detect the loss, retransmit the data, and carry on. For realtime stuff it's harder. With video you can drop frames. It's not ideal, but it's not too bad. Dropping audio sounds way worse than dropping video looks, though, so that's not really an option for something like Airfoil.
One option I've always wanted to explore was to use forward error correction on the transmitted data. Send out enough redundancy that the receiver can put the original data together even with some loss in the middle. I never have tried it out, but I think it could really cut down on latency.
Without that, you're left with detecting loss and retransmitting, which eats up valuable time. Combine it with code written to do synchronized output with a substantial buffer, and you just can't do realtime transmission.
I don't think the encoding/decoding step (AirTunes uses Apple Lossless encoding for the audio, for those unaware) adds all that much latency to the process, but I will admit that I never went in and measured the bits on the millisecond level, since we were looking at 2000ms end-to-end latency.
If you want to experiment, there is a hidden setting in there somewhere that lets you adjust the latency Airfoil tells the receiver to use. It might be in the window that pops up when you hold option while launching the apps, or you might need to track down a hidden defaults key, I don't quite recall anymore. In my experience you probably won't be able to get it below about 200ms without lots of audible problems, and that's still plenty for a highly perceptible delay.
OK, I'd better stop this before it turns into something long enough to be a blog post on its own!
Is there a generic UDP-based FEC transport library that one could use to wrap a simple Opus or FLAC stream and test this out? Edit: I found http://openfec.org/ but it seems to implement something other than packet loss recovery
AirTunes uses 352-frame packets. At 44100Hz, that's about 8ms per packet. So you definitely do not want to do FEC over more than a fairly small number of packets, or else you'll hit audible latency. (In my testing, if two sets of speakers are playing identical audio but with different delays, latency differences can be heard down to about 10ms with the right audio. Under 50ms is usually OK.)
Of course, if you're doing your own protocol you could use whatever size you want for the packets, keeping in mind that overhead will start to dominate if you make them really tiny.
Another problematic question is, what causes the packet loss? If it's just random, that's one thing, but what if it's outside interference? If some outside transmitter blasts your network for, say, 200ms and causes all data to be lost in that time, then a 2s buffer can recover, but a low-latency scheme is screwed no matter what. I don't know what the answer is here.
One model widely used for simulating packet loss on wireless networks is the Gilbert-Elliot model, http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.90.... which is supported by http://info.iet.unipi.it/~luigi/dummynet/ if you want to play with this yourself.
Apple used to have some sample code but I can no longer find it.