Apparently it's some variant of SIP. I'm surprised that it's been a decade and no one has RE'd it all and written an open-source client (or have they all been threatened by lawyers...?)

http://blog.krisk.org/2013/09/apples-new-facetime-sip-perspe...

https://news.ycombinator.com/item?id=1509717