No, it has always used its own protocol.

But apparently there is an XMPP extension that implements the Double Ratchet algorithm that Signal is based on:

https://en.wikipedia.org/wiki/OMEMO