Matrix uses an encryption protocol of their own devising that employs the double ratchet, which is one component of Signal Protocol. But it's not Signal Protocol, it's their own thing.
That said, we've tried to architect it such that we could implement compatibility with Signal if Moxie and OWS were ever open to it. But it looks like we'll need to first prove that Matrix can support a rapidly evolving yet federated ecosystem without compromising UX or privacy :)