For me the issue that prevented me from really using matrix is that none of the big clients support multiple servers. As a non enterprise user, this has prevented me from seriously adopting it several times.