Matrix is also experimental. About a year ago it kept losing my key meaning I couldn't decrypt old messages. I still use matrix, but I don't have any illusions that it's more secure than tox.
That said, we've also just implemented the optional ability to encrypt and backup your keys on the server, but obviously comes with other tradeoffs.
In terms of security, the core crypto has been audited, as per https://matrix.org/blog/2016/11/21/matrixs-olm-end-to-end-en...
Great to hear you've added the option for server stored keys now.
The new online key backup stuff landed a few weeks ago on the develop branch, and will be making it onto the main release over the coming weeks :)
That conclusion doesn't follow. How does the client loosing your key cause you to consider matrix itself as less secure?