When I and a friend use E2EE between us neither of us are helping to hide a criminal's conversation regarding their criminal activity. The same can not be said about a mixer. They are not at all equivalent.
1. You are using and supporting the same protocol that they are also using. As the application grows and improves because more people are using and supporting it, the criminals are also being helped.
2. You are creating cover. The more people that use Tor and Matrix, the more secure it becomes for all users within the network.
If you and a criminal are both using the same Matrix server, neither you nor the host would know. Your plain text messages are going into the protocol, getting mixed and encrypted, and then spit out the other end.