Matrix isn't peer-to-peer, it's federated. Messages are stored on "homeservers", which is similar in concept to an email server except it's much easier to host your own Matrix homeserver. Only the homeservers which have involved in a chatroom ever see a copy of the message. For private chatrooms, everything is E2E encrypted by default so the homeservers don't have access to the plaintext of your messages.
Current implementation uses a server, though you can fairly easily host your own, or find another server to your liking.
There is significant work being done to get matrix working p2p too, though. There is a demo of it, but it's not production ready. See https://p2p.riot.im/