E2EE, by definition, means that the server storing the data can't decrypt it. Your server can decrypt the data. Thus, it's not E2EE. Is this a problem? No. You've decided that the server is trusted (a "peer device"), so it's fine if the server can decrypt the data. That's a perfectly reasonable security posture, but it's still not E2EE.