IPFS isn't designed for private communications. It's designed for highly decentralized publishing of content. Safe, in this context, means the data is safe to "be".
This should be addressed by the client, not the server. The server's function is to serve the data, not serve data and pretend it doesn't know who it gave it to.