I'd imagine this is a concern for some people with this type of service similar to how in the past Chrome extensions have proceeded down this path without anyone being the wiser.
It really seems like people don't realize the potential pandora's box that they can open by sharing their emails with someone or an organization, even if that organization is currently trustworthy.
And a couple of questions for you about your infrastructure security. What is your infrastructure security and when you get breached what will you do about it? I'm assuming given your company's ability to access emails someone will find you a juicy enough target and if you get big potentially even criminal organizations and state actors would. If that's the case you will almost certainly be breached to some extent because you have a juicy treasure trove.