I think the big challenge here is that I'd like my agent to be able to read my emails, but... Most of my accounts have Auth fallbacks via email :/
So really what I want is some sort of galaxy brained proxy where it can ask me for access to certain subsets of my inbox. No idea how to set that up though.
Though of the same idea. You could run a proxy that IMAP downloads the emails and then filters and acts as IMAP server. SMTP could be done the same limited to certain email addresses. You could run an independent AI harmful detector just in case.
But not really sure how to set up safe search. One idea I had was to say "nobody would ever put a secret in the subject line, right..?". Then you could let the agent read all the headers and just have it ask permission to see the body.
That's still not entirely safe since if you can search the body you can eventually infer the presence of arbitrary strings. But I think you could probably mitigate that risk by just setting up alerts for if the agent starts spamming loads of searches?