I agree with this completely--so much so that I decided to make my own communication protocol (and client-server implementation) that would do whatever I wanted and still be backwards compatible with email. You can see the results here: https://github.com/Zombition/nsmtp-monolith. So clearly it's not impossible.
> What am I missing?
The biggest difficulty isn't from pulling out metadata and mapping it to whatever you store locally. The specifications of different protocols determine the user behaviors that are possible with each protocol, and mapping behaviors between different protocols in a way that makes sense to end users is much more difficult if you want to retain quality communication with users who are not using your software to pull everything into one place. What I found from implementing a new protocol that aimed to be backwards compatible with SMTP was that maintaining backwards compatibility with SMTP guided 99% of my decisions for making the new protocol. If I wanted to add a feature, but it would be confusing for end users who might be trying to communicate with regular SMTP users (or the SMTP users that they were trying to communicate with), the only practical solution is to axe the feature or rehash it in a way that would be less confusing for communication between the two.
In practice this is quite difficult, because you have to balance awareness of both protocols with awareness of how data sent through one can be meaningfully translated to the other, and while you're considering those things, you also have to consider what the user experience is going to be like for people on both ends of the software. If you're trying to maintain compatibility with a widespread protocol like SMTP, there may be an enormous number of different client interfaces that can add a significant amount of variation to the user experience. It's a bit like unfolding a fractal.