The reason that it's not that straightforward is that lots of operations change state in your account, so you need proper ordering of requests for everything that comes in if you want to ensure consistency. Furthermore when you do a UI operation you want it to happen now, and anything longer than that is frustrating. But if there is a large IMAP operation in the background, we can't just pre-empt that and re-start it because it might have already modified some state, not all changes are idempotent. So you have to wait, and you experience slowness.
Add in message delivery, various background operations for different features, android sync, different third-party add-ons that hit via IMAP or other sync protocols, and a stack that is more than a few layers deep and it gets very complex, very fast.
Hope that helps,
-Andrew