The challenge is those labels aren't standards-based. Gmail has all sorts of weird issues when you interact with it over IMAP. Even Google's own Takeout will end up making multiple copies of your messages, one per label, as it converts to folders. For a while I still had Gmail but used platforms Google didn't support with first-party apps, and I had a lot of pain points. (Gmail also likes to "archive" items [or remove that particular label] that you tell Gmail to "delete" via IMAP. It's annoying.)

FastMail does an excellent job hanging on real standards, and so supporting these Gmail-specific features requires that the standards themselves, support such things.