Its weird because I've done AAC decoding on the ESP32 so I know for a fact that it works.

Maybe an issue with patents? All the formats there are open source with the exception of MP3, that from what I remember had its patents expired a few years ago.