Transmitting in-band (headers) seems ripe for arbitrary complexity. Someone out there would write a Turing-complete header DSL. And then someone else would write an incompatible alternative implementation.

At least file extension is limited and externally visible (and thus accountable) to third party behavior, which should limit the worst complexity excesses.

Is filesystem metadata actually different (theoretically) from extension? Or just data in a different format?

Extension seems a nice balance between simplicity / brevity and utility, albeit as a hint, not a commandment.