What do you need custom headers for? I think this problem is better solved with tagging file systems.
BIP/BSQ Image Encoding can be done just well (you have (2^16)^3 colors), but you need to leave the interpretation to the outside. It is definitely not in scope of this format to handle that internally.
Compression is external, so yeah, it definitely can be used.
All the things you presented are useful to have, but the question is: For which percentage of people is it useful to have? Is it solved by something else in a better way? You can never please the 1% without making everything harder for the 99%.