RGB uses less memory than RGBA (obviously), so if there is no alpha channel I don't want to allocate the memory for a channel completely full of 255 bytes.
A 3-byte layout can be worse for performance, I think the simplest solution would be to add a special format that always matches the PNG's format, that way you always get RGB from RGB images, grayscale from grayscale images, etc. This wouldn't require a conversion step.
Yeah, generally if the PNG file is RGBA, then I want to load as RGBA, and if it's RGB, I want to load as RGB.