JPEG can be YCbCr, RGB or even CMYK (usually from Photoshop or similar software). There can also be unusual subsample ratios for YCbCr, such as 4:2:1. I created some WASM Go bindings for jpegli, and I recall that such images cannot be represented in Go; therefore, for these images, I force RGBA output.

Also, XYB is an option; I use the adaptive quantisation option from jpegli (with progressive level 2) to get smaller files. I never bothered with XYB, as it looked complicated.