JPEGs are intrinsically YCbCr not RGB, so I don't know how you would avoid some precision loss when you combine that colorspace conversion with things like the quality level and (potentially) subsampling.
Also, XYB is an option; I use the adaptive quantisation option from jpegli (with progressive level 2) to get smaller files. I never bothered with XYB, as it looked complicated.