JPEG-XL is designed for archive-grade images. It hasn't been optimized (maybe not even designed?) for low bpp settings (less than 1 bit per pixel), and is awful below that, let alone 0.3 bpp or so. Plain old JPEG is much worse. Video codecs (and the image formats derived from them) have optimized for quality at low settings.
> 150ms to decode 12mp is also incredibly slow.
I think that's sufficiently fast. (Keep in mind that a 4k screen is about 8.5mp.) How fast do you want your slideshow to be?