Cool. You're getting different results to everyone else. Do you see the same on https://random-stuff.jakearchibald.com/apps/img-decode-bench... (make sure you use a browser that supports JPEG XL)?
At smaller sizes (more typical on the web), the results are opposite, the JPEG decodes much faster.
Various workarounds (for example RST markers or self-sychronizing properties of Huffman) have been proposed, but these are not great and do not work for all images.
JPEG XL ensures this information is always available.