Do you? I just tried it locally, and decoding a 4.9MB, 6000×4000 JPEG with djpeg took about 400ms; decoding its losslessly-recompressed JXL (4.15MB) single-threaded with djxl took 300ms.
At smaller sizes (more typical on the web), the results are opposite, the JPEG decodes much faster.
Various workarounds (for example RST markers or self-sychronizing properties of Huffman) have been proposed, but these are not great and do not work for all images.
JPEG XL ensures this information is always available.