Yes, there was a bit of a drop off in performance but only when YJIT is not used. With YJIT there are performance gain. Also note that YJIT in 3.1 was still experimental and more proof-of-concept. Only with the 3.2 release was it now deemed production ready.

And YJIT's creator, Shopify, is already using it in production with Ruby 3.2 for all their store fronts!