You would bother if you wanted reduced latency for a better user experience.
It's a micro optimisation that won't even register for your users, especially if you're reducing latency on 2mb of JS bundles
But, if I had to pay the electric bill on 2.5 million servers, I would definitely care about wasting resources sending extra packets.
Until there is dedicated hardware NICs I think you would find HTTP/3 has worse latency. That is until dedicated NIC accelerators come out on the market.