> best possible compression

This is a common incorrect assumption. You have to balance ratio with speed, otherwise you end up with something that spends more time decompressing than actually downloading, as is literally the case today with XZ. this is why alternatives like Zstandard are getting popular. 9 times faster decompressing for 10% worse ratio.