But a lot of these games not only Reuse textures but also duplicate them. They do this to improve load times for levels/etc (less seeking).
Playstation DOOM is an early example of this pattern. The WAD files were actually 'per-level' and contained all sprites/textures/etc used in said level.
And that's not relevant to initial install anyway, which can/should be a single download.
You need a program that decompresses no matter what, and the ability to reference existing files barely changes it. It's not a post-processing stage where files get copied around. It's the ability to reference arbitrary chunks of data inter-file just like it can reference arbitrary chunks of data intra-file.
At the most basic level it's like taking a compressed file and chopping it in half. The user already has the first half, then they download the second half and "resume" decompressing.
There are possibly challenges in that however, my understanding of most compression algos (which may be inaccurate or incorrect) is that you'd run into limitations either with size of data overall or the memory requirements.
Of course, I'm sure there's other ways around that (And I have no clue if they are in use). One option would be to send over the assets in a master 'assets' package and then duplicate/assemble the data as intended during installation.
If the data isn't pre-compressed, or doesn't otherwise significantly vary, then using the right tools should de-duplicate the data. Things like lrzip, rsync batch mode, bdiff should work well. If large blocks are identical, many filesystem archivers/backup solutions should do good deduplication, and squashfs can be kind of nice.