And it's why in any good hardware wallet, you need to explicitly confirm the address you're transferring to on the wallet's air gapped screen and make sure it wasn't swapped in the copy/paste.
I'm sure it would also be quite easy for this kind of attacker to generate thousands of addresses they control, and have their software pick one that looks 'closest' to the one being subverted to increase the likelihood that a lazy user just sanity checks a few characters at the beginning and end of the address before confirming.
Survival of the most paranoid.