Hmm, I use IPv4 mostly because nobody in their right mind can remember a IPv6 address…
Just because it's a 128-bit number doesn't mean it should be difficult to remember, the standard notation goes a long way toward that. 2001:db8::cafe:f00d and fc00:bad:beef::1 aren't what I'd call the epitome of "can't remember"
Mind that real-world global addresses often have four groups of almost-random at the beginning, but it's usually not terrible to commit to memory.
For sure this is a self-hoster thing, where you have pets not cattle, but so is memorizing your v4 address(es)
Asides from potential address conflicts, should your work VPN space overlap with your LAN subnet, for example.