> Sadly, our local ISP is still IPv4-only, meaning we cannot even access our IPv6 hosts while at home, so we need to fall-back to IPv4 quite a lot.

Have you perhaps looked into using tunnels to fix this? he.net can add IPv6 support to any routable IPv4 address that can respond to ICMP. If you need to stick to your internal ULA, something like NPTv6 can translate the /48 subnet HE provides to your own /48 subnet transparently.

Requires some setting up, but so does maintaining VPNs.