You can but the issue is that if the caller _thinks_ it can reach it over IPv6 it'll never send the IPv4 unicast packet which means the server won't wake.

Yes, and you can do the same for IPv6 using any ND proxy server.