It would be like complaining Vaultwarden is bad because the Bitwarden project is not fully open source even though Vaultwarden is fully open source and has most of the features implemented.
And Headscale kind of ticks all the other boxes mentioned, except "not headscale", because:
* p2p mesh network - it is a mesh network. And even when mesh is blocked, you can use multiple relay servers (derp) which will relay to the mesh from closest location. And you can host your own derp servers.
* Open source and selfhosted - check
* Not Wireguard (Signature-based blocking) - in cases where wireguard is blocked, the derp relay servers run over https and are usually not blocked based on signatures. For example, I use it with Traefik proxy in TCP mode so I could run derp and other http services on the same 443 port and it works great. So - check?
Packaged in nixpkgs - check
On top of that, if you add Headplane admin UI you get nice graphical management, very similar to the one of Tailscale.