VPNs are primarily used for remote access, to get random machines access to closed IP networks. Service meshes synthesize a new network (sometimes IP, sometimes something else) to connect a bunch of related machines, almost always with policy controls for who can talk to what, usually cryptographic.
It would be weird (but not "wrong") to use a service mesh to get developer laptops access to staging Postgres.
It would be weird (but not "wrong") to use WireGuard to connect an application server to its Postgres instance.
WireGuard is a much tighter and more limited design, intended for integration directly into operating system kernels, with a strong emphasis on performance. Nebula is a much more ambitious design; it includes direct DNS support, certificates, and server infrastructure. WireGuard is a few thousand lines of very carefully written C code; Nebula is a typical Go project.
They're both very cool.
(Backdrop: I have recently moved our various prod servers into a WireGuard based VPN to encrypt the traffic between them. I found it was easier/pragmatic to do this than:
* to setup SSL for my DB
* to figure out how to encrypt traffic between my application server and Redis or my application server and Nginx )
[1] https://www.noiseprotocol.org/noise.html#introduction [2] https://github.com/slackhq/nebula#3-a-nebula-certificate-aut...
In addition to VPN, Nebula added traffic filtering and spanning different clouds and data centers. I don't think Wireguard had those as goals.
They serve very different purposes. I use WireGuard to encrypt my mobile traffic but I wouldn't have picked it to connect the various hosts in my network at work. Nebula, however, might do the trick.