For anyone considering it, I found Tailscale + Jellyfin work a charm. There aren't great docs for doing so, and I beat my head against it for a little bit, but all you need to do really is to add both your local IP range and the Tailscale IP range to the allowed ranges for Jellyfin.
With that, any device on your tailnet can access it. I went further and set up a cloud VM with a public web address behind an auth, installed Tailscale on the VM, and set it up to reverse proxy port 443 to the Jellyfin tailscaleIP:port on my tailnet. So now I can get to it through any web browser or Jellyfin app on devices that aren't on my tailnet.
I'm extremely happy with the results, and the nice thing is that unlike Plex this setup is never subject to forced changes in the future.