Improving Tailscale via Apple’s open source
tailscale.dev
tailscale.dev
The correct way to solve the looping issue is by configuring the `excludedRoutes` property on the `NEIPv4|6Settings` object associated with your network extension. See: https://developer.apple.com/documentation/networkextension/n.... I imagine this is an important property for a dynamic mesh network to keep updated regularly as the topology changes.
The traffic egressing the wrong interface issue happens because the system handles swapping between WiFi and Cellular radios quite judiciously, and the BSD socket API isn't plugged into this management (AFAIK there's no way to get notified when the delegated interface changes, so you don't really know when to re-bind). Instead, you should use the provided `packetFlow` to write IP packets to the tunnel, which will always egress via the "correct" (whichever one the system decides is best at the moment) interface.
Tailscale's go library would need to implement hooks to call out to the platform implementation for Apple vs others, which is annoying. But, it's the right way and avoids these weird edge cases and bugs.
I know you’re not the author, but I think that type of background would have been an interesting/helpful addition to the blog post and have given it more grit/mileage. Personally I found the post to be link heavy and shallow. I read it wondering why you’re going through all this trouble, instead of understanding a clear problem and solution. A reader like I would be super interested in an explanation of the shortcomings of Apple’s framework and what product requirements or technical constraints make Tailscale’s bespoke approach necessary. I think it sets the stage better for a “we’re off the rails good thing Apple publishes the ifconfig sources, +1 for open source” type of message.
> A number of things in Apple’s APIs are 90% of what we need.
I know the feel.
In the early 2000s, when doing Mac drivers for an OS-bypass HPC NIC (Myrinet, a sort of pre-cursor to inifiband/roce), I spent 90% of my time reading the Apple source and about 10% of my time coding. In my case the first goal was to figure out how to use BSD ioctls rather than Mach based userclient stuff, as we had tons of shared kernel and userspace code for ioctls (across linux, dec unix, solaris, freebsd, windows, etc). And the second was figuring out exactly which iomemory descriptor variant would work well for registering memory.
Not just the unusual, but the usual too: Just this past month or so, a user on Fly.io forums helped debug just why reading from stdin / writing to stdout didn't work for non-root users (https://community.fly.io/t/10375/8). It all started from the fact that Fly.io had open sourced a snapshot-in-time of their init process back in 2021.
Also, it isn't uncommon in the Android world for developers to routinely find the right APIs to use by reading the OS source code.
I needed to tweak the behavior of an Android standard library function, so I copied it's source code (~20 lines) and made some edits. Blew my mind at the time it was that simple.
Sadly all the DriverKit stuff appears to be completely closed source. At least its user-space though, so perhaps a bit less painful than normal kernel driver development...
Unfortunately, I still don't love Tailscale. I do like it, a lot even. But their refusal to open-source all their clients (and the server) is baffling, especially considering that they have an employee contributing to Headscale, the community-led FOSS tailscale server. At that point just open-source the damn thing!
Issues aside, it's still a great product. It actually felt like magic when I first used it in a way few technologies have.
And if you don't trust our server, use https://tailscale.com/blog/tailnet-lock/ and then you don't need to trust us. Or run Headscale. :)
Honestly I (and I suspect most here) wouldn't mind if the server was not easy to set up. The goal here would be transparency. It's true that, with lock, a user can run Tailscale without having to trust it. But it is still a good show of good faith and goodwill to have everything in your infrastructure be as transparent as possible, barring actual user data and service credentials.
Same concept applies to the proprietary GUI clients. What's the rationale for not, at least, making their source code publicly available for reproducible builds (or, if those are too complicated to implement, the same goodwill and transparency I talked about)? You wouldn't even need to actually support the source releases.
And you can already do reproducible builds of our Windows build: the `tailscaled.exe` service and `tailscale.exe` CLI are open source. Only the GUI systray client (tailscale-ipn.exe) is closed.
For macOS and iOS, our development environment is kinda hell. It's great when it's finally working, but hard to get it set up and keep it in a happy place ... you have to get users into the right Apple teams for the right Network Extension Entitlements/notarization/etc, disable SIP to work on certain types of builds, be sure to clean up Xcode temp folders in ~/Library/ so the system doesn't pick up the wrong builds, etc, etc. Then you think things are good and in a few months a random keychain cert expires and you have to repeat the dance. Which sometimes involves a few macOS reboots for some reason.
Yes, maybe we could say that the macOS/iOS/Windows GUI source releases are "not supported" but that will stop approximately nobody from asking questions anyway and consuming time.
Plus I always come back to the question: if you care about open source so much, why aren't you running Linux?
The common reply is: but trust! but security! but auditability! Know how many corporate/paying customers have asked for open source Windows/macOS/iOS GUIs? Zero that I've heard about. Their trust relationship is with other companies, not with codebases they don't have time to build or audit anyway. Or they trust us that all the interesting code (non-GUI wrappers) is open source anyway and read that.
So, yes, we _could_ open source our GUIs. But it's not worth the resulting pain. It'd save me writing comments like this one, but then I'd be answering Xcode/mac build questions instead, and I'd much prefer writing this.
Yes, exactly. I'm one of the biggest FOSS advocates around, but if someone is running Mac or Windows, I have a hard time accepting criticism from them about closed source software. They are obviously pretty comfortable with closed-source/proprietary blobs. If you're running closed systems, you're voting for closed software. It's pretty hypocritical to criticize others to go open when you don't.
With my FOSS projects I do try to support macs, and if it's relatively easy then windows, but I don't blame anyone for taking a "like for like" approach to their software. Open for people who vote open, closed for people who vote closed. Live by the sword, die by the sword, or something like that.
Seems quite fair to me. It's also arguably better for Linux/open source, because if more software vendors took this approach with their software, we'd see more people choosing Linux because it would give us another competitive advantage. That leads to more Linux users, and the more Linux users there are the better it is for the whole community. More software vendors will support Linux, for example.
I can consume closed-source software and care about FOSS at the same time, do not discard people who are on your side just because they aren't extreme enough for your tastes.
Yes, but that's not really equivalent to this situation. This isn't just a "do you care about FOSS" question like the "do you care about animal rights" analogy that you raised.
In this situation, a person who pays for and uses closed source software (and not just one application, but the entire OS), is criticizing another company for making closed source software. It would be more equivalent to somebody who is a meat purchaser/eater criticizing a company for making meat.
Because it's not black and white.
I can easily see myself in a position where I have influence over the VPN we use, while not having control over the finance people using Windows (because of the Windows-only accounting software) and the designers using Macs (for all the usual reasons). As a more general point, "I have stricter openness requirements for my infrastructure than I do for client devices" seems like a reasonable position to hold.
I do appreciate that the armchair CIO crowd keeps raising auditably/trust/security as being much bigger issues than they are in practice, but I have had engagements in the past where the client demanded that any softare we wrote on their behalf had to be open-sourced. So such requirements are in practice unusual but by no means unheard of.
I think I just wanted to point out the similarity. You _have_ to maintain the dev environment, and you _have_ to have it documented in some form for new hires. If that was public, we could be reading a blog post 6 months from now from some company that ran into similar-shaped problems in their dev environment, and solved it using “the power of open source”
I don’t think that’s a compelling enough reason for someone trying to run a business, but I hope at least some people appreciate the irony here.
Sums up the book, Selling the Invisible.
(a pretty good summary: https://penniesintofortunes.com/2016/09/20/selling-the-invis...)
Open sourcing helps with the latter, even if the code's not very well optimized for general purpose use for the former case.
Supporting development on a made-for-self-hosting OSS project is the most user friendly option and allows them to avoid needing to handle the woes of public code.
There's headscale[1] that fills the gap somewhat.
they literally have on their website that they encourage people to use Headscale if they want an open source solution for the coordination server or just to learn how the internals work, they coordinate with Headscale maintainers to avoid breakages, and they don't discourage (but also don't require) employees contributing to Headscale
I don't know where you got this from. I did not say that anywhere.
edit: see comment below me; iOS now has that ability!
Go to the system Settings app, scroll down down down through your apps to Tailscale, and set the "Alternate Coordination Server URL".
https://www.zerotier.com/blog/how-zerotier-eliminated-kernel...
https://apple.stackexchange.com/questions/337715/fake-ethern...
We did this to eliminate the need for kernel extensions to create L2 interfaces.
I chose a dotfile like .gitconfig, but then the iOS Files app wouldn't show it to me. It would show me that there was one item of nonzero size in the folder, but I could not see the actual file, and now (I assume) I can't delete it either unless I delete the folder.
AllowInsecureGuestAuth 1
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\ParametersFrom a different perspective, it appears Zerotier is investing a lot of engineering efforts in maintaining its own protocol, with arguable benefits. Tailscale seems to be deploying new features on an almost weekly basis and are having huge velocity, and to me it just appeared their trade-offs made more sense, and I have more confidence in their business doing great on a long term.
Tailscale has captured mindshare, reminds me some of Cloudflare. There are many solutions out there offering unique value props relative to what Tailscale is doing, and it all comes down to customer needs. I'm not sure how broadly Tailscale has been adopted outside of the dev / HN crowd, so that's always something to dig into.
In all honesty, Tailscale being so good, we never seriously looked at Zerotier. We only evaluated it on paper, not actually tried to use it.
Being able to run Tailscale under otherwise very restricted userland environments, without changing too much code is nice.
Being a mesh means most connections are direct and do not go through any infrastructure which costs Tailscale money, making a Free tier economically workable.
In terms of finding a syscall, you don't usually need to dig through source code. On Linux, strace and gdb will show you a program's active system calls, and on MacOS it'd be dtruss.
We (tailscalar here!) have another post out today on more throughput optimizations and the story there is related in this way: when we started working on that code path the docs didn’t exist! (There are some docs now, in Linux). But either way the stuff is seldom used outside of specialist implementations - so when you need to understand what’s going on you’re headed for the source, or a debugger, or a symbolic tracer rather than an interpretive one.
Once you get up out of the POSIX layers the Apple documentation has improved a lot lately.
Instead of doing that, I would recommend submitting a patch against golang.org/x/sys to get that autogenerated. The Go folks tend to merge such PRs very quickly (<48h).