Choose your own IP
tailscale.com
tailscale.com
NAT... the cause of, and solution to, all of life's problems.
0% of them have correctly been able to explain what BGP is used for.
Would "I'm just getting lists of IPs blocked by local censorship authority/IPs which are better to access via IP from OTHER country to put them all in VPN" count? :)
p.s. I'm not network admin and never put "BGP" on resume.
It sure is the cause of some of the worst problems.
(Just in case you don't recognize the allusion:
https://arstechnica.com/information-technology/2014/05/what-...
That episode aired in 2001. I wonder if the NAT quote is inspired by that?
Oh, and I can give citation on how NAT breaks something. Until the day we can magically remove application-level gateways[0] I consider NAT a fundamentally broken hack.
[0]: https://en.wikipedia.org/wiki/Application-level_gateway
Liars, you definitely did.
I was a bit surprised when I learned tailscale was addressing out of a single global pool and wondered how they would fix it when they ran out of IPs (and I knew they would, Tailscale was and is obviously that good to me). I vaguely suspected this would be kind of solution they would employ because it's really perfect from an end-user experience point of view, but, thought they might not because it's definitely more complex on their side. Shame on me for misunderestimating the tailscale team.
HN falls for dev marketing really hard every time (see: vercel, stripe, etc)
I'm probably biased because some tailscale employees gave me some of the best written explanation for the evolution of networking infra protocols I've seen, will dig it up from my comment history in a sec
What frustrates me is that people keep building solutions like this that heavily rely on IPv4, even when forward-compatible options exist. With clever use of IPv6 transition technologies, you could have retained support for legacy devices while generally using IPv6 everywhere else.
This is a needed feature if you have no IPv6 AND are stuck in CGNAT hell.
And I’m an IPv6 evangelist.
But there is no traversable NAT if you’re stuck in CGNAT hell with no IPv6 and the CGNAT subnet they gave conflicts with the one you have. Unless you NAT it again or do some other route fuckery.
NAT4444 sounds really great!
There are many more IPv6-centric solutions to their problem. Sounds like they didn't even try to think of alternatives and instead reached straight for NAT. That wasn't necessary, at least from the amount of information we can glean from this one post.
Alas, the "apps and OSes are all willing to use IPv6" problem is a persistent one, so we have to make IPv4 work too.
Nice! Learning ipv6 is on my bucket list, any resources that you like to recommend?
- 128-bit addresses, expressed in hexadecimal. A single character is 4 bits (making every 4 bits a nibble boundary, making allocations really easy)
- All subnets are /64 (if you really want to have a different size subnet, you can, but it’s against the standard, and anything other than /64 will break SLAAC. There is one exception to the standard—point to point links are allowed to have a /127)
- The concept of a network address or a broadcast address within the subnet doesn’t exist. ff02::1 is the all-nodes link-local multicast address (serves the same purpose as a broadcast address in v4).
- ARP is gone. A very similar protocol, ND takes its place
- The preferred way to assign addresses to endpoint devices is SLAAC. Which is basically the router telling the endpoints to self-assign. Ridiculously small chance of a collision, and in case a collision happens, just run the rng again. It’s 64 bits after all. You can use DHCPv6 instead or in tandem with SLAAC if you need more granularity.
- You don’t need to use NAT. Which means you have to set up a firewall on the router correctly. Default-deny, while still allowing ALL ICMP traffic through, as ICMP is kinda vital for IPv6 because it’s used to communicate error conditions.
I’m sure I’ve missed something, but these are all the differences I can recall from the top of my head.
I do think using NAT in the form of NPTv6 is awesome for home use because it allows you to have a consistent address regardless of your ISP prefix assignment.
Think of NPTv6 as a kind of "stateless NAT" where the prefix is mapped 1:1 to your internal prefix. This means if your ISP changes your address, you only need to your external DNS versus all of your devices.
Yes. If your prefix is dynamic, stateless NAT66 can reduce some headaches. But that’s just about the only use case.
https://ipv6.he.net/certification/
When you run into issues, google.
I can't upvote this enough. It is my experience that IPv6 advocates deflect hard when this gets brought up.
The vibe I get is people stuck with IPv4-only IPSs are insignificant and unimportant. They simply do not matter.
Once promoted as a way to give every internet device you would like a reachable external IP … has instead been a new protocol locked behind giant corporations and telecoms. Now mobile providers and some ISP play gatekeeper … don’t allow you to keep any IPv6 devices (rotate assignments) so we end up with little benefit to the end consumer.
The idea that a home user ca setup a desktop and share anything I want via IPV6 another person without the need for a middle man “cloud” provider just didn’t happen. In fact requesting blocks of IPv6 is not easy … and at the end of the day for the average user who cares.
Great so Verizon can now assign my phone and IPv6 address… that I can literally do nothing with.
Corporate greed / government control have severely limited any usefulness of IPv6 at least here in the US
I see this as a feature not a bug. Use dynamic DNS if you need a stable endpoint here.
> The idea that a home user ca setup a desktop and share anything I want via IPV6 another person without the need for a middle man “cloud” provider just didn’t happen.
You can, you likely just need to change the default firewall settings on a typical ISP provided router.
> Corporate greed / government control have severely limited any usefulness of IPv6 at least here in the US
I don't see how either of those have to do with this.
It’s true a few isps will provide you a static up but most will not.
https://www.arin.net/resources/guide/ipv6/first_request/
In most implementations the end user sees no benefit from IPv6 or simply an isp using carrier grade NAT. When IPv6 first rolled out it was promoted as a way for everyone to easily obtain addresses .. reality is not so much. End user functionality crippled Or gatekeeped
Hypothetically should be to host a simple file share of let’s say photos on a desktop and share using SMB or another protocol via IPv6. No need for cloud provider / iCloud Photos, Google or whatever. Just a direct connection with a desktop that stays on 24/7. This is not the case and effort has been made to make IPv6 really more of a useful protocol to mobile isps managing large numbers of new devices.
IMO #lame
How do you tell an embedded device to push a DDNS update?
So i hope one day you will be able to register with user and password
If you want something fancier Authelia isn't too bad, I got that running in an evening and hooking it up to Tailscale took another hour or two. Most of that spent figuring out how I want to do webfinger.
1. Tailscale has their custom OIDC docs that tell you everything you need, plus the Webfinger setup: https://tailscale.com/kb/1240/sso-custom-oidc/
2. I set up Webfinger first, so assuming you're setting it up from scratch you can either run a Webfinger server yourself, or just configure the paths in whatever web server you have for your base domain. I didn't feel like running Yet Another Server and since the Tailnet's only for me I just plugged the following section into Caddy:
@webfinger {
path /.well-known/webfinger
method GET HEAD
query resource=acct:MY@EMAIL
}
rewrite @webfinger /webfinger.json
header @webfinger {
Content-Type "application/jrd+json"
Access-Control-Allow-Origin "\*"
X-Robots-Tag "noindex"
}3. For Dex you can just set it up like any OIDC connection; Authelia was about the same but they have their own page: https://www.authelia.com/integration/openid-connect/tailscal...
Took me about an hour or two, most of that being wishy-washy on how I wanted to serve Webfinger.
If the admin console is run by them, it's pretty trivial for them or an attacker to add nodes to my network. Zerotier suffers from the same issue.
Tailscale is cool and the third party login is also a problem for me, but the hosted service in general is a much bigger core issue with it for me that not only affects privacy but also security.
For me the headache of running my and maintaining my own server isn’t worth it (+ would still require a GCP/AWS account)
But why would it need a gcp or aws account? It could run on any vps right? I'd run it on something much cheaper like scaleway.
Probably having a full time job that people will actually support? Open source software doesn't reward effort.
Headscale does not have a nice UI, its basically all CLI usage. There's very good reason to use Tailscale for companies and there's also good reasons for Tailscale to support an opensource implementation of their control server, I've seen it from both ends as people go either way, it's a symbiotic relationship and probably one of the best examples in open source today.
(Yes the company _could_ do a sudden 180 and start intentionally breaking compatibility and forbidding that their clients be used with third-party servers - but the risk of that doesn’t seem much different than the risk of an open-source alternative being abandoned)
I remember watching many years ago a talk about a mesh network scheme where its users would unambiguously assign themselves addresses through some hash function. I was fascinated by this concept of generating my own address (instead of having it assign to me) and that it could possibly be mine forever, perhaps associated with some biometric marker.
Anyway, this is also cool, just less ambitious :)
Was it this one -> https://github.com/cjdelisle/cjdns/ ?
Anyway, I fully agree with you. Assigning meaningless (and therefore collision-prone and spoofable) numbers to people/machines just seems like such an arcane idea in light of all the cryptographic advances people have made.
Whatever network I configure, I always need to worry about collisions or possible misconfigurations. Whether I set up some private VPN for myself and have to worry about potential spoofed IP packets from outside, or whether I connect to a corporate VPN from home and suddenly I can no longer ping 1.1.1.1 because their IT department couldn't be bothered to configure their routes correctly. It's insane.
But yes, someday, maybe we’ll laugh at the current state of things and the shortage of IPs.
I may be putting my ignorance on display here, but I recently completed a site-to-site network between two farms in rural America, no other ISP can serve these farms, and they needed to communicate cow data between the different farms. Tailscale did the majority of the heavy lifting thankfully, and we were able to get them all sorted out.
I could not get Wireguard to work, and that may be down to my limitations in networking, but I was sucessful with tailscale, so make of that as you will.
If you signed up for a $5 VPS to forward packets you probably would have been fine.
I'm sure I'm leaving money on the table, but I'm a one man show, and I have trust issues with relying on others, so here we are.
If anyone knows of a Wireguard walk-through to bridge two separate LANs together, I'd love to see it
There are tools to automate this like wg-quick and systemd-networkd, depending on your preference.
The thing about Wireguard is that it's very simple and minimal. It does just one thing, and that is - establishes a layer 3 tunnel for sending IP packets between local machine and some other peers. It doesn't do mesh, it doesn't do routing (it just knows the IPs of its direct peers and that's all it does), it doesn't do bridging - all this stuff is done by other pieces such as Linux kernel, but not Wireguard itself.
> Wireguard walk-through to bridge two separate LANs
Same or different subnets for those LANs? If they're different and non-intersecting, and if you don't need cross-LAN broadcast or multicast, the simplest option is to establish a Wireguard connection between those LAN's default gateway routers (assuming you can do this), and on each of those routers set up a route that sends opposite LAN's traffic to the opposite gateway (in case of iproute2: `ip route add my.other.lan.subnet/mask via my.other.lan.gw`, how to make this persistent depends on your distro). Then, on each gateway, allow packet forwarding between Wireguard and LAN interfaces (with e.g. iptables or nftables or whatever you use there).
If you can't run Wireguard on gateways, the overall principle holds, but you'll need to distribute routes to your respective LANs via Wireguard-running routers through DHCP or whatever you use for routing on your LANs (e.g. OSPF).
And if your LANs both have the same subnet, or if you need multicast, things get significantly trickier (plus, there's inevitable question of what should happen if two machines on different LANs have the same IP). You'll probably need to run something like L2TP or GRETAP (or something else that can encapsulate you layer 2) over Wireguard.
Or maybe just use OpenVPN in TAP mode (if you want all stuff independent of any third parties) or Tailscale (because it already works).
And Tailscale surely has one benefit here - it's one single product, with essentially no variations, so it's (I presume, I haven't ever used Tailscale myself - never needed it) easy to write a step-by-step instructions for. Generic "GNU/Linux software router with Wireguard" is an extremely vague target that impossible to give instructions for, unless you spend a lot of time describing the problem in finer detail.
You could use Headscale, which is an open-source/self-hosted reimplementation of Tailscale control plane, so you can eat your cake and have it too.
Curious to know, why does distrust towards Tailscale come up so often around here? They seem extremely transparent about their strategy and incentives.
I have a guess - I suspect it's because in the domain it addresses, attitudes are towards self-reliance, privacy and autonomy.
If someone uses Tailscale in some cloudy (aka all someone else's computers) setup, they probably don't bother. They already shift trust and rely on other people.
But Tailscale is infrequently used to manage own devices, which is why it clashes with self-reliance attitudes. If you run your own private hardware and networks, all or many of those in your own castle, it may bug you to introduce (or I'd rather say trust) a third party unless you're forced to. Not because it's not trustworthy, but because of the self-reliance attitude and desire to have full control over what you think of your own systems.
Sure, you're forced to trust your electricity provider (but you have an UPS) or network uplink (and even then you make security precaution), but trust in Tailscale is kind of optional (it's not irreplaceable), and not everyone feel like they want it.
Pretty much the same why a lot of people frown upon IoT stuff, even if it's from rare vendors who go extra mile to make things reliable - because of the hypothetical "but what if?" and feeling that you're losing the control here.
Just a guess, though. Others' mileage may vary.
In my personal case, It's a mix of self-reliance and a committment to open source that makes me want to have an alternative to tailscale (although I use Tailscale for company stuff and that was my call). On top of that, for personal stuff I just have very simple networking needs, and I don't want to add yet another service (a headscale instance) I have to maintain and keep running, and in the case of headscale it's particularly important because I might lose my network if it goes down! For that reason, a tailscale-only solution that is all client and no server is attractive.
Side note: the existence of headscale is the reason why I decided to pay Tailscale for company stuff. Had the clients not been open source, and there had not been a compatible FOSS server implementation, I would have spent the time/money slinging wireguard or using some other solution. I love Tailscale for opening the clients and allowing/even supporting headscale.
You don't have to use their service to implement what they describe, in most cases.
This may be verging on off-topic, but it's too good an opportunity to not deliver this very fitting reminder: in 2018 Anja Karliczek, then-Minister of Science of Germany, made headlines for the quote "5G nicht an jeder Milchkanne notwendig" [1] ("you don't need 5G on every milk jug") aka the rural areas (that had been left behind even with prior mobile phone/data and even landline DSL deployments) didn't have enough need for fast Internet access.
So, to answer her question, what exactly are these farms doing that they need to exchange cow metrics? Tracking of milk production per cow?
[1] https://www.spiegel.de/politik/deutschland/anja-karliczek-br...
You need tailscale when:
1) You have a lot of road-warriors
2) You need decent ACLs that isn't just IP based.
3a) You don't ability to directly connect to a site (just one needs to be accessible)
3b) You can't add 3rd site to relay traffic or punch holes in NAT.
4) You're afraid of terminals and/or text editors.
Here is a good write-up on the subject: https://www.procustodibus.com/blog/2021/11/wireguard-nftable...
I live in Iowa and the problem we have is nobody markets their stuff very well. We have a fixed wireless provider that, looking at their website, looks like they are just a cell tower climbing service. But they also lease space on just about every tall structure in the region (grain elevators etc) and can connect anything to anything. We got to them because we asked a WISP if they would do a private backhaul and the guy referred us over to somebody's cell phone. I guess they are happy to just have word of mouth and only have the website to reel in bonus work with the out of town telcos.
Never had worse than next day service on any lightning strike event from this guy, which is more than I can say for mediacom or lumen lol.
The good news is that it's relatively easy to punch through the NAT. This blog post describes an approach that is working well for me: https://www.jordanwhited.com/posts/wireguard-endpoint-discov...
You should be able setup a Wireguard instance on some cloud provider like AWS, then setup each Starlink endpoint to point to that public cloud IP. I have done this for some time on a cheap $5/mo Digital Ocean droplet to work around my old ISP giving me a RFC to 1918 addresses.
Also don’t forget that if you have a line of sight between the two farms you can use the unlicensed 60Ghz spectrum and grab some cheap Ubiquiti millimeter wave gear.
Honestly, in none of my use-cases will it matter - I can't see myself running a gossip protocol across servers that I do and don't control.
This can be fixed by manually setting the IP in the Tailscale console though, so for this particular usecase I think it's okay, too.
Do the "minimalist" people have good reason to prefer "anything-else" other than "heavy feature-rich Tailscale"?
If you're a home user, the advantage to Tailscale is that it's going to "just work", with a couple clicks, on any supported device (of which there are lots). There's no configuration to get started and, for a lot of users, no configuration ever after that. The onboarding experience is spooky; it's upsettingly good.
If you're a corporate user, the advantages are drastically greater: you get SSO integration (this is historically one of the annoying pain points of corporate access VPNs, to the point where a significant fraction of pre-Tailscale netsec teams just punted on this problem and hand-provisioned VPN creds for people, which is a nightmare) and trivially simple group-based access control.
To be honest, in 20+ years of working in IT, I never understood the point of the latter until recently, on a gig salvaging systems for a client with ~650 users after their sole IT guy unexpectedly resigned after 20 years and left for the mountains.
IRL, SSO is gold. Many hackers, like me, underestimate it.
And that's for the free and cheap tier. If you want the fancy stuff (like SAML and automatic user provisioning / filtering), they've apparently got that, too, but it's in the more expensive tiers.
And trying to do access control without SSO is crazy: you need to keep track of application and users and their interactions. I wouldn't run any team with more than 10 people without it.
Less battery life on your iPhone. :)
I have both in my home network and love both.
At work, I have used Tailscale in my Kubernetes clusters to allow devs to get into the private subnets, so that is awesome, and way better than trying to give a group wireguard key pairs.
Other than that tailscale is really great. It just works.
I personally haven't deployed it though I've toyed with it, but I think as well as UI and integrations a core topology differentiator is that, like Nebula, Tailscale does/can do meshing. Plain vanilla Wireguard is pure classic hub-and-spoke, which is 100% fine for a basic VPN use case like "I'm out somewhere on the WAN and want to talk to this LAN stuff" or "I want to tunnel some/all my traffic through some specific alternate exit".
But say you've got main site A which has a public static IP and is where support is for administrating others, site B which has a full backup server but no public IP, and then sites C/D/E/etc where people are doing work and having significant on-site storage and comms needs, all of which are behind typical ISP NAT from multiple different ISPs with no static IPs. Everyone wants to be able to do high bandwidth things like video chat directly together, or back up/restore to site B. Plain WG could do that, but would funnel it all through site A's link which isn't very scalable and likely to become a choke point in a hurry. A meshing VPN can let two private sites talk directly with a public address only serving to facilitate hole punching and setting up the connection each time. It's definitely of real value. Another thing would be not bandwidth but latency. If you're within a few hundred miles on land that probably is irrelevant. But if different sites/people are across continents adding an unnecessary extra hop may become a very big deal even for simple web apps. Resiliency also enters the equation, what if site A goes down? A mesh can help with those too.
Then Tailscale adds a lot of cool QoL on top. Meshing does raise new challenges in terms of access control vs when everything is funneled through a single convenient point. But regardless, other topologies can be of basic interest too even without extra sugar.
No it's not. You can do any to any just fine (and any topology in between these extremes).
However, outside some well-connected datacenters with multiple peering exchanges, I have no clue where in the world you can run everything in IPv6 with even a single nine in availability.
On my home I would say assuming single nine availability of IPv6 traffic is too much availability - it's very common that IPv6 is borked for several months in a row.
The VPN clients for Mac and iOS are on the App Store, which may not mean much, but having developed VPN apps for both, what it means is: it is far less likely break or muck with your OS's networking in practice because it's sandboxed and can only use Apple's SDK for interfacing with the OS. This is compared to every OpenVPN client I've used on various platforms, which must run as root and often is setting up and tearing things down with shell scripts that can get hairy as you add more complexity / moving parts.
(Note that this is also true for Wireguard's client, just not OpenVPN)
The first three users are always free, so we're able to demo it easily. It's also listed on AWS marketplace, so as we move to start buying some licenses, it's billed through our AWS bill (i.e. I don't need an act of Congress to get a credit card number entered and a new monthly invoice reconciled within my company).
You can configure how often it forces reauthentication, which is probably the biggest benefit over vanilla Wireguard. Wireguard doesn't have mechanisms for expiring and replacing keys, so it solves that.
There's also an open source implementation of the master service (called headscale) that you can run on your own, and I was able to fairly easily set it up and get the existing Tailscale apps from the App Store to be reconfigured to utilize.
Honestly it's the cleanest VPN experience I've had if you need to deal with any kind of SSO and/or dynamic user/client provisioning. If you're just setting up point-to-point between a few of your own servers and clients that won't change, maybe just stick to Wireguard. But once you start needing anything more than that--I'd give Tailscale a shot first.
I've never configured Wireguard from scratch but I have managed an OpenVPN deployment in the past. One of the most fabulous aspects of Tailscale is that it's very self-served; we configured our Tailscale account to allow email addresses from our main domain name with O365 integration. When someone wants to bring a new node online, they log in with their O365 credentials and magically new keys are assigned to the node and associated with the user who created them. In the past with the OpenVPN deployment, it would usually take me 15-30 minutes to get a new node online (generating keys, getting them handed off to the user, helping them debug, etc); now it takes me 0 minutes because the user can just generate their own keys and I can be completely hands off, while still having a nice view that I can use to revoke keys if needed.
In comparison here's the Tailscale setup instructions: https://tailscale.com/download/linux. If you're into running shell scripts that you pull with curl, you can set up Tailscale on a new node with:
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
I will probably try playing with naked Wireguard at some point for my own home network (since the Tailscale client doesn't seem to handle two orgs at the same time very well).
A technical person who’s familiar with what a VPN is and does but has never configured one can have it working on a bunch of their devices in like 30 minutes flat, with no notable ongoing maintenance to worry about.
If you’re already configuring your home networking with Ansible or Helm or something, it’s probably not a win.
Here is what I will say. I manually setup wireguard for my home network, it was annoying but doable. Then when tailscale started to get interesting I just decided try it out using a free account and rolled back my wireguard configuration... and never went back. It's so much less fiddly.
SSO for Auth: before we had to go through the key exchange process for every employee and then manually update the Gateways wg conf. Now it’s just: login here with your work account and you’re done.
Authorization config: I like the ACL abstractions on top of Wireguard. It’s a part is completely missing and building it yourself would be a nightmare. I also don’t want to manage iptables for every device.
All of the nice little feature that they provide:
Funnel Kubernetes operator Magic DNS
I used to have open ports publicly, but have since closed everything down and set up tailscale. Got tired of having to manage a firewall, religiously keeping everything up to date, and script kiddies trying to get in.
Tailscale reduces management to near zero, or as close to zero as you can probably get. Even family members can use it, and without my help.
On Android I don't want Tailscale connected all the time, so I have Tasker auto-connect whenever I open an app that needs LAN access. It's pretty convenient, but I'd still say it's the most inconvenient part of my current setup.
Now I need to investigate!