I finally understand Cloudflare Zero Trust tunnels
david.coffee
david.coffee
I actually have no idea how private networks with WARP are here, but that's a pretty big privacy downgrade for tunneling from the Internet.
I also consider P2P with relay fallback to be highly desirable over always relaying traffic through a third party, too. Firstly, less middlemen. Secondly, it continues working even if the coordination service is unavailable.
> client-less access
JFYI, Tailscale Funnels also work for this, though depending on your use case it may not be ideal. Ultimately, Cloudflare does handle this use case a bit better.
This may seem like a lot of effort and it is definitely not nothing, but Cloudflare Tunnels also has a decent number of moving parts and frankly their authentication gateway leaves a bit to be desired for home users.
Not so! I have custom domains on my Tailnet with Caddy. You just need to set up Caddy to perform the ACME DNS challenge to get certs for your domain (ironically I use Cloudflare for DNS because they make it very easy to set this up with just an API key). No plugins or open ports needed.
1. End-to-end encryption.
2. Performance and reliability. 100+ PoPs in all major clouds running their data plane routers if they host (still E2EE), or run routers anywhere if you self-host. Dynamic routing to find best paths across the routers.
The productized version, NetFoundry Frontdoor (doc here https://netfoundry.io/docs/frontdoor/how-to-guides/create-mt...) is what offers mutual TLS support.
It'll still terminate TLS at the servers, though. It's not mTLS all the way through to the endpoint.
> It'll still terminate TLS at the servers, though. It's not mTLS all the way
> through to the endpoint.
+ e2ee via netfoundry's zero trust products
+ non-e2ee via netfoundry frontdoor
One of the design decisions I made was P2P or nothing: there’s a small signalling service, but no TURN/relay servers. If the peers can’t establish a direct connection, the tunnel just doesn’t come up.
The trade-off is fewer successful connections in weird NAT setups, but in return you know your traffic never transits a third-party relay – it goes straight from your client to your friend’s endpoint.
- With a TURN/relay, you’re introducing a single, purpose-built box that: - sees all the tunnel metadata for many users (IP pairs, timing, volume), - is easy to log at or subpoena/compel, - and becomes a natural central chokepoint if someone wants to block the system.
- Without that relay, your traffic still crosses random ISPs/routers, but: - those hops are *generic Internet infrastructure*, not “the TunnelBuddy relay”, - there’s no extra entity whose whole job is to see everyone’s flows.
That would be great!!
This doesn’t sounds zero-trust at all to me. In fact, it’s as far from zero trust as you can get.
But are you accusing someone of promoting vendor lock-in (cloudflare) while at the same time promoting vendor lock-in (tailscale)?
If you’re ok with vendor lock-in, shouldn’t you in theory be ok with any vendor?
The always-free infra remains free, you just have the chance of incurring a bill if you make selections that aren't free or exceed block storage/egress (200GB/10TB) limits of the always-free tier. Leaving the free/trial tier gives you access to a much larger pool of instances. I never successfully deployed an A1 instance prior to becoming a "paying" customer - now I've done it hundreds of times without ever having an issue.
I've been running a small k0s cluster and a standalone webserver for months while incurring about $2.50 - $3 in spending each month, primarily from being slow to remove instance snapshots sitting in block storage.
Even things that are oddly expensive on AWS - like NAT - are free on Oracle. There are zero gotchas.
The free tier is also based on capcity usage, and not instances. If you want 3 cores on 1 machine & 1 on another, they're cool with that. I personally run Pangolin on a 1 core & self-hosted github runners on a 3 core.
Only caveat I see is they reserve the right to delete underutilized/ idling instances
Thanks
I have a server at home that works well. I don’t reaaaally want to pay an extra $30-$40/yr and have an extra thing to manage when the CF tunnel works fine for free. I like Tailscale more, but I want to share this with family who won’t install TS and also want to use a specific domain.
The solution we've found is running a white IP container (or VPS) which looks like regular Wireguard outside, while inside it "forwards" to your existing tailscale network.
I don't remember if we use https://github.com/gravitl/netmaker or https://github.com/juhovh/tailguard
...which doesn't even try to get a p2p connection. Instead you always get the thing you didn't want. If you're okay with that you could've just ignored how Tailscale connected those devices, that's kind of the point. You've also in the process converted your entire security model to Cloudflare's idea of "Zero Trust" which involves 100% trusting Cloudflare.
The rest of the blog post is fine, but the motivation is honestly baffling.
On the other hand, my experience with Tailscale is that they're very, very good at NAT hole punching and I'd rather have a direct connection where possible from a latency standpoint.
This is my experience, we are a fully remote world-wide company and we recently migrated away from Tailscale to Cloudflare and it has been much better.
Peering in Europe is such a mess that even Cloudflare can be pretty bad. Sometimes you have to manually calculate "okay, there's a colo in this particular city that will force the correct route if we proxy all our traffic through it ..."
Perhaps CF could license it and slap it in their docs!
This way I can upload big videos when I get home.
The specific term is: https://www.cloudflare.com/service-specific-terms-applicatio...
I use the tunnel because my girlfriend cant install tailscale on her work laptop, so this way she can still login to jellyfin while traveling.
I'm not saying they can't have that rule, it's their infra - I'm just saying that "a boatload of bandwidth" can be anything, depending on who you ask.
I think it’s more of a “we reserve the right to…” situation than anything else.
It for any large files. They want to limit bandwidth usage but not blanket limit everyone. One user using Plex of Jellyfin probably doesn't move the needle.
I'm not even talking about the copyright implications here, just the bandwidth costs. A single movie download would cost more than many hundreds of typical simple HTTP website sessions.
I would expect that a freemium service selling encrypted "zero trust" networking should have no idea what traffic is being pushed through my network making enforcement impossible.
Nobody's asking for a free lunch, but the reasonable thing to do would be to simply bandwidth limit freemium accounts across the board, not make exceptions for certain kinds of traffic in what should be a secure network.
(I can, in practice, serve small amounts of video through the free CDN, by using HLS; I'm also aware that this needs to stay below the radar)
I use IPv6 only Emby and Jellyfin servers with my friends. We all have native IPv6 and it works well for us.
In the mean time: the images in the article seem to be broken — produce 404 errors. Like this one: https://david.coffee/targets-config-screen.png
Then probably the hosting place is an easier target than a data center.
Given i work in Tmux, its super convenient to take a laptop with me and just use it as a thin client to my Desktop wherever I am.
- Most of the clients are open source probably.
- Tailscale allows you to run custom control server of your own.
- One open source control server "headscale" is sponsored by Tailscale themselves.
I don't see why I want to loop in a 3rd party to connect back to my house.
We use our Windows workstations as WSL SSH tunnels, protected with email verification (only for our domain), and it’s been working perfectly.
I’m curious, though, about how we can expose Docker services. It would be fantastic to have a remote build server set up with Cloudflare Tunnel.
It is an alternative to a traditional corporate VPN that addresses a few architectural issues; namely:
- L3 connectivity (which allows for lateral movement) to the corporate network. - Inbound exposure to the VPN gateway (scaling can become a challenge, not to mention continuous vulnerabilities from… certain vendors) - Policy management can get convoluted if you want to do micro-segmentation properly.
ZTNA is essentially an “inside-out” architecture and acts (kind of) like a L4 proxy. I’m going to butcher this explanation, but:
1. Company installs apps/VMs/containers throughout their network. These must have network reachability to the internal apps/services the company wants to make available to its users.
2. These apps/VMs/containers establish TLS tunnels back to the company’s tenant in the vendor’s cloud.
3. Company rolls out the vendor’s ZTNA client to user devices. This also establishes a TLS tunnel to the vendor’s cloud. Hence the vendor’s cloud is like a MitM gatekeeper.
4. Company creates policies in the vendor’s cloud that says “User A can access App X via app/VM/container Z”
5. Even if App X is on the same LAN segment as App Y, App Y is invisible to User A because connectivity to the internal apps happens at L4.
It is an interesting architecture. That being said, ZTNA solutions have their own issues as well (you can probably already spot some based on my explanation above!)
(Note: I worked for a security vendor that sold a ZTNA solution as part of their ~4-5 years ago. Things could be different now.)
https://github.com/alecbcs/hyprspace has penetrated every NAT I've ever encountered. No megacorporation required.
> CNAME homeassistant.mydomain.com a2f17e27-cd4d-4fcd-b02a-63839f57a96f.cfargotunnel.com
> Now all traffic going to this domain will go through the cloudflared tunnel, which is configured to route homeassistant.mydomain.com to 192.168.1.3. No Warp client needed, Argo tunnel does everything for us.
It boggles my mind that Cloudflare ever considered this acceptable for production, let alone that this is still how tunnels work. The whole configuration scheme feels like something that someone might have kludged up as a technology demo and launched in a staging environment. But the fact that a very security sensitive production system where a “DNS” record that looks like a CNAME to a magic hostname causes traffic to get proxied and sent to a “Zero Trust” private network is just … unreal. It’s almost impossible to tell WTF is going on or what policies apply to what. Does Cloudflare’s proxy really try to fetch an upstream resource, notice that the configured domain name ends with “cfargotunnel.com” and invoke some special handling? What happens if, say, someone else adds that same CNAME to their own network? What if some route goes to foo.bar.com and foo.bar.com’s nameserver reports a CNAME to cfargotunnel.com?
I’ve been using this product for several years, and the documentation and configuration pages have slowly evolved from abysmal to very slightly better. At least now it’s sort of clear how tunnels interact with strict TLS.
The only specially handling is cloud flare has a mapping from subdomain to your private network via it's agent and that's it.
I don't get what's the wrong or complicated about this.
1. I have a cloudflare domain with a working tunnel (managed through Access). In DNS Records, it shows as a CNAME to [redacted].cfargotunnel.com. But:
$ dig [redacted].cfargotunnel.com
; <<>> DiG 9.10.6 <<>> [redacted].cfargotunnel.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5851 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
and no records are returned. Interestingly, it's an empty result, no NXDOMAIN.
2. I have multiple subdomains that appear to be CNAMEs to the same [redacted].cfargotunnel.net. And yet they are entirely different sites that just happen to share an instance of cloudflared at the origin. The sites aren't even served at the same origin address!
They are different "Published Application Routes". They don't even have the same protocol!
2. The tunnel above is on a domain with "Full (strict)" TLS. But traffic to the origin emerges from cloudflared in cleartext.
This whole configuration schema is nonsense. What should happen if a CNAME points at a tunnel that doesn't have a route for that application? What if a tunnel has a route for an application that is CNAMEd somewhere else?
I imagine that what's going on is that Cloudflare internally has a rule that traffic with a cfargotunnel.com origin goes out their Tunnel infrastructure instead of out to the normal Internet. And Cloudflare applies the same JWT that it would apply if the request went out via the normal Internet, and cloudflared verifies that JWT if "Enforce Access JSON Web Token (JWT) validation" is on (maybe the request is literally TLS wrapped inside the cloudflared tunnel? I've never tried to inspect what's going on inside). And then cloudflared unwraps everything? And if you configure cloudflared wrong, then it's totally insecure?
However, home users live in IPv4 NAT world and look for solutions:
> Expose private services to the public, on public hostnames, no matter where they are running. You could even put your router running at 192.168.1.1 on the internet, accessible to everyone, no Warp client required
Trusting Cloudflare mitm 100% is a means to their goal.
Cloudflare network carries identity with traffic. If someone else adds the CNAME, they need an identity in the Zero Trust account that controls the tunnel. If you use the browser, Cloudflare IdP MITM the request and requires login to Cloudflare first. If you use Cloudflare Warp, then identity you use to login to Warp is injected.
>CNAME to a magic hostname causes traffic to get proxied and sent to a “Zero Trust” private network
That's also commonly called a load balancer.
From the Cloudflare UI, it works like:
- URL Normalization
- Redirect Rules
- URL Rewrites
- Page Rules
- Configuration Rules
- Origin Rules
- IP Access Rules
- DDoS protection
- Web Application Firewall
- Bots
- Rate Limiting
- Access
- Bulk Redirects
- Modify Request Header
- Cache Rules
- Snippets
- Cloud Connector
- Workers
- Custom Error Rules
- Modify Response Header
- Compression Rules
The "Access" step is key. Cloudflare acts like an authenticating reverse proxy. Once the request is authenticated, it continues processing and can route to the private backend over the Cloudflare tunnel.
Of course, you can make your app public. This is no different security wise than me adding a CNAME my-special-google.my-tld.com to google.com. Whether is works or not depends on the recipient server setup