Tailscale Kubernetes Operator | Hacker News Reader

Tailscale Kubernetes Operator

tailscale.com

157 pointsl2dy50 comments

The amount of trust placed in DockerHub to serve the correct images to everyone for uncritical download and execution is insane to me.

The whole industry does it, and it's the same as "curl | bash" to specify image:tag and not image@hash.

If I were TAO I'd apply for a job at Docker or their hosting provider.

That's why you only pull official images and signed ones.

And that's why I have an in-between step.

Harbor.io allows you to configure it as a proxy with approval mechanism and cve scanning

Docker content trust (ie signature checking) is disabled by default.

We won't even do this for webpages, but we find it a fine default for code that executes inside critical infrastructure.

It's utter madness. Cool to see someone is doing something about it.

We have collectively decided as an industry to place infinite trust in a handful of vendors for the sake of convenience.

Auth providers (OIDC), cloud providers, and numerous software repositories and SaaS providers effectively have root on the entire universe. A major compromise of any number of these large vendors could expose millions of systems to total compromise.

Resultantly, these systems are almost certainly already compromised by organizations skilled enough to not get caught.

I'd say nation states, which means the US and possibly other nation states effectively have root on virtually all SaaS systems.

It's exactly the architecture I'd promote if I wanted a total panopticon, but I'm not suggesting a conspiracy. Ease of use is the most powerful force in computing and history has shown that people will trade privacy, security, freedom, cost, and virtually anything else for it.

It does make a certain amount of sense. Time is non-fungible; no amount of money can buy more of it. So ease of use by saving time is extremely valuable and commands a high price.

The problem is that IMO the cost of all this SaaS automation is higher than most people understand. There's some rather huge hidden costs here.

What's TAO?

I am not sure but maybe referring to this? https://en.wikipedia.org/wiki/Tailored_Access_Operations

Might be referring to “Tailored Access Operations” which was an NSA office

You could say this about any service serving software artifacts?

Unsigned ones, yes.

That just moves the trust requirement up the chain, and assumes that the image wasn’t compromised pre-signing (either intentionally or unintentionally)

It doesn't, it removes/reduces one step, and leaves another one that is there regardless.

What's the alternative? I don't know of any reason to think that tarballs or <os> installers are any better. I suppose I could clone the code and look for security flaws myself but I'm no expert and on something like nginx it's certain to be a waste of my time.

OCI registries.

Harbor + Notary + admission controllers - AKA private image repository with image signing.

Sigstore. Another method for signing & verifying artifacts.

> it's the same as "curl | bash"

While I understand the argument you're making, the exposure of running "curl | bash" on your local machine is much higher than running arbitrary code inside of a container.

Even if you specify a hash, are you actually checking all of the code and binaries in that image?

What about the image base (e.g. Alpine, Debian) and their packages?

There have been a lot of container escape exploits

Sure... but it's still not "the same as" running on your host.

That depends on how you're running it. You wouldn't want a compromised image to be the base for your build containers, which get access to your product's source code via volume mount, or to secret keys via environment variables.

I'm a big fan of Nix Flakes but they appear to have the same issue here as "curl | bash".

I mention this because Flakes can do much of what Docker can do, but do not use containers or virtualization.

The official NixPkgs are carefully vetted. Nix Flakes make it more convenient to pull in sources from outside NixPkgs.

As Nix Flakes become more prevalent, perhaps there needs to be some way to containerize them. There are more benefits to Nix Flakes than native execution so containerizing them somehow might be an acceptable compromise.

> As Nix Flakes become more prevalent, perhaps there needs to be some way to containerize them.

You may be interested in nix snapshotter, which can run nix packages in containerd.

https://news.ycombinator.com/item?id=37407758

One thing that i think is really missing is redundancy on the proxies

Currently there's no way to have two proxies that listen on the same tailscale hostname/ip

Ideally in Kubernetes every pod is redundant to allow downscaling of nodes efficiently, so this means we have to eat a minute or so of downtime randomly every now and then on our tailscale endpoints

Can't you expose the services to two load balancers?

Not ideal for anything customer facing but fine enough for staff.

they would have to have two separate hostnames/ips so you wouldn't have a single redundant endpoint

I agree, it's a feature that I find sorely lacking in my tailnet.

These are the relevant Github issues to follow, hopefully they address these someday:

https://github.com/tailscale/tailscale/issues/465 https://github.com/tailscale/tailscale/issues/4324

That's really neat. Cloudflare tunnel for external customer egress, and Tailscale for internal tool egress. No more costly cloud specific load balancers !

You can use Cloudflare access for internal which is tunnel + identity access management for end users.

I'de rather have full network isolation for internal stuff like admin portals. Plus, I already use Tailscale to sync DBs between regions and clouds.

My only issue with Tailscale was that it can't seem to stay logged in longer than something like 45 or 90 days.. making it a fun toy, but not for enterprise use.

As someone who travels a lot with machines all over the world, if a node goes offline I can ask someone to reboot a machine .. but there is no way I am giving random people credentials to my machines and network to fix issues.

There is an option to disable key expiry in the machine settings, unless you're talking about a different issue / bug. In my case, simply turning off key expiry is enough to keep the machine online for months inside tailscale network so far.

Thank you, I will take a look for this and give it a try.

Machines can have unlimited expiry

API keys have 90 day expiry but you can get around that with an oauth app that has credentials that don't expire

Hi Tailscale engineer here. The operator also supports Tailscale Funnel so you could use that instead of Cloudflare tunnel if you desired.

Hey ! I see Tailscale Funnels as maybe a good replacement for Ngrok, but not for Cloudflare Tunnels.

Your Funnels are in Beta, MUST use your tailnet’s domain name, have bandwitdh limits, no failover and no load balancing. If my website goes down, I close shop. Cloudflare Tunnels are just way more mature for production loads. CF Tunnels technically don't have load balancing, but if you set multiple Tunnels with the same ID, you get some sort of load balancing AND failovers if a tunnel goes down. And after that, they have a paid Load Balancer option.

Even for internal admin portals, the mention that "Traffic over Funnel is subject to bandwidth limits." with absolutely no defined numbers is just a turn off. If you added a number to that, like a limit of MBPS or GB/Month of transfer, it would be something I can bring to my colleagues, something we can discuss and weight on. For now, with no number, it's just a threat.

Everything else about Tailscale is chefskiss tho ;)

Does it support headscale? https://github.com/juanfont/headscale

We've been using https://github.com/mvisonneau/docker-tailscale/ on k8s clusters. Good to see an official option coming.

Same however I'm using HashiCorp Nomad so hope for an official option there.

Maybe not wise to post here yet as the docs mark this as a private alpha.

Maybe so, as what we're doing in absence of proper docs is rightly classified as something worse than alpha. The best docs they had so far up to this point:

https://tailscale.com/kb/1185/kubernetes/

These are great, for someone who knows Kubernetes. You should understand that creating the tailscale subnet router as a pod directly means the connection is not resilient. It's also key to understand that tailscale will break if you have more than one instance of subnet router at a time, so substituting Deployment in place of where these docs use Pod is not a really good choice without some fine tuning because of the risk that a rolling update creates another copy of the pod before the old one has shut down.

Maybe stateful set, if there was a way to permanently imply that statefulset can only have one replica. I appreciate the link anyway. I figured all this out on my own, and I'm using tailscale productively with Kubernetes based on the old docs, with my open source project. Tailscale has a great and generous free software tier for supporting OSS maintainers. :tada:

I think an alternative solution would be nice, where services are just registered with their service discovery, though I suppose that would require them providing an official API for that part of their product.

In my current setup for Tailscale + Kubernetes, I just use their subnet router[0] and add the kubedns server for the cluster.local domains to their MagicDNS. Having proper service discovery would make this into a nice round solutionl.

[0]: https://tailscale.com/kb/1185/kubernetes/#subnet-router

It's a neat idea but I wouldn't put this in my k8s cluster.

Keep it simple st*pid!

You can use, instead:

Keep it simple, smarty

Keep it super simple.

Is "stupid" really one of the words that the Ministry of Truth no longer permits us to use?

They felt the need to censor it as st*pid so I merely offered the alternative I use.

It's just rude

1984 is when people dont like being called stupid

k8s cluster and simple are rarely found together.

Love it!!! This is going to make (my) dev and testing env cluster so much easier.

Now imagine running derp-server with in the DC with your k8s.

Nice! I think I'll try to implement this in my TrueNAS this weekend, as it makes way way easier to access all the services I deploy there.

TrueCharts charts have Tailscale support, but not all my charts are from there, and also I'm kind of avoiding them.

Also, it's nice if you deploy something without a chart.

Reply on news.ycombinator.com