Kubernetes ingress-nginx controller vulnerability
armosec.io
armosec.io
The only fix so far is removing the feature.
>According to the publication, multi-tenant environments where non-admin users have permissions to create Ingress objects are most affected by this issue.
Ouch! I guess this means lots of secret/password rotations for k8s admins coming soon. I've always wondered if there are public Kubernetes multi-tenant setups in the wild rather than just ones with admins and developer roles/namespaces.
The issue links to itself for "more details"…
(Also, yeah, it'd be nice if the URL of the OP had just been the Github issue. It is just as informative (or uninformative), and isn't an advertisement…)
I have war stories up the wazoo - at one point we had 16k random internet users on a single cluster!
The reason that an ingress controller need permission to read all secrets across all namespaces in the first place, is because ingress objects can specify their own certs (stored in namespaced secret objects) within their namespaces, hence the ingress controller needs permission to read those secrets and load the certs into the reverse proxy (in case of ingress-nginx, it is nginx). This is such a scary permission to grant that cluster admins have to hand over all the secrets in the cluster to the ingress controller and hope it does the right thing.
Now, ingress-nginx actually allows you to specify a default secret to be used by all ingresses it manages, and that would be the only secret it needs access to. I am not sure how many users out there are willing to dig into thousands lines of its manifest yaml files to tweak its rbac settings.
The upstream URL seems more clear as well: https://github.com/kubernetes/ingress-nginx/issues/7837
The issue contains a comment where they're also advertising their service, so I think that using the GitHub issue would be better, and would be a good compromise for them since they'll still have their ad.
Still, some of us on hackernews dislike advertising. I 100% agree with everything you've written in this comment, but as I mentioned in the comment you're replying to, the GitHub issue also contains an ad for the same tool - it just doesn't make it front and center, allows for any competitors to make their own comments on an open platform, and it feels less biased. Less advertisey.
Instead of "install our product", it could have been something like "run this thing specific to this exact vulnerability that does the absolute bare minimum. Oh, btw, if you liked that, you should check out our product that does much more, kubescape"
But as it stands now, it just feels different
(I also just learned that the word Kubernetes is in the default Chrome spell checker thing, which was sure helpful in writing this comment!)
there's definitely an attack path that gets the ingress-nginx service account token, which has list rights on secrets at a cluster level (so allowing for all secret values to be retrieved).
[0] https://github.com/kubernetes/ingress-nginx/issues/7837#issu...
I'm afraid not. HTTP GET on a collection endpoint (which is the operation represented by the list verb) returns the full object content.
https://kubernetes.io/docs/reference/access-authn-authz/auth...
To this point, handling of secrets in the Kubernetes ecosystem always seemed like a mess to me. Both `vault` and AWS SecretsManager/KMS in combination with IAM allow me to finely partition access to secrets by instance or developer identity whereas in the Kubernetes world one expects "secrets" to be handled by a small core group and machines seem to have mostly unfettered access.
Specifically[1],
>> Caution:
>> Kubernetes Secrets are, by default, stored unencrypted in the API server's underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd. Additionally, anyone who is authorized to create a Pod in a namespace can use that access to read any Secret in that namespace; this includes indirect access such as the ability to create a Deployment.
>> In order to safely use Secrets, take at least the following steps:
>> * Enable Encryption at Rest for Secrets[2].
>> * Enable or configure RBAC rules that restrict reading data in Secrets (including via indirect means).[3]
>> * Where appropriate, also use mechanisms such as RBAC to limit which principals are allowed to create new Secrets or replace existing ones.
When you dig deeper, both [2] and [3] look like they are someone's job security plan. No wonder it was developed at a place which employs people just to keep them off the market.
Especially since you really don't gain much unless you also have a KMS Provider[4].
No wonder most K8 setups I've seen still have BASE64 encoded secrets in some YAML file.
There is also no shortage of tutorial-spam -- so unless you happen to be lucky enough to apprentice with people who know what they are doing, it is hard to gain a solid understanding and move confidently.
[1]: https://kubernetes.io/docs/concepts/configuration/secret/
[2]: https://kubernetes.io/docs/tasks/administer-cluster/encrypt-...
[3]: https://kubernetes.io/docs/reference/access-authn-authz/auth...
[4]: https://kubernetes.io/docs/tasks/administer-cluster/kms-prov...
Companies / organizations / team members go rogue sometimes. We've seen this even recently with e.g. kicad, freenode, the timezone database, etc.
Just because you trust the source doesn't mean you should trust all of the scripts they tell you to run. Even if it's a good-faith script, you have no idea if it's making assumptions about your system that are not true and opening you up to sidechannel attacks and the like.
Curl to a file first, inspect the script, consider it within the context of your own system, then run it if you deem it's safe.
But you can still do that if you want? It is not like this is a hidden executable or something.
Huh? The whole point is to review the the code before you run it.
If it seems like many people _don't_, well frankly, that's their problem.
Encouraging bad practise is not the right direction.
Looking at the blog URL and header bar, and the script URL, this is pretty clearly a company blog recommending to use the company's own product. I hardly think that context counts as "unknown".
The article in question which _today_ looks all legit and points to a nice working script, might tomorrow be pointing to someone else's script that's a lot less legit.
And yes, in this imaginary scenario that wordpress install leads to a host of other problems for the company.
No need to make them your problems too though. ;)
With curl | bash, the URL of the bash script is right there. I can copy it, plug it into my browser, and inspect with ease the shell script that will run. If its on Github or a similar site, I can see in plain language the exact organization who published it (I wouldn't implicitly trust whoever "armosec" is, but I would trust, for example, "kubernetes", "microsoft", or "google" e.g.).
Alternatives?
- For something like this, to generate a report on whether you're affected? Maybe just a copy-paste bit of code? Isn't that the same thing, but less convenient?
- apt-get install something. Who published it? Does apt have security scanning (no lol)? Where's the source code for that package (scattered to the wind no doubt)? Does it have reproducible builds so I can be certain the code I eventually find is what I install (lol no. but bash scripts do!)
- Let's talk about `snap` for a second; Did you know that there's a `snap install aws-cli` package available [1], published by "Amazon Web Services", with a verified checkmark. Sounds great. Except, for years, no one on the actual aws-cli team had any idea where it came from [2]. They didn't authorize it, and they don't maintain it. It turns out, it was probably the result of a hackathon project by one AWS engineer. "Better security practices" for sure!
The only legitimate risk I've heard about 'curl | bash' is that web servers can detect whether the site is being accessed by a browser versus curl, and serve different content. Legit risky, good to educate about; don't curl | bash from sites you don't recognize. But this is a GitHub link!
I would agree that package managers are superior from a security angle, if most package managers had strong security review processes, on both the package content and the publisher. The problem is: they don't. They're all lacking in this regard, from apt to flatpak to the Apple App Store. The most sophisticated is probably the Play Store, but binary command line distribution channels are nowhere close to having reproducible builds, verified Real ID developers, static analysis on distributed binaries, etc. Package managers are literally just "curl | bash" with some steps added. By and large, anyone can publish anything under any name; educating people that they're "safe" is actually a massive disservice.
If the source is some shady site, it's possible to serve two different versions: one for curl/wget users and one for browsers.
This is the reason why you shouldn't do curl ... | bash, you should do bash -c "$(curl ...)"
The former is truly vulnerable, for the exact reason you mention, while the latter isn't.
> With curl | bash, the URL of the bash script is right there. I can copy it, plug it into my browser, and inspect with ease the shell script that will run
Not exactly, see http://thejh.net/misc/website-terminal-copy-paste
More savvy users might have set their shell up in a way that mitigates this risk, and some users will, as you mentioned, copy and paste URL, but go ahead and try that with the command in the link I provided - note that if you copy just the git URL, it only copies that bit, and then you can paste it into your browser, and not realize that there's a malicious bit before it
Of course, copying and pasting a command involving apt/dnf/etc is also vulnerable to that same attack :)
It does mitigate a different issue whereby the website can detect if you're piping the output of 'curl' directly to bash.
The safest option is definitely --> curl to file, audit what you downloaded locally, not in a browser, then bash -c that downloaded file.
But, again, this only really mitigates attacks from sketchy websites. If the website is trustworthy; both the website you're getting the curl | bash download string from, and the website the script is hosted at, such as Github, where 90% of these scripts are hosted; its not really mitigating a legitimate threat.
The copy-paste thing is interesting, but frankly: That's an argument for why the web is insecure, and that insecurity impacts every single method of software distribution which touches it. Copy-pasting a 'sudo apt-get install' line is the same class of risk here as copy-pasting a 'curl | bash' or 'bash -c $(curl)' line; you could swap the script to a new URL, or you could have them install a malicious package from apt, or even just nuke the apt-get install part and just do a curl | bash anyway. Definitely something that's great to know and educate about, but its not the strongest argument against curl | bash.
> 'bash -c $(curl )' doesn't mitigate a website's ability to detect whether you're downloading a file via curl versus accessing it in a browser
You're totally right, I forgot about this one! It's wild how many tricks there are in the security world
.... oh man
Here is presumably a security company, in their own blog post outlining a high-severity vulnerability alert, advocating that you run untrusted code from some rando GitHub repo. It also outlines my fear about dealing with some "security consultants", because on the one hand they outline all of the things you need to "box check" for some security audit, but at the same time leave you less secure because they've just opened another giant gaping attack surface for your company - this is exactly what happened with the SolarWinds attack.