Helm local code execution via a malicious chart
github.com
github.com
1. Attacker crafts a malicious Chart.yaml containing arbitrary code
2. Replaces Chart.lock with a symlink pointing to a sensitive file (like .bashrc or other startup scripts)
3. When you run helm dependency update, Helm processes the malicious Chart.yaml and writes the payload to whatever file the symlink targets
4. Code executes when the targeted file is next used (e.g., opening a new shell)
Why This Works: Helm follows the symlink during the dependency update process without validating the target, allowing arbitrary file writes outside the intended chart directory.
As I'm typing this it's occurring to me that you probably shouldn't be able to do that. The fix they applied was to prevent the actual write from occurring when trying to write the lockfile and determining that the lockfile is a symlink. They could (should?) also validate that like, the package itself hasn't been screwed with in this manner.
This doesn't affect things like installing or upgrading a chart. Dependencies aren't updated at that time.
True enough, but if you have a victim unpacking and building untrusted tarballs there's no security boundary being crossed, is there? You don't have to bother with this symlink nonsense, just update the install script to include your payload directly.
Honestly this vulnerability is dumb. I don't see any realistic scenario where it can be exploited by an unprivileged attacker.
It doesn't matter whether it's "from a repo". If you can't trust the repo it can feed you whatever it wants.
(Those templates, once rendered, might then refer to pods, etc. that might be put into a k8s cluster (or perhaps we merely render then YAML, and never `apply` it), and in that sense, one might imagine that that is an install, but that's not the security boundary being crossed here; this would presumably result in execution on the host running helm, which would definitely be surprising.)
Certainly, it would be better to trust the upstream completely, but let's not kid ourselves? See the entire current state of software supply chain in the industry.
But when I visit a website, I don't expect the website to LCE me. Why should turning a YAML adlib into YAML LCE me, regardless of the trust of the upstream. This is not a privilege I'm expecting to give the upstream ever, and this behavior is a clear security bug, to me…
Git just moves symlinks across systems as is, so yes, you can use git to deploy the exploit.
This is almost becoming a joke at this point, "assuming an attacker has access to the system, they can change things on the system".
If you are new to helm or haven't considered the security around it, it is good to know what to look out for.
You download charts either as a tarball from a helm repo or oci registry with helm and helm will create the files and links with your permissions, and send me whatever I wanted to extract from your system.
Yes, you should check things you download from the internet. But also, that is not how a chart is supposed to work.
This has nothing to do with whether you are running it in sudo or whatever. (and in fact on MacOs, I don't believe this requires running it with sudo permissions to overwrite ~/.zshrc for example)
Sorry, just can't really recover from trauma of counting spaces and messing up newlines, etc. when writing Helm templates. You know, Lisp "sucks" because "you need to count parenthesis" (you actually don't), yet Helm is a widely accepted technology where you need to count spaces for (n)indent ;)
The alternative here is something that manipulates the data structure directly. E.g., it might permit me to say:
my_config_map.data["key"] = some_string_value
And then at the end, the thing itself takes care of converting the resulting objects to YAML, thus preventing me from inadvertently turning what is meant to be a string into something like an accidental YAML-injection that results in terrible errors because I miscounted the number of spaces to indent something.
however, this is usually true with working with helm in general if you are using charts other people maintain. That’s one of the strengths of helm. you just shove your values into the chart and it should work. Maintaining charts is not fun though which is why I wrote the wrapper for my purposes.
Even the helm infrastructure that I work in is completely wrapped in custom shell scripts that call all sorts of other commands to populate helm variables.
But yeah it's silly that helm templates require all sorts of {{ indent | 4 }} type incantations when the final YAML output is just sent through some kind of toJSON anyway.
When that's the case, bash if often the better choice, especially if you know it well. It has an excellent REPL, is easy to trace and is already installed everywhere.
That's a totally fine trade off for actual sane array/list functionality, robust string manipulation etc. I'd rather form shell commands in a programming language than in bash. People seem to love it.
I do not think for a second though that the average person that "knows bash well" can read and comprehend a multi hundred line bash script written by someone else as fast or even correctly has a seasoned python dev reading python written by someone else.
OK, but, you know... those tools were created by literal devs. Not in yaml, in a "real language"! So apparently devs thought they needed that.
The argument should be towards all people - we love creating new abstractions and "simplify things", but we suck at honest evaluation of the impact of our creations.
Still: I absolutely hate Helm templating and think that the very existence of "helpers" (even in a default chart!) is an abomination.
All `kubectl` does is create REST API calls to the control plane. So to be fair, what I'm grousing about can be accomplished just fine by a developer like me constructing API calls from python to update objects in k8s.
The problem is I work in devops where tooling written in proper languages with standard libraries that have things like useful arrays or robust string manipulation or ergonomic concurrency is a non-starter for some reason. The argument against that being mostly "I have to install the interpereter first".
Many orgs and many teams allow that. That's why alternatives to Helm exist like cdk8s, or Pulumi for rest of infra.
For Kubernetes there is such a variety of tools, it's overwhelming. The only problem to tackle is that Helm/Argo is so popular, that alternatives are hard to sell regardless of real benefits.
Look at how big is a scene of Operators for k8s - this is where the programmable part of k8s went. You configure operators with their own CRDs (usually) that can be static/plain declarative text. Then you write actual code of the operator that deals with all the complexity of enforcing that state. For me this is where typically what you are talking about goes, while high level "non programmable" code sits in yaml in a other repo for others to maintain. Maybe this is where you should also go, standardize your work for others to consume in a form of operator?
I fully get neglect in adopting more complex tooling. sh, curl, sed, awk... those things are present almost everywhere and it's not that hard to get lucky and make a script that will run on almost everything your org has. And it actually might be fine for a decade or so.
I myself could not do code (even scripting) because one of my companies literally treated scripting in anything as development which was strictly regulated (so forbidden for any non dedicated dev role). Or an org that had not a single server in whole DC that has Python 3, years after it's release. Or more recent: some damn Ubuntu LTS that can't be easily upgraded to just 2-3 minor versions that this cool k8s library uses. Maintaining python versions on VMs is a pain in the ass, especially if your org has strict controls. Internet access to pip is not granted as well. UV gets the job done nowadays, if it's allowed, but long story short: that "fear of real language" can be as much lack of knowledge/skills as pragmatism that came from painful experiences.
Though it's lacking in several ways, like good destroy functionality.
kustomize build | kapp deploy -a my-app -y -f -- You have access to my file system
- You have access to the helm repository
You place malicious binaries outside the helm directory. Helm will now execute malicious code through the helm chart pointing outside the helm directory.
Don't I have already bigger problems if you have access to my file system to place there malicious code?
Is the danger here that one can get an execute permission? But if you can manipulate my helm chart why can you not also place the malicious code in the helm directory?
No, helm is the one doing this part in the vuln. Chart.lock is made a symlink to some important file, and helm will happily write to it.
If you can manipulate my helm chart, why not just do the RCE directly in my kubernetes cluster or whatever?
Is the vulnerability that you ship a chart with `Chart.lock -> ../.bashrc`, and then helm writes to `Chart.lock`?
Why is the fix specific to Chart.lock (https://github.com/helm/helm/commit/76fdba4c8c2a4829a6b7abb4...), wouldn't the fix be instead that "A chart cannot contain any symlinks outside of its root"?
I agree that it's not clearly explained why this isn't a concern though. A cursory search for other instances of os.WriteFile doesn't seem to surface any thorough controls...
edit: ok actually it looks like the lockfile is special because it's the only instance of helm itself directly writing a file on behalf of a package consumer
If you have a chart that has `deploy.yaml` symlinked to `/home/john/testcharts/redis/deploy.yaml`, that chart is clearly not going to work on anyone's machine except john's, so that chart is useless on anyone else's machine.
If you're saying the use-case is for charts that aren't distributed, well, I'm saying we should ban all symlinks on distribution (downloading and unpacking a chart should fail if it has symlinks outside of the root), and I just can't imagine any use-case where a distributed chart with external symlinks makes sense.
If this whole thing is about charts that aren't distributed, but local to some developer's machine, well, in that case who cares if the developer can pwn themselves by typing "ln -s ~/.bashrc Chart.lock", they could have just pwned themselves by typing "bash" even more quickly.
https://helm.sh/blog/2019-10-30-helm-symlink-security-notice...
Smattering an --allow-symlinks flag all over their commands seems to be the least inelegant way to handle this while still giving users an easy way to maintain compatibility. Maybe they'll come around to it after this.
i.e. you have 3 different charts that all depends on `cache`, `load balancer` and `database` charts and you want to only ever have 1 version deployed of those subcharts so you want the parent chart locks linked
Basic tech news?
Capitalist news?
Vulture Capitalist news?
Given the details in the article, I think even something as simple a templating a chart from a repository might be vuln., but it likely depends on a lot of exact specifics.
> Where are the security boundaries?
I expect templating does not result in LCE.
> How does the attacker gets their repository with a symlink in it to the victim?
The attacker owns the repository. They can serve whatever maliciousness in it they want. But should templating a malicious chart result in LCE?
> Is Helm typically run as a privileged user?
Enough so, yes, because the rendered result is often pushed to a k8s cluster. "Privileged" here might not be "root", but it might be "this user has k8s API access".
Imagine, e.g., that the attacker's LCE here might be "push ~/.kube to attacker".
> And why doesn't the vulnerability description give answers to these questions?
Familiarity with the tools involved is an normal assumption.
Fortunately, my dotfiles are managed with nix so trying to write to those files on a read only partition will raise many red flags for me.
I don't use bash, but maybe should write a dummy .bashrc (and other start up script equivalents for fish) as some sort of canary.
If I happen to overlook the malicious shell script crafted in a dependency on helm chart, I would get nasty errors that a process was trying to write to a read only file.
Allowing LLMs to generate charts and what not via shell execution is a bad idea.
Although, the whole can of worms regarding synth/audio exploits is a pretty wild scene ..
This is not the first RCE involving YAML and it won't be the last.
The reason YAML was popularized is because it was a response to XML which isn't user friendly to write. It's unfortunate that the spec got so convoluted, and uses a lot of implicit behavior, but I'd rather write YAML than XML, JSON or TOML for things like configuration files. Nowadays there might be better alternatives, but YAML is the de facto standard.
It's also unfortunate that YAML got abused by people who wanted to turn it into a DSL, so we ended up with thousands of lines of Ansible playbooks, CI workflows, and Helm charts, but here we are.
So that leaves scientific notation.
"\ud83d\udca9"
Rust's `serde_yaml` will fail on this.
I don't know about other parsers, but I'd be curious to.
The standard itself isn't well written here, IMO.
> The content of a scalar node is an opaque datum that can be presented as a series of zero or more Unicode characters.
The example here is a "quoted scalar", which can contain the escapes you see. Those escapes represent "Unicode characters", specifically,
> Escaped 16-bit Unicode character.
But "Unicode characters" is never defined by YAML.
Most implementation seem to treat them as Unicode code points, and so thus the resulting string type in almost all cases in something like [UnicodeCodePoint]; in Rust, that means no unpaired surrogates, or we can't convert it to a Rust `String`, which is roughly speaking `[USV]`. In Python, that's workable, since that's Python's `str` datatype, but that means no surrogate decoding occurs.
The grammar also further implies that it's [UnicodeCodePoint] and not [USV], and the prose never restricts unpaired surrogates. (The JSON standard strongly implies the UTF-16 decoding should happen on escaped values, though it too waffles around unpaired surrogates. Whether unpaired surrogates are accepted is variable in JSON.)
But compare with a JSON string: a JSON string decodes to a something like a [USV], so surrogate pairs are decoded to their corresponding USV.
Go doesn't use tabs or whitespace as a part of its syntax. It's a part of the formatting, but not the syntax of the language.
Python on the other hand, one extra tab or whitespace can cause havoc.
> Showing 1 - 25 of 6,749 results for XML
Searching for YAML:
> Showing 1 - 25 of 288 results for YAML
But glad you vented, I guess.
If so, then I agree on blaming this on YAML.
Also this vuln has nothing to do with YAML
https://www.sciencedirect.com/science/article/pii/S136984781...
I’d still wear one, but also try to be more careful knowing that the helmet provides a false sense of security.
I do believe the analogy holds very true with programming habits.
> this systematic review found little to no support for the hypothesis bicycle helmet use is associated with engaging in risky behaviour.