checksum.sh
I started messing around with a way to verify the checksum of scripts before I execute them. I've found it a really useful tool for installing things like Rust or Deno.
It's written entirely as a shell script, and it's easy to read and understand what's happening.
I hope it may be useful to someone else!
1. Never use echo to output untrusted content as the first argument
Let's say `s='-e 1\n2'`, then `echo $s` will output:
> 1
> 2
Instead of:
> -e 1\n2
Always use printf if you want to start output with untrusted content, e.g., `printf %s\\n "$s"`.
2. Never use unquoted variable expansion when trying to exactly reproduce contents of the variable
Similarly, unquoted variable expansion re-tokenizes the contents and will not preserve spaces appropriately. Say `s='"a<space><space>b"'` (where each <space> is a literal ' ', HN seems to be collapsing 2 spaces down to 1), then `echo $s` will output:
> "a<space>b"
Instead of:
> "a<space><space>b"
You can get the latter with `echo "$s"` but use `printf %s\\n "$s"` to fix both issues.
PS: If you fail to use quoted expansion with printf, for example like so, `printf %s\\n $s`, then you'll notice the problem right away, as it will effectively turn that into `for i in $s ; do printf %s\\n "$i" ; done`. That's actually a very useful feature of printf if you know to use it.
Edit: These problems exist for bash/POSIX sh at least. Perhaps you're using a shell that works differently, like zsh, because otherwise issue 2 would probably have led to some checksum fails for you already.
Re --check, I suppose the way to do that would be to download the file to disk, which --check requires as fair as I can tell. So I could download the file to disk, --check, and then remove it. I think most of these installs scripts are trying not to leave any artifacts around from install, other than the resulting binary.
> $ s='1<space><space>2'
> $ printf %s\\n "$s" | shasum -a 256 > tmp.sum
> $ printf %s\\n "$s" | shasum --check tmp.sum
> -: OK
So you can just `printf '%s<space><space>-\n' "$c" > tmp.sum` and check with `printf %s\\n "$s" | shasum --check --status tmp.sum || { echo "checksum failed" > &2 ; exit 1 ; }`
Having to create temp files is a wrinkle (could probably avoid it by using process substitution if you want to give up on POSIX sh), but so is writing bash scripts in general.
I ended up trying with process substitution so no tmp file.
It works. Trying to decide if it’s more difficult to read
So just put double quotes around all your variable expansions unless you know you shouldn't -- 90% of scripts would be "fixed" with just that. And don't bother putting curly braces into the variable expansion unless you know you need to. People tend to think `echo ${s}` is somehow better than `echo $s` when it's exactly the same -- the curly braces are just a way to allow you to, e.g., write `"${s}_"` as distinct from `"${s_}"`. AFAIK in fish `${s}` is identical to `"$s"`, but that's a different kettle of sh.
That said, to write good shell scripts requires actually learning the shell paradigm, instead of just trying to write "Python with Shell syntax".
In other words: you can be confident in the bootstrapping script you've just downloaded because it passed its checksum, but that script is just going to download more binaries from the Internet.
Not necessarily. A number of these scripts either configure a package manager or the shell script contains the binary itself which is unpacked when the script is run.
Even if there's a shar-style[1] packed binary in the script, you have no idea what that binary does when you verify that the checksum is correct.
> you have no idea what that binary does when you verify that the checksum is correct.
This isn't any different from using a package manager. You're still downloading a binary that could do anything and you have to have some level of trust in the source.
I'm old enough to remember when apt packaging got burned because it used http instead of https even though apt packages get signed.
If you're downloading software from websites protected with HTTPS, and that's good enough for you, then downloading and executing a script from those same websites using HTTPS is also good enough.
Would it be better if those things were signed with a key for which there is a code signing certificate? Eh, maybe, yes, if the PKI for the code signing is sufficiently better than WebPKI, which... is not necessarily obvious. Meanwhile, access to that PKI is probably sufficiently harder to come by than WebPKI TLS server certificates that a lot of people don't bother, and rightly so.
Now suppose you say "I don't trust this, I'm just going to clone their github repo and build from source". Do you get more protection that way? Maybe, maybe not.
Now, if you get packages from Debian and the like, you get them signed, and maybe the person who contributed the package to their repository did a thorough code review and audit of the upstream they are packaging, or maybe not, who knows.
This is why containerizing this stuff helps. But it's not really accessible to people yet.
What might be nice is that any program that a user executes automatically gets some level of isolation corresponding to how it was delivered, authored by whom, etc. So programs from the OS get the least isolation, and programs written by the user less isolation, and programs of unknown provenance get the most isolation.
s=$(curl -fsSL $1)
...
c=$(echo $s | shasum | awk '{print $1}')
It appears to work because the author calculated the checksums with the same script and is just validating that they are not changing.
In other words, it's possible to make whitespace changes such that the hash won't change.
Here are two scripts: a harmless one and a malicious one, which produce the same whitespace-ignorant SHA256:
$ foo='# this is a comment
> # rm -rf /'
$ echo $foo | sha256sum
8b87547d4d214038b153ce57d929be4c835b7690c930c1e83a25fc1509390cf9 -
$ foo='# this is a comment #
> rm -rf /'
$ echo $foo | sha256sum
8b87547d4d214038b153ce57d929be4c835b7690c930c1e83a25fc1509390cf9 -
(I know about GNU Coreutils' safeguard in rm against removing / recursively, by the way.)
If the script is deliberately malicious as originally published, then the publisher will provide a valid checksum; so it doesn't help.
If the script source is subverted by an attacker, then it only helps if the attacker doesn't also have the means to change the published checksum too.
If an attacker can modify the site which publishes the URL for the script and the checksum, they can modify both at the same time.
is that much different than just storing the verified copies of the scripts?
However I'm still not sure it really makes sense. Do you also manually review the code of the binaries that the bash scripts download?
For Rust you can ignore sh.rustup.rs and just download and set up rustup manually.
CARGO_HOME="${CARGO_HOME:-$HOME/.cargo}"
mkdir -p "$CARGO_HOME/bin"
curl -Lo "$CARGO_HOME/bin/rustup" 'https://static.rust-lang.org/rustup/dist/x86_64-unknown-linux-gnu/rustup-init'
chmod +x "$CARGO_HOME/bin/rustup"
hash -r
rustup set auto-self-update disable
rustup set profile minimal
rustup default stable
rustup update --force
rustup self update # Create hardlinks under $CARGO_HOME/bin/The only difference here is that you’re running a few commands by hand instead of running them in a single invocation of a shell script.
checksum() {
hash="$1"
file="$2"
sha256sum -c <<< "${hash} ${file}"
}
checksum() {
hash="$1"
file="$2"
printf '%s %s' "$hash" "$file" | sha256sum -c
}
I also think it's weird to use `alias` inside a function, instead of just using a parameter to store the name of the program to execute.
I'll work through these suggestions. Appreciate it. Feel free to send a PR if you want.
For the here string I think that won't work because the file isn't being saved locally, it's just being piped (so $2 is a URL). I can't do the usual `shasum -c <<< "132e320edb0027470bfd836af8dadf174e4fee00 install.sh" which takes a local filename but not the file content. As far as I could tell anyway. I'll try it some more
Whatever you are doing to protect sending the checksum can also be used for protecting the script itself.
I download the script from A, and the checksum from B. And then I verify them locally. So A and B both need to be compromised. It all assumes the script was safe to begin with, and this just verifies that nothing has changed
Checksum.sh could keep track of checksums. Then an attacker has to alter the original script and checksum.sh.
“and now you have two problems.” —jwz
We haven’t been able to trust public pgp keyservers for a decade or more (possibly never, really).
So now we’re back at having to trust where-ever we get the proof from, whether that’s the file hash, or the public key.
(Which, as you say, is what package managers provide, and if you don’t trust your system’s apt/yum/pacman/whatever, then you have a bigger problem that trusting any random install shell script)
That's also kind of the issue with a lot of these shell injection attacks. Sure someone could insert environment variables or other shenanigans to take over your machine, but if they have that much control over your shell there are countless other ways they could also do it. Guarding against this one particular case doesn't buy you much.
checksum https://sh.rustup.rs/#8327fa6ce106d2a387fdd02cb95c3607193c2edb | sh
But I didn't but a fancy domain name :-)
The hash only verifies file integrity, and that the content of the url doesn't switch the script later. But keep in mind in most scenerios, and attacker would also just change the hash listed too (they're usually on the same website). This only mitigates one very specific attack.
Why don't we use GPG here? That way we can verify ownership and file integrity with at minimum TOFU, plus optional manual verification? If we're going through the work of adding a wrapper and all that, we may as well no?
This has the benefit that you only need to import the owner's cert once, all future changes have the same cert. Where hashes are obviously different every time, you have to trust the source of the hash every time it changes. With GPG at the very least you have TOFU with certs - and very best can have better assurance of the initial download too.
EDIT: Just want to clarify - I'm openly asking why the "developer community" is going the direction of hashes for script verification vs GPG signatures.
I don't mean to diminish your project, your project looks fun, and does make verifying hashes easier :)
With GPG the developer has a key pair (one private, one public). They can then sign all their scripts with their private key and publish the public one wherever. You can then take that public key and verify that the script has been indeed signed by the developers private key.
This is the overview:
Developer generates a private/public key they use for all of their projects.
You import their public key once - you can verify this from their github, twitter, etc but that's optional.
They can sign a file with their key. You can check this signature against their public key. This will guarantee the file was signed by using that key and is unmodified.
If someone hijacks the website after this point and signs the new downloads with their own key - then you will be able to see it's invalid.
If you manually verify the key then you'll know your initial download is valid - if you trust on first use then you at least know all future files signed from that developer with that cert are valid.
They also are effectively a hash for file integrity.
tl;dr - hashes tell you if a file is changed. Signatures tell you if the file is changed, and who the person that made the file is.
A GPG sig proves that the file was signed & uploaded by the author, which defacto doubles as proof that it's the same file. The idea here is that the author uploads their public key, signs the package with their private key, and now there's an association between the package and the author. An attacker would have to obtain the author's private key, or replace the public key with their own. Changing the public key, however, is a big red flag.
I couldn’t care less if the Chinese government hosts an install script, if there’s no possibility they could have changed a single byte of the script.
Assuming I have a trusted way of knowing the installer script hash (which is a big assumption), I don’t need authentication for the script download, I only need integrity checks.
In my project I "host" the hash on a different medium, so in order to compromise the file download the attacker would have to compromise both the file hosting server and the hash hosting medium (which in my case is GitHub).
I also don't really display the hashes, as the download only happens when the script is updated, so your current version of the script will check the hash on GitHub vs the hash of the file download from the file hosting server.
EDIT: To be clear, this doesn't solve the problem with the initial install and it is also not related to the Checksum.sh script.
Does the script get the new version url&expected hash from the website alone? Or does it get the expected hash from the website, then calculate the URL from github?
Basically I'm wondering if that prevents just needing to attack the website - if the url to download the update and the expected hash are in the same place then it's still a single point of failure.
The version number and latest file hash are also fixed URLs, stored on GitHub.
So for an update, the script checks GitHub for latest version number, if newer it downloads the latest version from my server, computes the hash and compares it to the hash stored on the fixed GitHub URL before proceeding.
I think there's no way to replace the file with a malicious one that will be distributed to the users unless you get access to both my server and the GitHub repository.
It does have the downside still that changes to the website/github might break future updates in a way that isn't (easily) verifiable.
While this is a solution personally I still like the idea of GPG more since it'll work for any new files, works for your new projects automagically, etc.
But I think you did at least fix the future update problem with auto-updates, which is a lot more work then most people put into it so thank you for addressing the issue!
Thanks for sharing this work OP! I didn't see a license mentioned -- did you intend this to go into the public domain? I like how you set up a cool domain name and did some sick graphics, but I'm not sure how I can legally use your code in the future.
That being said, I appreciate the work you put into this project.
I'm not going to list off specific examples, but MANY open source projects serve either PGP keys or hashes in the clear. Or they serve just hashes over HTTPS and now you have a trust issue.
Or, in one case, my favorite -- they had lovingly listed out the MD5 sum for the program... but they served both that checksum, and the code itself... over HTTPS.
Now, to be fair, HTTPS does provide an integrity check, so there's a benefit beyond privacy or whatever but... this is a RAMPANT problem in the open source community.
I ran into it mostly when trying to find esoteric security tools when I was attempting OSCP and interviewing around for penetration testing roles.
I got the sense rapidly shifting from "I was so scared of the CFAA I did an entire master's thesis on the design of censorship circumvention tools" to "Oh gee, I used to be such a narcissis, demanding a high falutin salary when I couldmn't even fire up Metasploit to wipe a server."
(The implication being that some folks abused their access when my powers were week, and now, in time for spooky season, it's time lean in to letting people take whatever drug they want if they feel scared -- reality scares me too some days.)