Curl to shell isn’t so bad (2019) | Hacker News Reader

Curl to shell isn’t so bad (2019)

arp242.net

40 pointsmaple314280 comments

This gained traction a few month ago. Previous discussion ===

https://news.ycombinator.com/item?id=21490151

Yeah, not sure this needs to be discussed again since it was just 80 days ago (and I say that as the story's author).

I mostly agree that curl to shell isn't terrible, but he misses the big negative security implication: the shell scripts are unsigned.

It is almost always far, far easier to hack into somebodies website and replace replace "good shell script" with "evil shell script" (or hack into someones GitHub account and make a commit evilifying a shell script) than it is to both do that and get a hold of the developers private key and sign the script. The signing key is usually far more protected and non-public than the web server is.

So no, I'm probably not going to review the entire install script either way, but I'm FAR more comfortable running an install script I found online that has been signed by a developer I trust rather than one that hasn't. Package managers and OS installers handle this for you and freak out if there's any issues.

In other words, the question with package managers/installers is "Do I trust this developer?". The question with curl-to-sh is "Do I trust this developer AND do I trust that the website hasn't been compromised".

> "Do I trust this developer AND do I trust that the website hasn't been compromised".

You also have to verify that the script is coming encrypted over TLS. It's not mandatory, and there's no warnings if its not. Sure, it's an easy check (barring redirects, which also occur), but how many people who are copy/pasting curl|sh scripts will think to do even that basic validation?

Your mistake is assuming that people only do that for curl|sh — in my experience, the same percentage (high) will manually download and run a program / shell script, add an APT/YUM repository, etc. Once someone decides they need to run something the details aren’t very significant.

The problem isn’t curl but the challenge of trusting code. The default needs to be either Apple-style sandboxing (note how you get prompted if a new program tries to access your pictures, contacts, etc.) or something like Docker where the default is private with exceptions for the resources you enable.

Package managers at least provide a path to trust - the PGP web of trust. curl|sh does not provide any path to (cryptographically secured) trust.

That's only for people using the global trust mechanism in a mainstream repo. If you think about it from a security perspective, how much difference does installing some random project's GPG signing key make?

Yes, very true, I was sort of assuming that the Hacker News folks would know to check for "https" and that any person offering a curl|sh solution would be sane enough not to offer it over http. But yeah, it's a good point that it's easy enough to miss.

I wanted to update the article to adress signing in more detail last time this was shared, but ... I'm lazy :-) I does mention it though: "Package managers are more secure due to checksums, signing, and auditing".

Maintaining packages is a lot of work: there are many of them, all with different policies/update schedules/etc. It can be very time-consuming to maintain a package for all systems. Package managers are great, but does come with some trade-offs, like most things.

There is nothing really stopping you from signing shell scripts; although I'm not sure if you'll still be able to run them with some clever "curl | gpg | sh" pipeline; but you can "curl>file; gpg [..]; sh file" it.

I think very few people actually verify signatures manually anyway, not in the least because it's kinda hard to do so (gpg is a beast, although there are some better alternatives now, such as signify/minisign).

My reaction is that curl is just an aggravating factor.

The real problem is shell, or really any custom scripts in any language. Acquiring or building software should happen via well-understood reusable tools. Any time building executes bespoke code, you're reducing the maintainability of a system.

Distro package managers are one good solution, but a properly developed language package manager would also ideally not rely on custom scripts.

A contributing factor to the event-stream issue was that it was considered acceptable to distribute code minimized by _who_knows_what, rather than by any reproducible process.

> Distro package managers are one good solution

Distro package managers allow execution of custom scripts at every point of the installation. If you're getting packages from outside the distro repos, you're at risk. Sometimes you're at risk even when you get the packages from the distro repos :-)

A package is signed with a known key, so you can at least trust that the script being executed is from the developer.

What about the distros that modify the package before giving it to you? I'm thinking of Debian's modification of openssl for example.

https://www.debian.org/security/2008/dsa-1571

Just because it comes from a distro doesn't mean it's from the developer.

It's still a trust issue. If I'm running a Debian based distribution, I trust Debian packages, and the content of those packages.

It's a chain of trust that can't be provided by web servers and X509 certs.

I thought about that, but like falcolas said, they are at least signed and have some attention paid to them.

At one level, any build/install tool is executing extremely complicated code on your machine, and could do lots of weird things. The advantage is that you can learn what it does, and have trust in its authors. So well behaved custom scripts created by a trusted party that has articulated standards about how things should work are much better than using shell scripts from an arbitrary project.

Still, I'd certainly prefer it if the distro didn't have custom scripts either. It would simplify the task of knowing "this is how the distro does things, and there are no exceptions".

You missed something, my main point: not all packages that are installed with the distro package managers come from the distro. There's plenty of applications that package their software as .deb/.rpm/...

From a security point of view, those are the same as curl | bash, basically.

Ah yes. I did gloss over that. I certainly feel a little sketched out when I see a project that says "add this package repo".

Download and look before running gives you an opportunity to spot something that's obviously, totally wrong... Not that one always uses that chance but at least you had it.

Installer systems that chain these "download and run" scripts (Node RED is the most recent I've used) scare me despite all the valid points TFA makes. "Not invented here / history must be reinvented" seems like philosophical point for some of these systems.

Also, I know what git, configure, and make are supposed to do when invoked. I have no idea what piping a random script to sh is supposed to do without reading it.

You might be surprised. As the OP mentions, configure scripts could contain literally anything. Ditto makefiles. If you're not reading them, you really have no idea what they might do. It's a little harder to embed something awful in git, but with hooks etc. it's not impossible. So when you do "git clone" and "configure" and "make" you're just as subject to vulnerabilities from file replacement or DNS hijacking, and there are just as many opportunities for something bad to happen as if you had piped curl through bash.

This is why checking signatures is the most important part of code distribution. It doesn't solve every problem, but it solves most. At least then you know whose code you're getting, and that they attest to the code's validity (including safety). Maybe they're wrong or maybe they're not as good as you thought they were, but it's still a big improvement.

If we ignore the potential security issues with either approach for a moment and assume for the sake of argument that neither is more risky than the other I "might be surprised" by the behavior of running `./configure` or `make install` but I will almost always be surprised by the behavior downloading and running a random shell script.

I prefer a package over either case, but lacking a package I'd prefer to use something that has some expectations of how it should behave. As a packager I find that software with a Makefile, CMakeLists.txt, or a configure script is likely to be easier to package than something that just provides a shell script. Those might still need to be patched or tweaked but there's an expectation that it's going to try to behave a certain way.

Also gives the same opportunity to your AV software if you happen to have one.

I agree 100%, all this does is reinforce the errors of the past and trains end users to be dumb. This is NO DIFFERENT then Windows users double-clicking on every email attachment.

Every time somebody suggests this, the author needs to get slapped with a trout. Copy/Pasting code to get entered is BAD ENOUGH; nobody learns anything that way AND it is not secure!

In a race to the bottom everyone looses.

There is a difference between downloading, say, the Docker install script from docker.com and opening a random email attachment and that is that I trust docker.com.

If I don't trust docker.com because I think it might deliver malware in the install script or a poorly written install script, then I shouldn't use Docker because why would anything different apply to their main software.

If I don't trust docker.com because I fear it might be hijacked then I shouldn't use Docker because who guarantees me that their source code or any derived binaries aren't also hijacked.

That trust comes from education and experience. Just like an experienced computer user could launch an email attachment safely if they know what they are doing.

My point is telling new users to blindly copy/paste/autorun ALL scripts just to get $thing to work only hurts US (FOSS proponents) and the end user.

From the article:

===

Partial content: the shell may execute half the script due to a network error.

Easily fixable by running in a function:

    do_work() {
       : 
    }

    do_work

All of the cited examples already do this.

===

This is the one argument that I think the author gets wrong. Just because the projects he cited do this correctly doesn't mean that all `curl install.sh | sh` scripts will do this correctly. And this is the main fear that I have. I'm not all that concerned about people publishing malicious installation shell scripts. These things are normally public, so it would be easy for them to get caught. I am concerned about the scripts being either (a) altered in transit or (b) corrupted in transit (which is really the same thing).

It seems like there could be some happy medium where there was an install script that had a published signature (SHA1 and/or full cryptographically signed), and a tool that downloaded it, verified the signature, and then called a standard function: `my_install()` (or something).

Well, altered in transit and corrupted in transit are both covered by HTTPS. (I think error detection is even covered in TCP, hence also HTTP?)

If a vendor does deliver software over HTTP, then you should verify the script before executing it via a checksum delivered over some secure channel. (This is also what apt, the Debian package manager, does by default afaik.)

Not just a checksum, a cryptographic signature, which verifies both the contents, and that the contents are from the owner of the key.

It's still possible for data to be silently corrupted before it hits the HTTPS server [1]. After dealing with a similar issue that corrupted data silently, but still had valid checksums, I tend to not trust implicit checks. (My example wasn't an HTTPS specific issue, but it in theory could still happen with HTTPS).

[1] https://stackoverflow.com/questions/34610581/is-it-necessary...

If you are curl|sh'ing while blindly following some random online to tutorial, it is bad.

If you are curl|sh'ing and know why you're curl|sh'ing and know what you're curl|sh'ing and understand the implications of curl|sh'ing, then I don't see the problem.

Like most things, it's subjective.

How can you know what you're "curl|sh'ing"? If you're piping it, it's impossible to know what you're getting and executing. That's the problem.

I typically load it up in a web browser, have a peek to make sure it looks sane, and then go for it. If I'm particularly paranoid I can wget it then execute it. But if it's something I'm familiar with, curl|sh is fine.

Like I said, it's subjective. No such statement as "curl|sh is bad" or "curl|sh is fine" can be accurate on its own.

Checking in a browser may not be representative. https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-b...

And furthermore, it's a matter of trust. If I don't trust the source that I'm running the software from, then curl|sh isn't the problem.

Trust, and what you download from a web server, are two distinct issues. There is no way to verify that the content you get from a web server is what the developer wrote (particularly when you get into chains of 301 redirects to a github repo. Packages, at least, provide a signature from the developers (or distro packagers) that you can verify independently.

> If you're piping it, it's impossible to know what you're getting and executing. That's the problem.

Exactly the same problem as with apt-get, brew, yum, npm, yarn, cargo, <the list just goes on and on>.

Most of those will run any number of scripts during execution without your knowledge.

I can trust packages, since they're signed by a developer, or by a distribution. I can't trust that the content of a script I obtained via curl is from the developer.

npm has had multiple incidents with malicious packages [1][2]. apt had its own CVE [3]

[1] https://securityaffairs.co/wordpress/88433/hacking/npm-insta...

[2] https://securityaffairs.co/wordpress/74497/hacking/malicious...

[3] https://lists.debian.org/debian-security-announce/2019/msg00...

NPM packages are not, to the best of my knowledge, cryptographically signed. If you're using them blindly, then you are indeed in the same camp as those who curl|sh. Needless to say, you shouldn't blindly install and run NPM scripts either.

And if it takes a CVE to allow a MITM attack to inject a malicious payload into an apt package, that's already better than curl, which does not. It's also worth noting that the CVE had a stable fix available as of that announcement.

> It's also worth noting that the CVE had a stable fix available as of that announcement.

It's also worth noting that the problem was there before it was discovered, fixed, and then disclosed.

Ultimately, it comes down to how paranoid you are about your sources of software: https://wiki.c2.com/?TheKenThompsonHack

Not impossible at all. If you don't trust the https site from which you're curling, first curl it to a file and check it out. Once you're satisfied, run the curl|sh as documented, or run what your already downloaded.

And sometimes you want a functioning hands-on demo of some open source software that has many moving pieces. If the author has collected a bunch of installation steps into a bash script, I have no problem blindly running it on a fresh isolated VM.

Later on if I decide to use the software, the bash script usually makes a nice reference for starting my own Ansible role or Dockerfile.

Classic: Is curl|bash insecure? (Sept 25, 2015 | 163 comments)

Link: https://sandstorm.io/news/2015-09-24-is-curl-bash-insecure-p...

HN discussion: https://news.ycombinator.com/item?id=10277470

I'd like to be proven wrong, but isn't thinking that executing unsigned script (bash, js, …) on your computer is somehow inherently less secure than running some binary is nothing but a superstition?

I just ddg'd "sandbox shell script" and learned about chroot. First hit:

How can I “sandbox” a shell script?

https://unix.stackexchange.com/questions/363363/how-can-i-sa...

What would a proper sandbox look like? Every session runs within a Docker image?

I don't know anything about this stuff. My intro to the sandboxing and security notions was Java and its Applets. I guess I thought we'd eventually do everything that way.

Um, no. Curl-to-shell is bad.

Check this out: https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-b...

As the author notes:

> You’re not running some random shell script from a random author, you’re running it from a software vendor who you already trust to run software.

Without any form of validation that the shell script I'm running is actually from the software vendor in question. That trust is even further stretched when the request is 301 redirected to another host entirely.

Between custom domain names, TLDs, Let's Encrypt (I love Let's Encrypt, but it provides is encryption, not trust) and cheeky developers using combinations of those, how can I really know that a script coming from (fictional example) https://install.dock.er is actually the Docker company?

The same arguments apply for downloading a binary.

If you want to pull this even further. When is the last time you verified the signing keys of your OS distribution repo without relying on the internet?

A lot of install methods that are not curl/sh are like: here copy this bash line to add apt GPG keys for our repo, apt update and install. A lot of people don't bother to check those keys.

True, some people don't check those keys, but it's possible to do. There's a well trodden (and cryptographically secure) path for gaining trust in a key that's distinct from downloading and unpackaging a file.

This is (currently) not possible to do with scripts downloaded from a web page. Especially when immediately piped into a shell.

As with most issues in security its a balance between usability and security. The real problem is the majority of the users will always try to find the path of least resistance. No matter how well documented a secure procedure is, if a user can find a oneliner they will use it instead.

I've seen this in other places as well. Vendor makes a comprehensive guide. Some people condense that to a minimum and will even boast they "made" an easier way to do X or Z, whilst ommiting all the caveats. Of course that will become the popular 'standard' people find. And they'll go complain to the vendor if it doesn't work without even reading the original instructions.

You're not wrong with your first paragraph. That doesn't make the use of curl|sh good however. It doesn't justify developers laziness in eschewing secure methods of distribution, or writing articles like this that justify that same laziness.

Wrong. You are trusting much more than that, including network components outside of your control and the author's.

Plus you are pushing forward the bad pattern and being part of the problem.

You are encouraging other authors and users to do the same.

Tell me how this is different for downloading a binary instead. Unless you get something like a GPG through a completely different channel.

A major point of the article is that there's nothing unique about curl-to-shell which enables this exploit; it would be equally easy to perform it with a fully-signed and checksummed binary downloaded through a secure channel.

You do need to control both the key and the channel in this case.

That major point is plain wrong.

https://sysdig.com/blog/friends-dont-let-friends-curl-bash/

https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-b...

These attacks can be mounted on the network side.

Also you can inadvertently execute data from a benign captive portal or a server error page and so on.

I can do similar attacks on ANYTHING in the form of grabbing code and running it on my machine.. If you aren't doing 'curl | bash' recklessly it isn't a problem.

If you don’t actually read or verify the script this doesn’t matter. If you cant trust the server you probably shouldn’t run any code from it.

But you can trust the large userbase of a popular Linux distribuiton, especially when big companies use it.

Distro package managers have other advantages than trust. Typically they at least verify cryptographic hashes but many of them also verify public key signatures as well.

This is why its safe to download packages from mirrors.

With that having been said, Debian takes an extra step into the absurd by using plain HTTP with no TLS for downloading packages. There’s no obvious security issue with that, but it does feel like a bad decision, since anyone in your request path can see what software you’re installing, and if there ever were a vulnerability it would be much easier to exploit due to the lack of security at the transport layer.

How does one trust a server? What assurances can any server provide that they are who they say they are?

x509 certificates certainly don't provide those assurances.

No no no, human trust, not machine/cryptographic trust. The same way you have human trust (or distrust) over many things running on your computer. You didn’t get to verify any of the code in the firmware either.

There’s differing levels of paranoia. I don’t pipe curl to shell at work. I do it at home sometimes, usually via Github.

Did you not read the article? He addresses those points for the most part. The primary idea is that when you run ANY SOFTWARE in reality you have no clue what it might do, you didn't look through every single line of code. There is nothing wrong with 'curl .. | sh' as long as you're running a script completely blind.

"Compared to running with razor blades, running with scissors is fine," is not much of a justification of an activity.

Instead of providing justifications, perhaps we should focus on education (or making it simpler to be secure)?

> You can regenerate them from autoconf

Not that the autoconf input is readable itself, either, anyway...

> if there is a potential security problem and no one is exploiting it, then is it still a security problem?”

Yes, surely? Somebody is probably exploiting it and you just aren't aware, because why wouldn't they have that in their toolbox?

> Somebody is probably exploiting it and you just aren't aware

I tried finding an example, and asked last time this was on HN as well, and thus far I've not found a single example.

Absolute security isn't realistic; it's often a trade-off between convenience, cost, etc. I can probably break in to your house by throwing a brick through your window ... should we now all invest in more secure windows? Doesn't strike me as a good trade-off for most people.

It's not that curl isn't so bad, but that the lack of notarization is really bad. Distributing software safely is still a pain.

Linux distributions have this kinda figured out a long time ago with signed packages - almost a requirement if you want to distribute your software on mirrors that you don't control. This is why it's safe to download a package from a Debian HTTP mirror and install it. The thing afaik any of them still don't do, and is really important too, is checking the signature with a revocation policy.

I actually started writing about how I distributed a CLI for macOS, Linux, and Windows a few years ago but never finished it.

If you are writing Go code you want to take a look at Equinox.

I used it to distribute a CLI in my previous job, and it worked like a charm. One of the things I liked most was that I was finally able to push safe updates because I authenticated every release with my private key, and only allowed users to update if the key matched. If you have a primary and a secondary key, and a strategy to revoke it, this means you have a quite safe release process.

https://developer.apple.com/library/archive/documentation/Se...

https://equinox.io

https://developer.apple.com/macos/distribution/

https://developer.apple.com/documentation/xcode/notarizing_m...

https://blog.jessfraz.com/post/why-open-source-firmware-is-i...

https://docs.microsoft.com/en-us/windows/win32/seccrypto/cry...

https://docs.microsoft.com/en-us/windows/win32/seccrypto/sig...

https://www.mothersruin.com/software/SuspiciousPackage/

https://successfulsoftware.net/2012/08/30/how-to-sign-your-m...

https://arstechnica.com/gadgets/2012/02/developers-gatekeepe...

Curl to shell is usually indicative of a package manager that could be doing a better job.

If only there was one standard package manager... dealing with brew/apt/[yum|dnf] and handling multiple yearly vendor releases (Ubuntu and Fedora) gets to be annoying.

Not only that, but I always end up with impossible-to-diagnose phantom errors if I start mixing and matching some installations from package manager and some installations from elsewhere (like curl | sh or ./configure, make, make install). I avoid using package managers as much as possible because they do too much hidden, undoable, inscrutable magic behind the scenes that breaks everything when it’s most inconvenient.

And here, I go the other way -- I keep to only using package managers as much as possible because at least they have some coherent plan for where installation data should go. Random install scripts can put data anywhere and can easily mess with global or system installed data. If I can't set the installation prefix myself, I don't use the code.

(I'm sure this issue has something to do with the popularity of containers)

...not to mention that (some) distributions do extensive security and legal compliance reviews...

And some random shell script installer is going to better than a dedicated package manager at dealing with the differences between N many distributions and M many architectures?

Sure, some install scripts will be quite simple, but in that case, why do you need to have the install script in the first place? You'd only need it if the installation procedure was too complicated for an `INSTALL` or `README` document.

> And some random shell script installer is going to better than a dedicated package manager at dealing with the differences between N many distributions and M many architectures?

Surprisingly, yes. Especially for *nix systems where a lot of things (but not package managers) are more-or-less unified. The author may not have the time or desire to learn the half-a-dozen or so package managers and set up a build system to create packages for all of those, and then do the work of getting them into the standard repositories.

That's what distributions are for.

Just write software that is not unnecessarily difficult to package.

Reply on news.ycombinator.com