Hacker News Reader

Comment by orblivion | Hacker News Reader

I control a fleet of servers. I have a saltstack or ansible script. One of the steps in provisioning a new server is to pull library X from github.com.

One day Egor Homakov finds a new hack and finds his way into access to the master branch for library X. As a prank, he force pushes a change to master.

Being aware of such a possibility, instead of setting up my script to pull from master or even a specific tag, perhaps I pull from a specific sha1sum. That way I know I'm getting the exact version I want. That's today. Tomorrow, when preimage SHA-1 attacks are cheap, it will no longer save me.

View on HN

Sounds like sentence three is where you got it wrong. You should be pulling your software to a development host, building it after verification, and uploading the deployment package to an artifact repository inside your security perimeter. Sentence three becomes 'pull library X from my artifact server.'

There is no way to securely deploy a package directly from the internet. The sooner you understand that, the sooner you'll be able to sleep through needless catastrophes like this hypothetical attack -- or leftpad.js.

> There is no way to securely deploy a package directly from the internet.

Fetching a file from the internet and verifying it against a hash or signature is the only way to securely deploy a package from the internet. This is exactly one of the use cases theses cryptographic primitives are built for. Don't blame the users when they utilize the tools the way they are supposed to be, blame the tools when they fail when doing so.

And by the way, if you ever need to run `apt-get upgrade` or `apt-get install` on your production server from a public mirror, then you are guilty of "deploying a package directly from the internet", too. (My apologies and congratulations if don't.)

apt-get can (and do) check signatures of repos

But yeah, you could put an unverified repos to be used

For the argument at hand (Can you deploy anything on a production server from a third party that is only verified cryptographically?) signatures and hashes fulfill the same function (and btw I also wrote "verifying it against a hash or signature" above). More to the point, under the hood GPG signatures only sign a hash of the file in question anyway. Verifying a file via a GPG signature is strictly less secure than verifying it by its hash (assuming you use the same hash function as the signature).

Git has tag signing, surely there's a way to clone a specific tag and check the signature against a specific GPG key fingerprint?

Tag signatures only cover the mapping from tag name to commit hash. In other words, specifying a manually-verified commit hash is actually more secure.

Tag signatures are mostly worthless now from a crypto point of view -- with the caveat that you can still get some value from them if you still trust sha1 to be secure against second-preimage attacks.

    git checkout 0.1.0
    git tag -v 0.1.0

Either way he'd be pulling from Github.com at some point, whether in the build environment or production. He won't ever know that the repository was compromised.

Besides, some people use Heroku simply to deploy their rails or node apps, and they don't have an artefacts server.

> I control a fleet of servers. I have a saltstack or ansible script. One of the steps in provisioning a new server is to pull library X from github.com.

Ansible gives larger and easier to exploit attack vectors than git.

And then, with pulling semi-random things from internets you have much bigger problems with your deployment procedure. You should never ever download a git repository with software, instead you should be using package system supplied by your operating system.

> you should be using package system supplied by your operating system

2005-me vehemently agrees with you.

Package managers allow you to download from git repositories too. He'd suffer from the same attack vector.

Erm... Which ones?

I believe Homebrew on macOS uses it for formulae (Ruby scripts to actually download and build the application)

Well, I don't hear too often about a fleet of servers running MacOS (because this branch of comments started with provisioning in a fleet of servers).

Also, from rants I hear about Homebrew and its breaking random libraries on upgrades, I don't think it should be mentioned as it was state-of-the-art or something.

Almost all of them:

Npm: https://docs.npmjs.com/cli/install

Bundler: https://bundler.io/git.html

Homebrew: https://github.com/Homebrew/brew

And which of these are "package systems supplied by your operating system"?

While I agree that the OS package should be used first and foremost, it just often doesn't have the required software. (Even after adding extra repos that it might support)

You are picking on the useless details. Those are all commonly used package managers in production environments. They often provide software that simply never gets packaged with the OS. They likely always will, because they have more focused design goals.

A better way to argue this would be point out that specific ways to use the package managers better. For example: bundlr supports saving all the required packages offline. This provides the opportunity to do a security review and save the packages locally/internally rather than always trusting the whatever is on the Internet.

> While I agree that the OS package should be used first and foremost, it just often doesn't have the required software. (Even after adding extra repos that it might support)

OS-supplied packages don't grow on magical trees. If you don't have the necessary software in official repositories (or if it's your software), you can package it yourself. Deployment then becomes a breeze, and you save yourself otherwise completely useless process of recompiling things over and over again.

> You are picking on the useless details.

Quite the contrary. Those details make important difference.

> Those are all commonly used package managers in production environments. They often provide software that simply never gets packaged with the OS.

Apart from Homebrew, which is for workstations (hardly anybody runs macOS servers), none of these "package managers used in production environments" provide you a complete way to rebuild your software. You can be fine for a while if you stay away from modules that are interfaces to C or C++ libraries and from tools from other languages (e.g. I have used Python's Sphinx to document Erlang daemons quite successfully), but once you hit that, deployment starts to be PITA, because you'll need to remember to install all the required libraries, -dev packages, compilers, and what not.

On the other hand, DEB or RPM with artifacts will just automatically pull the required libraries, and its build dependencies give a dedicated and standard place for the necessary build tools.

Your comment supports my opinion that today's programmers usually don't want to be bothered with learning things that have been working for sysadmins for twenty years already.

Gentoos emerge and FreeBSD port are package managers that preferably build from source. At least emerge also accepts git sources, with tags and commit specifiers. I haven't used bsd in a while on a prod system, but I'd not be surprised if port gained the same functionality - after all whether you're pulling a tarball and use a checksum for integrity checking or hand the task off to git makes exactly no difference at all.

And I have yet to hear that port and emerge are incapable of doing dependency resolution. They're battle tested systems that work well in production environments.

First, portage and ports are OS-supplied mechanism for installing software. They are nothing like pip or gems or npm, which only can install things written in their respective languages of choice and fail miserably for modules touching any library external to them (unless you manually ensure the library's and compiler toolchain's presence, that is).

Second, ports and portage have support for and networks of mirror servers that keep copies of software available through these packaging systems. It's trivial to switch if one of the mirrors goes down. For pip, gems, or npm you need to plan ahead for the problems and deploy your own package cache, from what I know.

Third, I was using Gentoo with one of these "battle tested systems that work well in production" for several years. It was doable, but it wasn't pretty, could lead to breaking software after updating some random deep dependency (if it was recompiled with different flags), and generally required more work and attention than APT would, all that for very little gain (if any gain at all). Oh, and it ended up working with binary packages, after all, I just needed to compile them myself instead of having a half an hour downtime of production MySQL because it needed to get compiled (which could fail, leaving me with no working database installed).

Gems can be used to installing anything. It often build java, go, C, C++ and I think I heard it could do rust once, but generally anything in faster languages to create faster versions of the implementation of the functionality for a given gem.

I bring that up to highlight some of the wrong assumptions you make. You make several needless assumptions and use those to draw funny distinctions between things. I am not even sure of the point anymore.

likely any of these systems could be used in a variety of environments for a variety of purposes.

You can do anything, including Rust, yeah. You can also distribute pre compiled stuff.

The ones that actually matter are more like apt or yum.

Most language-specific ones do. OS-specific ones less commonly do; Chocolatey, for example, can (IIRC) install packages by running PowerShell scripts, which may or may not include pulling from GitHub.

> Most language-specific ones do.

Those are development tools, not deployment ones (even though they are used as such; programmers usually don't bother with learning what sysadmins do, so it's not a surprise).

getoos emerge can. A package definition contains a source URL and git urls (with branches and tags) are accepted sources.

yaourt on archlinux

Pip, too.

> Tomorrow, when preimage SHA-1 attacks are cheap, it will no longer save me.

Well, Moore's law is petering out isn't it? :)

Computers are not getting better at the rate they used to, but we are still improving both on performance and power.

However, the largest change we have seen the last couple of years is the accessibility of computer power. These days an attacker can be concerned mostly by how many CPU/GPU hours he needs to rent from Amazon or equivalent providers to achieve his goal. So the accessibility, convenience, and price of processing power is still improving quite fast.

Depending on your interpretation. If you go by the relatively useless transistor/fixed price cpu then it is dead, but if you go by more practical measures like instructions per second in whole system/dollar the it is as fast as ever. GPUs are really fast and still advancing.

Moore's law is, but the cost of a given amount of computational ability is still going down quickly.

If you want a specific hash, why pull it from github everytime?

Because, when provisioning a new server, where else can you get it from?

A host that you control.

Where did it get it from?

From github, the first time only. Not every single time.

Isn't getting it from GitHub the first time vulnerable to attack? Consider when you're scaling up or replacing a failed host.

From a custom package you have built? Did you know that RPMs and DEBs don't grow on magical trees, but are built?

So, the problem still stands, theoretically.

Not only do you likely need to populate your host with packages not from your host. But also, your host will also still be connect to a public net, even if only indirectly (e.g. private net), and hence potentially manipulated.

No, you misunderstood what is actually the problem here. Pulling under deployment some code from random resource from the internets that can go down or get deleted at a whim and you can't easily move to just some other mirror and you don't even control when the thing will be up back, that's the problem. Not the trust you need to put to use the code (this is still there, obviously). And the very same comment applies to third-party package repositories, like PPAs in Ubuntu.

Not to mention that with pre-built binary packages your deployment speed and repeatability get significantly better, as you don't need to rebuild the artifacts every single time.

I do understand, but "the trust you need to put to use the code" is what we were talking about, no?

Reply on news.ycombinator.com