Hacker News Reader

Comment by ohazi | Hacker News Reader

On a more serious note, I'm surprised there hasn't been much discussion about potential malware in official Linux package repositories vs. developer-centric source repositories like npm, rubygems, crates.io, etc.

One would hope that the bar is higher with strict maintainership rules, but there are a zillion packages, and you can't vet them all. Also, practically everyone installs binary packages, so until we have fully reproducible builds, hiding malware in some obscure but heavily depended on package could actually be relatively easy.

View on HN

> the [Debian] Release Team recently announced that Debian will no longer allow binaries built and uploaded by maintainers on their own machines to be part of the upcoming release.

https://reproducible-builds.org/reports/2019-07/

All the npm cases I can think of have been in javascript code, not 'binaries'. So that wouldn't have prevented any of them.

Fair point. For Javascript there is no binary step so it has to be in the source code. In compiled languages, this is different: you can hide it in the binary, and finding it there is much harder than checking the source code VCS history. So without reproducible builds, compiled package managers would be in an even worse situation than Javascript, at least what ability to review the uploaded packages is concerned.

Ultimately the problem doesn't have a good solution right now. Reproducible builds are a part of the solution, as is making it harder for malicious debian packagers to upload infected binaries. You need to protect the entire supply chain.

It makes absolutely no difference when binary packages are compiled on a trusted build farm, i.e. like everyone does already.

This plus reproducible builds will go a long ways toward preventing it.

I would be careful about claiming built artifacts play much role in the visibility of repository-delivered malware. Consider "reflections on trusting trust" for attacks on reproducible builds, and also that all the major scripting languages (PyPI, Ruby gems, and npm) have had malicious packages inserted over the last year or so.

> all the major scripting languages (PyPI, Ruby gems, and npm) have had malicious packages inserted

None of those package repositories are maintained by a limited set of curators.

Debian's repositories and other linux distributions repositories are curated.

Uploading a malicious package to npm is as easy as typing 'npm publish'.

> "reflections on trusting trust"

It's a very useful piece of art and a thought experiment.

In practice, it's not interesting for the average package. Very few packages are self-hosting and in such a position to leverage that sort of clever trickery.

A good debian maintainer will also review code of their packages and make sure such trickery has no chance to be afood (e.g. by preventing a gem from downloading a custom ruby interpreter to bootstrap itself, but rather manually bootstrapping up themselves).

The point of "... trusting trust" is not really the specific bug, but rather the observation that the whole supply chain matters. Built artifacts as exist on disk in the repo are a relatively small piece of that.

> None of those package repositories are maintained by a limited set of curators.

It's always interesting to me how few people seemingly know this. Python package maintainers realize how easy it is to add packages to PyPI, but many admins and even SOC operators inherently trust it because they think that it "must be reviewed by someone".

Also, there have been plenty of security issues with Ubuntu and Debian: https://wiki.ubuntu.com/UbuntuWeeklyNewsletter/Issue52#Commu... https://www.linuxinsider.com/story/32240.html

Also there have been bugs in apt: https://www.fosslinux.com/6167/massive-security-bug-found-in...

Also the official ISO from the Mint website: https://arstechnica.com/information-technology/2016/02/linux...

Also the AUR malware mentioned on thread: https://www.bleepingcomputer.com/news/security/malware-found...

Also the kernel.org compromise: https://www.zdnet.com/article/linux-kernel-source-code-repos...

Edit: let's throw in the redhat ceph intrusion while we're at it (note that this includes InkTank). https://www.zdnet.com/article/red-hats-ceph-and-inktank-code...

I don't like when "Reflections on trusting trust" is used as an excuse to throw the hands in the air and not doing anything. Sure, it's not perfect, but the reproducible-builds people are doing important work that makes it considerably harder to inject malware into distribution repos.

I'm sorry, but the logical fallacy of "We must do something; This is something; Therefore, we must do this." is the worst attitude to have for security. Not only you will never end with anything actually important done, it will also justify all the evil that will be done in the name of security.

And nobody says that not doing anything is ok, but please do it properly, do some basic threat modeling first.

P.S. I'm not against reproducible builds, I think they are useful for many things, just not for security. They may even let us do decentralized trustless peer-to-peer build system someday.

The hard part about the security of built artifacts is not building them, it's auditing the source looking for bugs. Reproducible-builds doesn't audit the source, but many act like they think it does. Added to this is the host of other security issues with the chain between the developers keyboard and your system. This solves none of them, but many act like they think it does.

Therefore, in my view it provides a smallish real benefit and a large moral hazard.

> The point of "... trusting trust" is not really the specific bug, but rather the observation that the whole supply chain matters.

There's a really fascinating project by the authors of TUF called in-toto[1] that addresses exactly this problem.

[1] https://in-toto.github.io/

Science is hard. Superstition is fast and might save your job, at least for awhile.

I want this on a shirt.

>A good debian maintainer will also review code of their packages and make sure such trickery has no chance to be afood

Except for that time that Debian developers "knowing better" completely neutered most cryptography on all updated Debian systems[1]:

>Affected keys include SSH keys, OpenVPN keys, DNSSEC keys, and key material for use in X.509 certificates and session keys used in SSL/TLS connections. Keys generated with GnuPG or GNUTLS are not affected, though.

Massive, rapid key rotation schemes had do be implemented to cover their screwup because all keys were trivially enumerable, to the point where all affected keys were blacklisted[2].

[1] https://www.debian.org/security/2008/dsa-1571

[2] https://security.stackexchange.com/questions/3422/what-is-th...

Reproducible builds can't prevent hidden malware. They don't do much at all for security.

They can't prevent it but they can help it become visible. Without reproducible builds, nobody has an easy way to verify that a specific binary corresponds to a specific deemed safe bit of source code. There can still be source code based attacks by upstream maintainers of course. In fact most attacks in the past were in source based languages like JavaScript but without reproducible builds the situation would be much worse.

> In fact most attacks in the past were in source based languages like JavaScript but without reproducible builds the situation would be much worse.

This is incorrect. It wouldn't make a single difference.

It forces logging of the malware, globally. That's worse than maybe getting caught.

If you assume that the build systems and the distribution are secure.

If somebody had stolen the signing keys and could do a https://www.cloudflare.com/learning/security/glossary/bgp-hi... there will be no trace.

Bugs in the build system are just bugs, quite possibly security bugs but still set in stone. Reproducible builds have no point if you distribution is broken and cant actually cant verify they are reproduced. You could have just asked "What if the reproduced build isn't?". It's not a reason to not be able to reproduce the image.

We know how to publish the input and output hashes. It's one of the use cases for a global ledger. Using CF's trust root would be pointless.

Nix/Guix are quite cool to tackle this.

They are progressing quite quickly towards reproducible builds, so you can avoid compromised servers sending malicious binaries.

Also, since package definitions are quite declarative, I guess it's easier to perform static verification to spot malicious code getting introduced there.

There's e.g. Vulnix that scans for CVEs [1].

[1] https://github.com/flyingcircusio/vulnix

I believe they aren’t just progressing towards reproducible builds, but that the whole system is already predicated on them. In Nix, you have to make special exceptions for packages that aren’t rebuildable from source.

Rebuildable is different than reproducible. The Nix system is predicated on rebuilding. Ideally the binaries would be all also reproducible however it turns out making compilation completely deterministic is actually pretty hard. When we reach that point and me and you can both independently produce identical binaries and hash them from exactly the same source code that is beyond fantastic for security and auditing.

r13y.com tracks the progress for NixOS reproducibility (well, for the tiny subset of nixpkgs that forms the minimal ISO image at least).

NixOS (and NixPkgs) is great, but there are still plenty of binary only packages from difficult to build projects like PhantomJS in the ecosystem. It takes a serious commitment to build complex projects in Nix and maintain them.

(edit) An example of binary packages in Nixpkgs: https://github.com/NixOS/nixpkgs/blob/master/pkgs/developmen...

You can also build regular packages, no matter how complicated they are, by spanning an FHS environment. That's sort of a sandbox that simulates a regular Unix environment with whatever dependencies you need in the usual places (/bin, /usr, ...).

Of course, you loose some Nix advantages. But it's a great way to get a regular Unix environment for one-off things or desperate cases.

Should you really be installing programs that are impossible to build? Also phantomjs is discontinued.

That's a great question, but it does not invalidate the point that there is nothing about Nix that states that binary packages cannot and do not happen.

Try for example FHS environments. That should make it possible to install software in a conventional way.

Hasn't this happened a few times? IIRC the AUR had some malware last year, but it was caught pretty quickly. I think why it doesn't happen often is more visibility thing. If it's a useful virus then it's going to be in a popular package, for maximum contamination (or at least some threshold). But people using things like the AUR are generally much more tech savvy then you're average computer user and will correctly flag a package as malicious much faster. Plus there is a higher percentage of users looking at commits. So if they get caught fast then there's more protection for the less savvy (or "lazy"). This is just a guess though.

I think people inspect AUR packages before compiling, at least I do, so something obvious will be caught. But something like backdoor hiding in plain sight as an RCE vulnerability or even just more obvious code sneakely integrated into the source and waiting for a specific time far into the future to activate is very unlikely to be caught.

This is a flaw with the repository model for software distribution: it confers the authority of the OS developers to packages not scrutinized to the same degree.

Users' metal models of trustworthiness don't track very well the actual scrutiny software is subjected to. This might be a problem with any distribution system.

FWIW, as far as "scrutiny" goes, I think the track record of Red Hat and Debian/Ubuntu on this stuff is about as good as... well, basically any other distribution system I'm aware of. The Android and iOS stores have hosted malware, as has Microsoft's. Steam has pushed bad stuff. NPM is a straight up train wreck...

I mean, I agree in principle that it's a hard problem. But the actual solution we've landed on in the Linux world seems like... well, just not really the first thing we should be worried about.

That's definitely the feeling I've gotten from the Debian repositories, which are very cautious about what to distribute. I was thinking about it in the abstract.

What's the solution? I would argue that the Linux model is way more secure than the windows model of "Google your software and download it from whatever website pops up". It also allows for much more freedom and availability than the locked down macos app store.

It's a tough balancing act.

I think you're overestimating the rigor of other software delivery channels. Consider for instance that the biggest Windows SSH client (PuTTY) was delivered over http from a super sketchy looking website until very recently. Now it's the same super sketchy looking website, but with a green lock!

Haven't used Putty (or Windows, for that matter) for a while, but the site does not look sketchy at all. It looks like either a two decades old site, or a site of someone who purposely objects the modern web design tendencies (and believes some of the earlier mantras of web design, like user should have control of line lenghts by resizing their window).

A "sketchy" web site is the one festered with ads, pop-ups, seemingly modern-looking but with no actual content or with dead links...

Depends on the attack vector. If the concern is inadvertent malware (i.e. an unaware developer incorporating a malicious library etc.) then I'd suspect the package repos would be a bit safer because they tend to act as a low pass filter (active source repos can change very frequently, downstream package repos much less so) and also act as one or more additional sets of eyes on the source code. Some developers can get so focused on getting things done that they can lose sight other aspects of their application.

I've heard far more stories about malware making it into developer repos (usually inadvertently) than making it to package repos. I suspect it's partially the relative areas of focus of developers vs package maintainers and partially just having another set of eyes at least glancing at what's going on.

Ahh the attack vector. Here’s a free one: KDE addons for things like themes and wallpapers. The package format allows running of arbitrary code. Well obfuscated stuff gets through (to the tune of 10k+) all the time. Pretty sure I inherited a malware or two from there in the past

Reply on news.ycombinator.com