Remote Code Execution in Alpine Linux
justi.cz
justi.cz
Pros: downloading and installing packages is faster.
Cons: vulnerabilities like this.
Reading the commit message in https://github.com/alpinelinux/apk-tools/commit/6484ed9849f0... I'm not convinced they fixed it.
Do I understand this right: They fixed the apk vulnerability but the packages are still downloaded without TLS in default repositories?
There may be a comment to be made about using random distros. Alpine was a tiny distro that became super popular despite presenting several problems, docker layers making size sorta not important, and despite the recommended base image being debian.
Probably too late for that point though, and the distros popularity does suggest some of these companies should indeed donate.
"HTTPS does not provide meaningful privacy for obtaining packages. As an eavesdropper can usually see which hosts you are contacting, if you connect to your distribution's mirror network it would be fairly obvious that you are downloading updates."
The worst sort of neckbeard nonsense.
Signatures are if benefit in bandwidth constrained distribution, and for store and forward. Microsoft uses them for package distribution too (x.509 code signing). But there is no doubt that TLS makes interfering much harder.
With the move to encrypted SNI and http/2 this will hopefully become moot.
GPG signatures on packages are the correct solution, apk just did it wrong.
Somehow the term "defense in depth" (aka "layers of security") has lost it's meaning.
The problem with this thinking is that it's always true, yet vulnerabilities keep on being discovered.
TLS only provides privacy[1]/some measure of inline MITM/tamper protection in cases like this, it can't fix a broken signing mechanism.
Now, TLS, as a mechanism for establishing tamper-proof connections can mitigate some attacks, but that isn't the same thing. For example, another post here links to a packagecloud.io blog about Apt-without-TLS being insufficient and being vulnerable to many MitM attacks. For example, an attacker without TLS can just strip the signed metadata info and hand it to you, or they can do a freeze attack, so you pick up old packages (that might have security vulnerabilities). But these problems fundamentally have to do with the design of apt's update mechanism, and not the lack of TLS on the repositories. TLS is the transport layer; apt is what has to enforce the security policy (and indeed, if your TLS-powered Apt mirror has gone rogue, it can still pull off many of the same attacks anyway, regardless of TLS.) It can only mitigate the issue, not actually prevent it.
Systems like The Update Framework (https://theupdateframework.github.io/) outline a design for secure update/package management frameworks that not only don't need TLS, but also are designed to prevent many of the problems that plague Apt, as well (freezes, rollbacks, endless data, etc).
All that said, TLS is great for a number of reasons and worth using and enabling where-ever you can, of course. (It's cheap these days, and PKI in general is impossible to ignore/live without, so you might as well jump on the bandwagon and be proactive!)
[1] Arguable, since the server can just log everything anyway and look at what you download. Most public package repository systems simply are not designed with transport-layer privacy in mind...
> Whenever possible, use current official repositories as the basis for your images. We recommend the Alpine image as it is tightly controlled and small in size (currently under 5 MB), while still being a full Linux distribution. [1]
[1] https://docs.docker.com/develop/develop-images/dockerfile_be...
Debian was the recommended base image previously, and Alpine grew in popularity despite of this. The issues were more around DNS resolution based on musl not having the same bugs as glic (I believe, this is from memory).
At the end of the day, I suppose the community has spoken, and docker backed that up by hiring an Alpine developer, and I suppose the rest is history.
Not that I intend to be having a go at Alpine, all distros go through this. I was just making the point it's a fairly small distro in contributor numbers, and it seems to have become fairly important to the eco system.
TLS for downloading won't provide authentication anyway, seeing as anyone can run mirrors and serve whatever malicious content they like.
But the page that says "here's our fingerprint: xyz" would better be served over secure connection (assuming someone is not using WoT).
My laptop can do 1,400 RSA 2048 sign operations per second per core [4], 16,000 P-256 ops and 25,000 X25519 ops per second per core [5], so I wouldn't care about handshake performance either.
With large files, you don't care about the performance of the handshake, but if you do care about handshakes you can reuse http connections and use TLS session resumption for new connections, which avoids the relatively expensive asymmetric crypto (with modern servers, ECDHE and RSA digital signature). libcurl will do it, among others, so build your client using a good https library.
What is expensive about serving large amounts of content over HTTPS?
1 - https://en.wikipedia.org/wiki/AES_instruction_set
2 - https://en.wikipedia.org/wiki/CLMUL_instruction_set
3 - run "openssl speed -evp aes-128-gcm", my laptop does 4.5GB/s
4 - openssl speed rsa
5 - openssl speed ecdh
This assumes that package hosts segregate themselves by all possible privacy-related dimensions.
They don't. If you host illegal-in-country-X content on AWS and someone downloads it there, HTTPS would've prevented that from being seen. HTTP does not.
---
HTTPS belongs everywhere. It has flaws, of course, but there's no reasonable alternative at the moment. You can't predict what governments will decide in the future, and HTTP-now means they can record it and apply rules retroactively if they change their mind. This kind of thing gets people killed.
Also, HTTPS is not expensive, contrary to popular belief.
> On our production frontend machines, SSL/TLS accounts for less than 1% of the CPU load, less than 10KB of memory per connection and less than 2% of network overhead. Many people believe that SSL takes a lot of CPU time and we hope the above numbers (public for the first time) will help to dispel that.
(source: https://www.imperialviolet.org/2010/06/25/overclocking-ssl.h...)
Wouldn't it have prevented this bug?
Before launching a campaign to ban guns, focus on triage for the current gaping bullet wound.
HTTPS would be a meaningful security improvement to this system.
HTTPS would not be a meaningful security improvement, not accounting for justicz's comment. At best, it would be the same amount of security, but more annoying to maintain.
They can't do that silently anymore, though, not without getting caught. Certificate Transparency addresses that.
Citation very much needed for this claim that (publicly trusted) Certificate Authorities are "regularly compromised" while some random guy's PGP keys on his laptop are "much more secure".
You claim to be worried about "sketchy governments" so here's a nice simple exercise. List the maintainers of the top say, 100 Alpine packages and for each the countries of which they are citizens and their country of residence. Someone worried about "sketchy governments" would definitely know all this of course, since it's _far_ easier for a "sketchy government" to lean on one individual.
Next I'd like to know what Alpine is doing to ensure all those PGP keys are kept safe and that, for example, nobody's PGP key for signing Alpine packages is on, say, an unencrypted backup they took back in 2017...
If one of the CAs in the Web PKI does something they shouldn't there's a public forum for discussing how it happened, what's being done about it, how it will be prevented from happening again, and so on. Where can I find the public forum where Alpine maintainers have to confess to anything that goes wrong?
- The package maintainer (aka random guy with a laptop, aka ncopa)
- ncopa's government
With SSL, I have to trust:
- The package maintainer
- ncopa's government
- My government
- The governments of anyone along the link
- Certificate authorities
- The governments of every trusted certificate authority
- A workplace's mandatory root certificate which MITMs all browser traffic
- etc
Which of these lists has fewer threats? SSL is generally more secure than the alternative, but it doesn't add security on top of a PGP-signed package repository.
SSL provides:
- Privacy, i.e. making it harder to learn what packages you are installing, or any data about your system that can be gathered from request headers. (Traffic analysis is still possible, but much more difficult and won't disclose request headers.)
- An extra layer of defense if there are flaws in the system doing PGP verification, as in this instance (or perhaps in the HTTP protocol implementation itself). Of course, the most important thing is to fix that system, and IMO apk should continue to be regarded as insecure as long as it continues to unpack files onto the root FS before doing any verification; there are just too many ways to screw that up. Still, in practice, adding SSL would have made it much more difficult for an attacker to get to a position where they could exploit the vulnerability.
Both of the benefits above are relatively minor. But in a world where SSL is now the expected default for the vast majority of things served over HTTP, the cost of adding SSL to one more thing should be seen as extremely minor, making it easily justified.
That is not an honest way to have a conversation. Knock it off.
1. Any citation whatsoever for your claim that "Certificate authorities are regularly compromised"
2. Any citation whatsoever for your claim that "PGP keys are much more secure"
3. The list of those maintainers and their countries you're actually trusting for some reason with your current stance even though you're supposedly worried about "sketchy countries". Yes, a list, I know, you don't have one. Which is illustrative that in fact this "sketchy countries" concern was pure invention and you don't actually care.
4. The measures Alpine puts in place to protect these keys. Again in reality there aren't any, and in reality you don't care, because your concerns are bullshit. In the technical sense.
5. The public forum where I, as a third party, can see that this is all above board and not, as is the reality, just nobody cares and it's all assumed to be fine. For reference for the Web PKI this forum is operated by Mozilla, mozilla.dev.security.policy
Things you did provide:
1. A big list of non-threats like on-path governments somehow tampering with HTTPS.
That's a Gish Gallop. You don't engage with the existing debate, you just keep spewing more "new arguments" of no value, in the hope people will accept you're right.
1, 2: I didn't cite explanations for these because I explained them directly.
3: https://git.alpinelinux.org/cgit/aports/tree/main/alpine-key...
4: I don't know the particular steps ncopa et al take to safeguard their private keys, but that doesn't mean that my concerns are technical bullshit.
5: You're looking for alpine-devel, a public mailing list.
I'm not going to entertain further discussion with you if you aren't going to be civil. Ask questions and I will ask them. Acuse me of bad faith and call my concerns bullshit and I won't.
Not technical bullshit, bullshit in the technical sense.
"Bullshit" is a technical term that's distinct from a lie. Lies are told on the understanding that they're a falsehood, with the intent to deceive, but bullshit isn't interested in the veracity of the claim at all, it serves only a rhetorical purpose.
Actually X.509 keys evolved under enterprise pressure and some devices have nice additional security measures. For example it's possible to attest that a given X.509 key was generated on a Yubikey (so it's only in hardware, no copies exist) [0]. Microsoft EV code signing certs require hardware keys. OpenPGP doesn't have this.
[0]: https://developers.yubico.com/yubico-piv-tool/Attestation.ht...
Symantec has signed and provided signing certs for multiple government agencies and proxy vendors, allowing them to sign for anything. That is part of the reason Google is distrusting them.
There are many more, but a majority of certs in use today are from one of those two companies. They also have numerous re-sellers with signing certs, so the names won't always be Symantec or Comodo, hence one more need for intermediate certs.
TLS today just means the average Joe won't be able to see the specific bits you are transferring. Sorry Joe.
Symantec certainly did sell certificates to the US government, and to companies that make breakfast cereal, and indeed to at least one bona fide children's party clown. But those didn't allow them to "sign for anything". I was part of the distrust discussions you're talking about so let's see how close to the facts you were.
Here are the real issues closest to what you've described:
Issue L: Symantec signed the US government's "Federal Bridge" PKI, which is a PKI for the use of the Federal Government, a vast sprawling mess.
Issue P: Symantec issued a subCA to UniCredit (an Italian bank) which was operated incorrectly and without an audit until October 2016
Issue T: The Korean firm "CrossCert" and three other Symantec partner companies were able to issue certificates as Registration Authorities, although Symantec was not effectively overseeing this activity. Non-compliant certificates were issued, mostly for Korean firms, and Symantec says CrossCert did not have any paperwork for these certificates, which Symantec should have known about back when the certs were issued.
Issue V: Symantec's "GeoRoot" programme provided subCAs under technical control to five companies to issue their own trusted certificates within certain constraints. These five were: Aetna, Apple, Google, Intel, and Unicredit (but see P above) and the audit paperwork for these subCAs was always reprehensible garbage, an unnamed root program (probably Microsoft, given that Apple and Google are on the list) was pressuring Symantec to fix this, but they had not done so before Symantec got themselves distrusted.
Issue Y: The "VeriSign Universal Root Certification Authority" CA has a couple of subCAs that are not themselves constrained to prevent issuance in the Web PKI but are also not sufficiently audited or subject to oversight. Symantec swore these subCAs aren't actually used in the Web PKI, but without a technical constraint that's hard to enforce.
Back when Symantec was operating a CA, the two companies together (all of Symantec, including Versign and other brands, plus Comodo) had than 40% of "market share" in the top 10K web sites, and far less in the deeper corners of the web. Most certs are from Let's Encrypt and that's still true today.
Because of how modern TLS works, bad guys would need either to use a large quantum computer for each individual session they want to break (problem: those don't exist) or they'd need not only a publicly trusted certificate, but also a live active attack on the session as it happens, which would of course be detectable.
I think you're conflating two incidents, one from 2008 when ZF0 broke into a poorly secured web forum on Comodo's systems and one in 2011 where "ComodoHacker" gained access to Comodo's issuance system because they used a single factor password login system, and was able to create a bunch of certificates for themselves using the account credentials they had.
You might even be further conflating problems at DigiNotar, which "ComodoHacker" also claimed responsibility for, where Iran was using certs DigiNotar had no record of to MITM its citizens. DigiNotar is gone, the company is defunct, that entire hierarchy was distrusted seven years ago.
You may ask "well why not just use PGP if it's not going to be publicly signed" and to that I would say "so the attacker can't even establish an HTTP session, further reducing the attack surface".
And for the record, the initial iso download is secured with SSL. I can also establish trust through my other Alpine installs, with friends who have also installed Alpine, via magnet links from friends (which are inherently sha'd), by emailing several package maintainers and cross verifying their signatures, consulting keyservers, etc. However you do it, once you establish the initial trust PGP is a superior option to SSL.
Did you check and sign Alpine's maintainers key yourself or do you rely on the Web of Trust to create a path between you two? In the latter case every person in between of you two is acting like a certification authority.
Again, CAs are only needed for PUBLICLY signed certs but we are talking about a PRIVATELY managed trust and repo. The repo cert can be self-signed by the Alpine group and added to the root store during install the same as a PGP key can be added during install. No 3rd party "trust" involved.
For maximum security and user experience I'd say both though - validate you are connecting to who you think you are and also validate the PGP signature. For how often packages are installed vs how high of a risk remote code poses to the system I think it's a bit foolish to say we can only do one and when we do we are perfect.
You can also always run your own CA and trust it for the purpose of the APK installer or pin the CA or some other method, it would only be a problem if a CA got compromised (which I doubt is "easy" as you say, esp. if you restrict which CA's are allowed) AND APK had this bug.
SSL isn't unbeatable security, it has holes, but it can help prevent a number of issues, such as the one in the OP to become critical at all.
Similarly, PGP signing helps against a certain class of attacks.
Both together cover a wider attack scenario than either alone.
A software exploit could undermine literally any secure system but in this case the addition of SSL would have prevented this exploit from being dangerous to a huge number of people.
But I think it's reasonable to talk about security systems by giving the developers the benefit of the doubt that their code doesn't have exploitable bugs. My main point is that, assuming it's implemented without bugs, PGP is more secure than SSL, and that SSL alone would be worse than PGP alone. Combining them would be an improvement, but I'm rejecting the idea that the current design is insecure.
Having the package maintainers personally tell you their package's checksums would have also prevented this problem, but it's clearly not practical. Sometimes SSL isn't practical. If you can design a system which is secure without it, it is an improvement, but not a requirement.
The problem with this assumption is that available evidence points towards software inevitable having, developing or gaining bugs in the implementation (through existing code, through new code or through new threat vectors respectively).
The current design isn't insecure but it could be more secure at little cost. In current times, SSL is fairly practical and easy to use, a lot of popular webservers either have or are developing ACME certificate mechanisms (including nginx and apache to my knowledge).
Due to the low cost, designing SSL into the system should be a requirement since it's an improvement of low cost with great effect.
That's a wonderful endeavor and I would gladly try to "convince" people I know to pledge money in it.
Monetary incentives are important for open source maintainers, as they cannot always afford the time to work on their projects.
For example, what is there to prevent someone from introducing a bug into an open source project, first of all, and then subsequently claiming a bounty for identifying and/fixing it?
(Maybe this can end up by forcing more careful pull request auditing for security critical projects, but that seems like a big ask..)
In case somebody is wondering about Alpine Linux based postmarketOS, this is what I wrote on /r/linux regarding this issue:
postmarketOS is directly using Alpine's apk-tools package from their repositories, so a simple "apk upgrade" will install the latest apk version where this is fixed.
Regarding the pmbootstrap tool we are using for development to set up Alpine Linux chroots, just update to the latest git version with git pull as usually, and it will make sure that the apk version is installed where this is fixed before you can use any of the chroots. Pulling from git may seem strange, but we do not have a stable release yet for pmbootstrap, so this is the normal way to update the code anyway. We're getting closer to a stable release of pmbootstrap though.
Docker just makes it super opaque when you have to rotate images.
Scenario: Anonymous docker publisher installs a kiddie porn i2p or tor downloader service hidden in a container for Apache Kafka. Random corporate trainer accidentially picks said container, blogs a howto with it and suggests its use in in-person and web-based courseware. IANAL but Who's liable? The end-users using it? The trainer? Docker?
Disclaimer: I love Docker, but it needs some chain-of-custody assurance and greater visibility of container contents.
If all that isn't enough, individual image layers can be accessed and inspected by hand (they're just tar files).
If you're looking for "end-to-end chain of custody" you're going to need to specify the ends. Image signatures get you point-in-stack authorship assertions. I'm not sure what else anyone could ask for here.
EDIT: Just looked into that.. it looks like just the java 8 containers.
ERROR: ${package_name}: BAD signature