How I gained commit access to Homebrew in 30 minutes
medium.com
medium.com
How is PGP signing not a no-brainer. What kind of workflow concerns would prevent them from signing commits?!
https://www.youtube.com/watch?v=AsNwon4fjqY
A publicly available webcam pointed at an RSA SecurID hardware token...
(The optimist ion me hopes this was performance art. But I've worked with people who'd do that if it made their day ever so slightly easier...)
1. Is the identifier mutable? Make sure it points to a content addressable identifier (SHA2), and sign that link. 2. Is it a content addressable identifier? Nothing to do.
When it comes to signing in git, signing tags is usually where you see the most value (mutable identifier that points to a git tree, which is content addressable).
You’re just trying to improve the trust in saying “Hey, v1.2 is this SHA digest”.
The problem isn't making it easy to sign things, the problem is making it sufficiently hard for unauthorized parties to sign things without affecting any workflows you'd like to preserve - that is, the real problem is a workflow problem. The real problem is figuring out how to secure automation so it has the privileges to do what it needs but isn't leaking access. The Jenkins instance was designed to do authenticated pushes - it needs automated write access to the Homebrew repos.
Also, signing commits doesn't help you if the risk is unauthorized pushes to master. You can pick up someone's test commit and push that to master, or push a rollback of OpenSSL to a vulnerable version, or something, and still ruin many people's days.
If I push a pull-request to master that wasn't approved to be on master (e.g., a maintainer did a build of "disable signature validation to narrow down why tests are failing", and signed that commit and pushed it to a PR with the intention of rejecting the PR), then I also have a valid and completely signed "history", and it probably isn't even a force-push to get it onto master.
Git commit signatures authenticate exactly one thing: that at some point, the holder of this PGP key committed this commit. They say nothing about the suitability or future suitability of that commit to be used for any purpose. They don't solve the problem Homebrew had here, and they cause other problems (like breaking rebases).
Git tag signatures are significantly more useful, since they include the name of the tag. So you're not vulnerable to the second attack, and you're mostly not vulnerable to the first since a client wouldn't intentionally request tag 1.0 after getting tag 1.5. But you still have the problem of the client knowing which tag is current, and frequent tagging isn't a great replacement for a workflow where you want people to follow master.
Also, how secure is the kernel on the air-gapped machine against malicious filesystems on the USB stick? (If it's running Linux, the answer is almost certainly "not;" I could imagine FreeBSD is better but I don't know how much people have explored that.)
To be clear I'm not opposed to air-gapping if the maintainer is excited about it, I just suspect there are many much weaker links on the way to/from the air-gapped system, and fixing those is a much harder project that almost nobody is excited about.
Or a virtual throw away machine just for mounting a USB filesystem, extracting the files, place them in some dump directory and then the whole virtual machine is wiped.
Heck, the virtual machine for copying should run BSD. :)
I verify that the source code being compiled is the source code which is published in a signed tarball. Yes, someone could have tampered with the internet-connected system where I do Tarsnap development, but their tampering would be visible in the source code.
Build dependencies are verified to be the packages shipped by Debian. If someone has tampered with the Debian gcc package, we've lost even without Tarsnap binary packages.
Also, how secure is the kernel on the air-gapped machine against malicious filesystems on the USB stick? (If it's running Linux, the answer is almost certainly "not;" I could imagine FreeBSD is better but I don't know how much people have explored that.)
I don't use a filesystem on the USB stick I use for sneakernet, for exactly this reason -- I write and read files to it using tar. (Yes, you can write to a USB stick as a raw device just like you would write to a tape drive.)
For secure airgapped computers I'd use one way low tech comm channels with no side bands, maybe IR or sound? (if you trust the device drivers)
For small and intermittently-active ones, the primary development constraint is the OSS maintainer having free time, and doing dev or builds on an air-gapped machine is a big time sink. I maintain very little open-source software and I have still tried to at least use a separate machine for builds + PGP signatures and not my day-to-day machine, and maintaining this machine is just overhead that eats into my, what, one evening every two months that I get to spend on my project.
The solution I'd like to see is a) better tracking of community code reviews/audits; if every line of code has been read by multiple people (perhaps identified with a PGP key, but something simpler would be fine), you're more confident that there are no subtle backdoors than if you make one person stare at the entire diff between major versions on an air-gappped machine until their eyes glaze over, and b) better ways to do builds on multiple clouds and verify that they're identical. The Reproducible Builds effort is a good approach here; if you do that plus a CI infrastructure that runs on two different clouds with two different signing keys, and client systems require both signatures, you can be reasonably assured that the build process wasn't compromised.
The article makes a good point that it's very hard for small projects, like the team running Homebrew, to fund their security, yet they are likely to be a target for quite high end attackers, given the access that can be gained by getting unauthorised access to package repositories.
As a side note it also shows that Jenkins tends to be a tempting target for attackers as it often has access to a wide range of systems to carry out it's functions.
This. I'd really love to stop us using Jenkins but none of the hosted macOS CI services scale to meet our needs (i.e. having jobs that run for multiple hours on better hardware). The ideal solution for me would be for us to have some sort of modified Travis or Circle CI setup. We can even pay for it now we've got Patreon money coming in.
Not affiliated, just a happy user.
- pipelines stored in source control
- hosted on your own servers (with easy one click setup in AWS if you're too busy for all that)
- excellent cloud hosted UI
Also not affliated, just a happy user who replaced a large jenkins setup where I work with buildkite.
It's just often misconfigured because there are a plethora of plugins and ways to store and use secrets, and nobody audits it enough to look at the console output or build artifacts for leaked secrets.
The project is simply missing someone familiar enough to configure Jenkins properly.
I would say that this is a specific case of the far more general one. Substitute "any small organization" for "the project" and substitute any configuration familiarity (i.e. Ops skill) for Jenkins configuration familiarity.
Even in "Devops" job postings for startups, when mentioning CI/CD tools like Jenkins, the main desire seems to be to hire someone who's more Dev than Ops, to create the code to run the CI/CD pipeline, with something like configuration or security a mere afterthough, if that.
I disagree. Homebrew’s security considerations in this case have nothing to do with their funding. There are lot of terrific services available for next to nothing for open source projects, Jenkins is one of them. It must have been a conscious decision the way HB set up their CI, unaffected by funding.
And such lapses are not an open door jyst to “high end attackers”. This was a single person with just internet access and a little knowledge about how modern OSS projects work.
If they had better funding they could have hired dedicated security staff, who would likely be versed in the ways of securing DevOps pipelines.
Time is money, more funding == more resources to look at security properly.
The cost of good security is high - audits, slowed down development, limited data retention, higher compute costs... and the return on that investment is only ever going to be a reduction of liability.
Big company with lots of resources, small company with no resources; it doesn't matter. Security is a cost center, and will only ever get a token amount of resources until the costs of doing nothing outweigh the costs of doing something.
Yes, security is a cost. There's a bit of tragedy of the commons effect here - many of the downsides are pushed onto others. I like Doctorow's general take on socializing costs of privacy and security breaches while privatizing profit: https://locusmag.com/2018/07/cory-doctorow-zucks-empire-of-o...
A policy in tech companies to grant a given amount (let say 50$) per month to each employee, the employee can attribute this amount to whichever OSS project(s) he sees fit.
If the employee doesn't chose a project, this amount is given to OSS projects in need.
To streamline this, the ideal would be to have a service where everything could be done:
* OSS projects register to it
* Tech Companies then give the money to the service, which then dispatch it to the OSS projects
* Employees of tech companies logs into the service to select which projects they chose to give money to (with maybe some suggestion to avoid over/under funding projects).
Let's make that amount dependent on the employee's salary. Not 50$ per employee per month, but e.g. salary for 1 hour per employee per month. Salaries vary a lot between countries (and even within countries).
As someone who runs a small open source project that clearly states that Patreon donations are accepted and even offers some convenience benefits to donors, I often see people jumping extra hoops to avoid it, including clearly profitable companies that saved many thousands by using my software, spending extra time that would amount to more than those donations. So far I get donations from less than one percent of my estimated users.
PS: are EcmaScript binaries actually a thing?
Facebook, microsoft, github, etc all pay $$ and our time into a pool that is used to incentivize the finding, vetting and fixing of security flaws in major software running the internet.
https://www.propublica.org/article/the-worlds-email-encrypti...
for example (at least that one has a happy ending - where after the right exposure a bunch of companies benefiting from his work stepped in to fund it...)
I know this is kind of a shameless plug, but this is the exact reason I launched BountyGraph [0] last week. I think that we can crowdfund security budgets for these projects to help encourage the discovery of issues like this one.
Kudos on that. Specially when the all team make only around $700-800 per month. [1]
Power Mac is a different story.....don't know how and when they're going to fix this. That's a real shame.
Yeah literally all many of us want is a standard keyboard with function keys instead of the touchbar. A couple regular USB ports would be nice as well so we can plug in a mouse and keyboard without needing dongles. Neither of those things would turn their "pro" laptop into a brick.
MacBook 2018: USB-C dock with replaceable cord
So in my use case I actually got rid of one dongle at a price lower than a replacement MagSafe 2 adapter. I really liked MagSafe though.
Never used the F-keys really. ESC is not that hard for me to live without but that's all personal. But the difference in performance is quite noticeable.
The Surface line by Microsoft has a very similar magnetic power+docking connector. The only issue I have with it is that the LED doesn't change color when the device is fully charged.
The line includes:
- Laptop,
- Pro (tablet + typecover)
- Book (tablet + keyboard + GPU + battery)
Unfortunately, many people do that. For example, here in New Zealand there is a new startup Onzo (bike sharing). When they launched, I tried to google about them (just out of curiosity). And in 1 minute I found their Jenkins server exposed to everyone as well. I could see how their build process works, who commits what etc. I decided to try to login using simple credentials (something like admin:password) and it didn't work. But there was a Register button. "Why not?" I thought and clicked it and created my own account. And voila, it gave admin permissions by default - I could delete their projects, change variables etc. Emailed them about that.
Moral: never expose your build/deployment systems. If you really want to expose some parts (for whatever reason), then use/write client/UI that has no permissions. 'Build Status' badge is a good example -- it exposes build status info, but doesn't show too much and doesn't give any permissions whatsoever.
As said in Raiders Of The Lost Ark, bad dates.