Codecov breach impacts GoDaddy, Atlassian, P&G etc.
reuters.com
reuters.com
1. The move to cloud means the "edge corporate firewall" monitoring point is forgotten about. Devops infrastructure and similar is now running off-prem in the cloud, and the perimeter is now being written off as "zero trust" (but without actually adopting the right assumptions there)
2. The move away from fewer, better tested, releases means we see more risky distribution methods (curl-bash without hash pinning etc), and an encouragement to always pull the latest from upstream, rather than to internally vet and approve a pinned release.
3. A general trend towards more and more third party dependencies and external connections to enable basic functionality, meaning that everything assumes it runs in an internet-connected environment with DNS resolution and the ability to make requests to any hostname and IP the software chooses.
It feels like once again, as other commenters have pointed out, basic sandboxing principles and isolation would prevent this being an issue, coupled with internal vetting of what is being executed in sensitive environments. If the environment is properly limited to read-only access to code, your biggest risk is exfiltration, so you should be focused on preventing exfiltration of anything (environment, state, code, secrets), rather than leaving the barn door wide open.
In the same way we do W^X in memory, to prevent an exploit writing to memory, then executing it, perhaps we need data^network - you can either have internet access, or access sensitive information, but not both. This breaks the X-aaS model, but after SolarWinds, this, and many other such failures, maybe it's time to really re-think the wisdom of giving third parties just so much access to core systems.
In other words, it will have to get worse before it gets better.
If investors lost confidence in a company, and sold their shares (impacting the share price), could this create a sufficient paper-based loss for other investors to pursue the senior officers of the company?
I think you're right though - unless it is something that makes the CEO lose their job, or go to prison, it is going to be hard to get anyone to take on any (even marginal) increase in the cost of delivering the product.
(I will point out that their uploader is literally a curl | bash, but that isn't really the problem here. Installing a Debian package or downloading a statically linked Go binary would be just as risky. Actually, it would have been harder to find the underlying problem.)
All in all I'm pretty disappointed. We trusted Codecov with a shortcut that makes it easy for them to deliver a better product. They didn't have any security in place to ensure that what was checked into source control was what customers downloaded, and now we're all hacked.
The Python one, for example, is downloaded via PyPI, and you’ll usually have a pinned version with the hash of it. Any tampering would make pipelines fail. If a new release were pushed by rogue actors, someone would notice an unplanned version bump.
Here's a snippet from a support thead with them last year:
>One thing I noticed is that you are using the python uploader to upload coverage reports to Codecov. We do not support this uploader as a first-party tool, but we do support the bash uploader (https://docs.codecov.io/v4.3.0/docs/about-the-codecov-bash-u...) with full-time staff. We want to make sure this issue isn't related to something in a third party uploader before troubleshooting further. > Would you be able to switch to using this uploader to see if it's related?
I'd stopped using codecov last year for pricing reasons, when they moved to per-user pricing.
(And honestly, an attacker that can build a Debian package and update the public key for apt-key is probably harder to find than one that can edit a cleartext bash script. Security through hoping your adversaries can't perform tedious devops operations is no security at all, as they say, but in the real world it probably helps a tiny bit.)
The "bash uploader" that was compromised is designed to curl and run a script on codecov's server. This script was compromised to in turn silently run a script from the attacker's server. This could have done anything but I doubt they just dumped ENV, probably opened a reverse shell to literally do anything they want as the user that originally invoked the script. This could be anything from stealing ~/.kube/config and ~/.ssh/id_rsa... If one of the 29,000 customers invoked as root it could be even worse, such as adding users and compromising entire platforms.
https://github.com/search?q=https%3A%2F%2Fcodecov.io%2Fbash&...
But yeah in general secrets management is a trainwreck.
I guess the protected branch segregation should go -both ways- and more effort should be made into auditing and vetting which CI step has run access to secret-authorised branches.
Like:
- always run all dev tools in a sandbox, including the compiler and language server
- proper secure (harden) developer systems
- don't install software outside of sandboxes which is known to be often less careful with security vulnerabilities, like steam (or games in general). Firewalls can help wrt. offline only games, but due to things like drm, invites, game updates etc. most modern game software is not offline only.
- split CI into components, make sure the part which builds (and potential deploys) artifacts doesn't run in the same sandbox as additional analysis tools. Potentially run different analysis tools in different sandboxes. Don't give CI sandboxes permissions to directly push to your repository etc. If a tool need to be able to push to git consider limiting it's access to a specific folder or if not possible sub-module (which tbh. are annoying).
- Limit internet access of CI sandboxes as far as possible.
Sadly some of this things are quite cumbersome or even impossible to setup with (at least non enterprise) github.
All of this can be set up in-house with open source tooling, there really is no excuse. Gitlab CI with your own runners that are forbidden from direct Internet access, Sonatype's Nexus to proxy Docker images, Debian/Ubuntu APT, NPM and Maven repositories, and stuff like Sonarqube on Docker containers with a no-internet configuration. For stuff that absolutely needs Internet access (PHPs Composer comes to mind, sigh), set up a caching Squid instance.
You don't even need VMs for that any more, a single machine with Docker running is enough.
Yes. Absolutely yes.
GitLab (CI) is SO. GOOD. Use it! It's incredibly flexible and powerful.
- One thing I'm missing to ditch Jenkins permanently is general purpose job stuff... like "synchronize the production database to integration". Yes there are workarounds but all of them are ugly.
- Developing pipelines in Gitlab CI is tedious at best. With Jenkins I don't need to switch between applications during development, with Gitlab I need to switch between browser (to see logs), command line (git) and an editor for the pipeline.
- Holy hell it's dog slow sometimes. A runner with Docker executors always builds a whole new instance for each goddamn step of the pipeline, ALWAYS. Jenkins is intelligent enough to re-purpose executors.
We had a job that backed up some data to a git repo and the gitlab ci cronjobs were really easy to use. It was a simple java -jar, git pull, git commit, git push script and tbh it couldn't have been simpler.
Oh, I have a wonderful setup consisting of just 30 different tools. Let's start with proper Kubernetes cluster...
Fwiw, you can do it all in-browser with gitlab's editor.
You use cloud services because they don't cost so much time and have a fixed amount of money (normally) that you can sink into. Also, there are people who fix the problems with the product. You can't just do everything on your own if you want to focus on your product.
Management at a former employer quite rightly freaked out when they realised that their devs on a certain subcontinent were routinely pasting their code into external web-based prettifiers to format it nicely...
1. JSON prettifying
2. JWT decoding/verification (bonus for "paste your signing key and we will generate JWTs for you too!")
3. PEM <-> DER conversion
I'd love to see "standard" prettifiers for HTML, CSS, JSON, YAML, and TOML. The required libraries exist and the problem isn't terribly hard if you aren't picky about corner cases. I'm sure countless programmers have written such tools already. Where is the disconnect?
There's a CLI linter for pretty much every language I've used, but how would you prettify a valid YAML file? Any reformatting will change it semantically.
(1): Which can be necessary to prevent accidental problems like NO defaulting to false not "NO" and similar. I personally strongly recommend against anything where string quotes are optional for human written/edited serialization formats (and comments are a must have).
I never understood why anyone who is remotely serious about security would run something like Codecov within their walls. Maybe an air-gapped install, but I don't recall that even being an option.
The level of access they get is too much for a secret-source project, especially for the little returns.
How is the risk lower? It's basically an RCE vuln which can poison open source artifacts. Massive blast radius.
I could almost see this work in private repos, albeit with an authenticated fetch token - I'm sure I've used a "deploy-only" key before with both GH and GL to do an authenticated pull. The repo would then be processed, and a response posted via commenting API.
Am I missing something here in why it's necessary to involve Codecov in your build/CI stage itself? Is it truly doing dynamic analysis, or is this some (in)convenience factor that could be eliminated? It just strikes me this could sit entirely on the side, without access to anything (beyond the source itself). For enterprise, you could spin up an on-prem instance, and give it the API tokens for authenticated pulls etc.
https://github.com/codecov/codecov-bash/pull/426
Days later and no reply except for the goddamn autogenerated codecov report.
Our company is currently looking into alternatives to Jira due to Atlassian going the cloud-only route. One of the must-haves for whatever replaces Jira is a possibility for an on-premise installation. It's still early in the decision phase, but a surprisingly large number of candidates has already been ruled out due to this seemingly-simple requirement.
I'm sure there are companies out there where this is possible, but most of them (I'd wager it's more like 99%) won't be able to match cloud-based solutions.
CI and GitLab infrastructure are the same story: run on premise on our own hardware for a decade, with only very minor downtime. It has also evolved from simple isolated Jenkins instances with manually configured build jobs to a shared build/CI cluster using containerization and a lot of automation for build job management.
And it's not that we have an abundance of administrative people. A team of three (sometimes being temporarily buffed up with a few more if large infrastructural tasks are planned) for about 200 developers in total.
I have a different suspicion: a lot of companies out there have simply forgotten that they are perfectly able to run such services for themselves. Having people telling them all the time that they wouldn't ever be able to do that anyway (with a lot of them having their personal agenda, for example being employed by a cloud service provider) doesn't make things better in that regard.
Years ago, many companies ran similar services in house and having downtime wasn't a common scenario.
Well let's see how this unfolds and who's behind this, maybe we get some nice surprise-opensourcing of interesting stuff. /schadenfreude
Specifically,
> Relying on a curl'd bash script downloaded off your server seems inherently vulnerable (though this apparently applies to many vendors in the space apparently). Checksum procedures should be built into your docs, or you should be using a package registry (ala codecov-node).
Even when (if, really) people check for hash, do they check that the hash was published and fetched through a second channel? Nope, people run arbitrary code unchecked all the time, so they're just as equivalent.
Even git clone can be dangerous. Do you audit all the repo branch names before cloning? Because depending on how your shell prompt shows e.g branch names it could be used to RCE.
EDIT: by this I don't mean to excuse curl | bash and such, I just mean to highlight that the issue runs deeper, curl | bash being the unburied part.
Well, it is, because any .EXE or .MSI or .DMG I download will pass through one or more virus scanners. You can also do this by forcing users through a proxy such as Artifactory that can also do scanning of Linux packages. But it's futile to rely on automated solutions for NPM or PyPI where there is no curation of the repo so anyone can do typosquatting.
SolarWinds, the Github actions thing, this.
This is a hard question, basically high code coverage does in no way imply that your code is secure (or correct working wrt. non security aspects) at all.
But at the same time not having reasonable good code coverage can often be an indicator for code being potential not that good, especially assuming non-malice of a author group I would start with reviewing the parts of there code with little code coverage (after reviewing parts which are common to be messed up, after running automatic analysis tools).
The important thing to realize is that most coverage is in % of lines covered which is even by 100% coverage far far less then 100% possible execution flows. And not seldom security vulnerabilities (instead of "just" normal bugs, assuming a security aware dev) are tricky unexpected execution flow (parts) in unexpected combinations through lines which as well might have 100% line coverage.
So what I would say in general:
- Don't get fixated on any specific number of % of lines with test coverage, especially not 100%.
- Focus on writing "good" tests instead of tests with high increase in coverage.
- Be security aware wrt. supply chain attacks and similar, like due to above's points I would argue that due to the points above in general there is no need to run coverage analysis anywhere but the CI, in which you could put it into a different container, i.e. separate sandbox CI stages for building artifacts from such which just do testing and/or analytics, don't ever allow CI write access to your repository directly.
Just an assumption though.