Official Response from JB: https://blog.jetbrains.com/blog/2021/01/06/statement-on-the-...
Any developer tools company worth their salt should be dogfooding their own build system, so they likely build IDEs with this tool. In the worst case, IntelliJ/GoLand/etc could be compromised as a result. This would be unlikely to mean there's malicious source code floating around, but it could mean lots of privileged access to software companies' networks. If the attacks are as targeted as the NY Times article makes it out to be, discovering the full extent of the damage may take quite some time ...
As a former pen tester/red teamer, your favorite CI/CD tool is also my favorite way to gain access to a large footprint within the org.
So meant it more as treat it like you do pulling in a dependency: a black box that constantly requires validation and may break your whole stack of boxes at any point
p.s. its dumb, but, thanks for coming back here and conversating. at a real low this week for sundry reasons outside the news headlines, having a small, normal, discussion where someone assumed good faith even after I failed to make a full effort to participate positively, was a ray of sunshine.
There's no reason why a merge request on a small tool repository needs access to the AWS keys for production for example.
Splitting instances into different needs. The CI/CD pipeline used for deployment shouldn't be the same one that runs tests for developers.
Especially if that CI/CD pipeline allows for the docker socket to be mounted inside of the CI/CD pipeline so that it can spin up more docker containers (as an example).
It's all about limiting scope. Far too many of these systems are not appropriately configured setup, or are exposed with far too many secrets as environment variables and the like.
Where possible also provide time limited scoped tokens. So a deployment takes ~10 minutes? The AWS creds for that deployment are scoped to 15 minutes. This way if an attacker were able to get those creds in logs or they were copied and pasted into a public paste bin, they were no longer valid by the time someone came across them.
It's all about reducing scope.
The other thing I've noticed is that the team that sets up the CI/CD tooling usually is someone that set it up in their spare time to complete a goal and suddenly it is infrastructure that is weight bearing. You want to make sure that it is maintained like the infrastructure it actually is. That means dedicated system administrators, looping security in, understanding the flows, understanding who has access and how, and what ACL's exist, how execution happens, understanding what kind of access it has.
The lack of dedicated team/updates after it has been configured is killer. Old Jenkins versions are vulnerable to various issues that allow remote code execution in the context of Jenkins itself, so even if your pipelines only have time limited tokens, generally Jenkins itself has access to the keys to the kingdom (or runs on EC2 and now I've got access to the IAM instance profile keys).
Regularly updating means that pipelines will likely break, someone is relying on that old feature that is now gone in the new version. It requires dedicated engineering time to feed the CI/CD system. That is ignored far too often.
Here's ways that the team I was on broke in using CI/CD:
- Pipeline logs were sent to an S3 bucket, which was public and we found a URL an engineer pasted into an open source ticket (the team had helpfully base64 encoded all the secrets in a debugging build, which was also in the open bucket)
- CI/CD service was outdated and we could bypass the authentication it required giving us access to update pipelines
- We could create a new pull request on an open source project that was using an internal CI/CD tool, and steal all the secrets
- Found credentials for CI/CD in the source code to trigger a build on a downstream project (creds were not scoped, and had full access)
- Figured out CI/CD tooling was not tied to corporate AD, so guessed user accounts based upon devs commenting on open source and one of them re-used a password found in a previous breach
- Found creds for a Docker registry, uploaded our own build containers that exfiltrated all of the secrets based upon their own container so they didn't even notice
- CI/CD re-used the same host for executing privileged builds as un-privileged builds, and branches were automatically built. Found an abandoned branch, pushed to it, and broke out of the container due it being privileged, and then waited for the privileged build to start to steal the environment variables for that container
CI/CD literally is remote code execution as a service. Treat it as such.
That sums it up. The code in question is just doing the job it claims to do. CI/CD services literally have forms that allow you to run direct shell commands. If a highly paid engineer doesn't understand the code they are running, then it's time to fire them and get someone in place who does.
It's laughable that we still use base64 in any sort of security context (username:password), because the only people interested in decoding it can do it effortlessly. It almost feels like a weird form of procrastination, where we know what we're doing is wrong, but we're just too damn lazy to do anything about it.
So to bypass that devs base64 encode the secrets (in this case the output from env) so that it is displayed and they can use it to debug that the right environment variables are set...
> Compromising and introducing a back door into a build environment such as TeamCity is the holy grail of a supply chain hack. It can allow an adversary to have thousands of SolarWinds-style back doors in all sorts of products in use by victims all over the world. This is a very big deal.
it read like this was much more widespread than I'm hearing now. The NY Times has even gone back and edited the title, replacing "Russian" with "widely used." I hope the editors have another take at this before it gets republished on all the other news sites ...
First, there's a WSJ article out now with some more details that appears to back him up[0]:
> Investigators believe that the SolarWinds hackers gained access to a TeamCity server used by SolarWinds to build its software products, but it is unclear how this system was accessed, according to people familiar with the matter.
Further, some rudimentary OSINT reserach indicates he is, in fact, a JetBrains employee. People can do their own sleuthing if they're inclined to verify this; I have no desire to dox the guy publicly.
[0]: https://www.wsj.com/articles/solarwinds-hack-breached-justic...