CircleCI says hackers stole encryption keys and customers’ source code

There should be full transparency on this and all the open source eyeballs should be able to study and scrutinize this. If not, anyone will be able to get away with data theft tomorrow saying, "my server got hacked". What will cause them to not do it except some sense of personal ethics which is rapidly degrading these days?

Are other companies any better? It seems like now it’s assumed that all of these companies will eventually be hacked, so if you use them you need to have systems in place to mitigate damage. And if you don’t use them them then you need to have mitigation strats anyway. Basically either way you’re screwed.

Defending against internal threat seems like a losing battle that can only be mitigated, slowing down more than preventing an attack.

Circleci's business model is hosting CI runners for you, so of course they need to be able to decrypt the data, and if you have access to dept new infrastructure, you likely have permission to read encryption keys (or deploy new infrastructure that can read said keys and then use those machines to get the keys).

What CircleCI _have_ done is set up enough logging and auditing that they were able to figure out who was compromised, how they were compromised, the time frame and the resources they accessed, which IMO is about as much as you can ask for.

You cannot shutdown access to resources as developer need them for productivity. Secondly, your competitor is most likely going to take risk of hack and move faster in bringing features while you reduce productivity.

Sort of.

There are ways to verify both the device being used to access a resource, and the account. This is a good use for TPM. One of the features is attestation, which can be used to do this sort of a thing.

For things like crypto keys, this is why HSM (hardware security module) exist. - they make it very difficult to get access to private key (basically, private key doesn't leave the HSM itself, but other cryptographic artifacts based on that key do).

You can also require an authentication loop for each major service, so only services the developer is authenticated to would be vulnerable to credential stealing.

And for critical infrastructure network security would help (e.g. a vpn). With hardware tokens required to initiate the connection, the attacker wouldn’t be able to spin up their own connection.

Finally, access to user secrets shouldn’t be a default level of access for engineers. Instead, that should require a break glass escalation of privileges, reducing the risk of pilfering.

The step-up authentication CircleCI mentions is probably more of a sudo model, with a long-lived baseline level of privilege that can only be elevated for short bursts. This is orthogonal to MFA, but it's at least less of a nuisance with a hardware token than with other options.

> and that site doesn't somehow restrict the session token to only being used on the machine that generated it

One simple practice for security-critical systems is to bind sessions to the client's IP address during authentication. It's not bulletproof given the assumption of malware an attacker can still tunnel through, but neither is locking the session to the device. You could, but the malware is also on the device and can do anything the user can (this is why it's preferable to test user presence outside the device by e.g. tapping a USB key).

1. The engineer was prompted to give the malware (PTX app?) access to the browser key, and agreed to this. Big mistake if so.

2. The malware has an exploit for macOS security.

3. The malware has a way to take over the browser cross-process.

Hardware security keys don't help in this case. What helps is either users not granting malware access to critical secrets, OR, operating system security being enhanced.

A lot of production AWS/GCP keys are likely stolen for those that deploy from CI/CD.

Security is pretty lax at most companies, and the more employees you have, the worse it is.

In this case, I imagine that CircleCI's controls involved developer workstations having anti-malware software, logs and audit trails for access to production systems, and some kind of intrusion detection system around those, and some SLAs and policies on how they react to alerts from those systems. It sounds like they had all of those things in place but the malware wasn't detected and their IDS didn't pick up the external access. Unfortunate, but both of those are entirely possible in many environments. SOC2 can't guarantee that your anti-malware systems or IDS are flawless and can't be bypassed by a clever attacker. Otherwise, since they've been able to identify what the attackers gained access to, it sounds like they did have logs and audit trails in place.

SOC2 auditors typically only have a limited understanding of security and technology themselves (the higher end ones might have more resources available to them) and are only able to sample a small amount of data during the observation window.

If you're trusting your sensitive data and credentials to a 3rd party, you should definitely require that they have SOC2 or better, but you still have to own your own security and consider how you need to protect yourself if/when they are compromised.

The method of attack sounds like CircleCI's production cloud (probably AWS) was impacted - "the targeted employee had privileges to generate production access tokens as part of the employee’s regular duties, the unauthorized third party was able to access and exfiltrate data from a subset of databases and stores, including customer environment variables, tokens, and keys."

But I am surprised that their SOC2 auditors didn't raise exceptions about their lack of controls. Sounds like a pretty immature program, they only talk about 2FA, MDM and SSO which is basic stuff. Where is the SIEM? Or CSPM? Or any alerting!? Yes there are SOC2 automation platforms out there that rubber stamp stuff, but at CircleCI's scale I'd expect more scrutiny.

> ...

> Review GitHub audit log files for unexpected commands such as ... repo.download_zip

I don't know what you'd get from GitHub other than source code but the CircleCI blog post explicitly describes the attacker downloading entire repos as a .zip

https://circleci.com/blog/jan-4-2023-incident-report/

> Updated headline to better reflect the customer data that was taken.

Probably should be updated on HN too.

That doesn't work in environments with multiple NAT origin IPs in place, or when they're using crap like Netskope/some other "security"/"privacy"/"VPN" software, as IPs tend to randomly change with these. It would generate way too many false-positive reports.

> One problem with this is that client IP addresses are easily spoofed[2].

Only if the backend servers are badly set up. For me, I always run haproxy as the frontend and forcibly delete incoming headers (X-Forwarded-*, Forwarded), and as an added precaution the backend software is configured to only trust the haproxy origin IPs - so even in the case an attacker manages somehow to directly access the backend servers directly, they cannot get a spoofed IP past the system.

> So, instead of storing the client's IP address; how about we instead store the clients' SSL fingerprints

That requires client-side SSL authentication, which is theoretically supported by all major browsers, but very rarely used and the UI support is... clunky at best.