Comment by f430 | Hacker News Reader

Comment by f430 | Hacker News Reader

This is the key excerpt, its quite shocking:

    Crowdstrike said Sunspot was written to be able to detect when it was installed on a SolarWinds developer system, and to lie in wait until specific Orion source code files were accessed by developers. This allowed the intruders to “replace source code files during the build process, before compilation,” Crowdstrike wrote.

    The attackers also included safeguards to prevent the backdoor code lines from appearing in Orion software build logs, and checks to ensure that such tampering wouldn’t cause build errors.

    “The design of SUNSPOT suggests [the malware] developers invested a lot of effort to ensure the code was properly inserted and remained undetected, and prioritized operational security to avoid revealing their presence in the build environment to SolarWinds developers,” CrowdStrike wrote.

So how do we guard against this type of attack? How do we know this hasn't already happened to some of us? What is the potential fallout from this hack, it seems quite significant.

This must be why the Japanese Intelligence agencies prefer paper over computer systems. The digitization of critical national security apparatus is the Archilles Heel that is being exploited successfully. One example is Japan's intelligence gathering capabilities in East Asia, especially China, which is bar none. Japan has a better linguistic understanding of the Chinese language (Kanji and all) but also interestingly much of PRC's massive public surveillance equipment like CCTV cameras are made in Japan.

Even if they hire Krebs, I believe that if its digital, it can be hacked given long enough time period and unlimited state level backing and head hunting essentially geniuses of their country to do their bidding. I wonder how Biden-Harris administration will respond, it is very clear who the state actor is here. I'm very nervous about the ramifications of this hack.

We're all screwed. Predicted long ago.

See _Reflections On Trusting Trust_ [1]

[1] https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...

> So how do we guard against this type of attack?

One big issue with a lot of security and enterprise ops tooling is that it doesn't follow good practice around, well, security. For example, security code analysis software with hard-coded passwords that you hook into your build tooling, or in this case, ops software that instructs you to disable Microsoft's security tools so they don't flag problems with the agent.

In a similar vein I've had BMC software want the highest levels of access to Oracle DBs to do simply monitoring, and so on and so forth.

The other observation I heard Bruce Schneier make at a hacker con is more profound, and probably going to take a lot longer for national security actors to accept is this: the norms need to change. There is no longer a clear separation between "our stuff" and "their stuff" the way that there was a few decades ago, when espionage was more on "your telco network" or "my telco network". As we've moved to pervasive connectivity it's no longer possible to say, "oh that backdoor will only be used by our side", or "that hack will only affect their SCADA systems" or whatever.

I think this is the best answer out of all the replies.

>So how do we guard against this type of attack?

Only allow restricted CI servers to build and deploy production code.

Whitelisted list of software and security monitoring for CI servers so malicious software is harder to install.

Whitelisted list of software that developer machines can run (ie: no arbitrary third party code). Restricted docker containers for all local testing. There's a reason various companies do not give developers admin access to their machines.

edit: This or related concepts (SDLC, audit logs reviewed weekly, security team sign offs on system changes, etc.) are in my experience pretty standard in large enterprise security reviews for vendors that were high risk. The issue is that everyone probably lies and doesn't actually do any of these things even if they say they do. The incentives are set up to certify vendors rather than fail them.

> So how do we guard against this type of attack?

You can't ever prevent it, but you can raise the attack cost/complexity and make detection much more likely.

Go for immutable infra and transient CI/CD systems. Provision the build nodes on demand, from trusted images, and wipe them out completely after just a few hours. The attacker will have to re-infect the nodes as they come online, and risk leaving more network traces. Anything they deploy on the systems goes away when the node goes away.

The attack against SolarWinds worked so well because the CI system was persistent and it was upgraded over time. For a reliable and secure environment, the correct amount of host configuration is zero: if you need to modify anything at all after a system has been launched, that's a bad smell. Don't upgrade. Provision a completely new system with the more recent versions.

This kind of architecture requires the attacker to compromise the CI image build and/or storage instead. (Or the upstream, directly.) It's an extra step for adversary, and a simpler point of control to the defender.

Recon. Compromise. Persist. Exfiltrate. -- As a defender you want to make every step as expensive and brittle as possible. You can't prevent a compromise, but you can make it less useful, and you can arrange things so that it must happen often and leave enough traces to increase the probability of getting caught.

Another thing that might have made this hard - if Solar Winds were distributed as source code and each client built it themselves, with their own options (though the old back-doored c-compiler "thought experiment" may not be as much of a thought experiment anymore).

Moreover, achieving the hack was likely costly given the effort and the benefit of the hack appeared once the Solar Winds binary was distributed. You can reduce the benefit of such a hack by not having information critical enterprises all running the same binary blob.

> You can reduce the benefit of such a hack by not having information critical enterprises all running the same binary blob.

That assumes that you actually inspect the source, right?

If Solarwinds was distributed as source to hundreds of companies, maybe many would not bother diffing the source from the previous version but it seems plausible that a few would look at these at the least, especially given you are talking corporations who follow deployment procedures.

The build process itself could spit out the diffs at the end, for example.

well, you still have a 'tragedy of the commons' situation if say 10,000 users are all counting on at least 1 user to inspect the source - but with no commercial incentive to do so, nobody does.

How does a user of open source ever know or measure exactly how well the source code has been scrutinized?

I'm getting this "open source doesn't guaranteed there's no problem" claim. And I'd agree open source doesn't guarantee there's no problem. But it seems clear that routes which have allowed malware to be pushed automatically are fundamental, guarantee problems and malware in source code is a potential problem with an obvious answer.

Besides all of these steps I think it's important to consider every convenience, every dependency, every package you add as another attack vector. This is even more relevant considering the product SolarWinds sold.

Part of me wonders if they'd have been better with GitHub cloud CI/CD in Actions, with immutable build infra (e.g. Ubuntu base images).

But, with the wild npm dependency community, where a small app can have 5K transitive dependencies, I feel we're going to be even more susceptible to these attacks going forward.

One of the big reasons why some shops refuse to use Javascript on server side because of its unknown attack surface that is NPM.

It's impossible to keep track of all of these libraries that can be written by anyone. Someone could hijack a github account and push malicious code to NPM.

This reminds me of the Ken Thompson hack: https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html

> So how do we guard against this type of attack? How do we know this hasn't already happened to some of us? What is the potential fallout from this hack, it seems quite significant.

Verified builds. That means deterministic builds (roughly, from a given git commit the same binaries should result no matter who compiles them. It requires compiler support and sometimes changes to the code) plus trusted build infrastructure.

To verify that you haven't been compromised do a verified build from two independent roots of trust and compare the resulting binaries. Add more roots of trust to reduce the probability that all of them are compromised.

Establishing a trusted root build environment is tricky because very little software has deterministic builds yet. Once they do it'll be much easier.

Here's my best shot at it:

Get a bunch of fresh openbsd machines. Don't network them together. Add some windows machines if you're planning to use VS.

Pick 3 or more C compilers. Grab the source, verify with pgp on a few machines using a few different clients. For each one, compile it as much as possible with the others. This won't be possible in whole due to some extensions only available in a particular compiler used in its source, but is the best we can do at this point. Build all your compilers with each of these stage-2 compilers. Repeat until you have N-choose-N stage-N compilers. At this point any deterministic builds by a particular compiler (gcc, llvm, VS) should exactly match despite the compilers themselves being compiled in different orders by different compilers. This partially addresses Ken Thompson's paper "reflections on trusting trust" by requiring any persistent compiler backdoors to be mutually compatible across many different compilers otherwise it'll be detected as mismatched output from some compiler build ancestries but not others. Now you have some trusted compiler binaries.

Git repository hashes can be the root of trust for remaining software. Using a few different github client implementations verify all the hashes match on the entire merkle tree. Build them with trusted compilers of your choice on multiple machines and verify the results match where possible.

At this point you should have compilers, kernels, and system libraries that are most likely true to the verified source code.

Make a couple build farms and keep them administratively separate. No common passwords, ssh keys, update servers, etc. Make sure builds on both farms match before trusting the binaries.

The good news is that most of this can be done by the open source community; if everyone starts sharing the hashes of their git trees before builds and the hashes of the resulting binaries we could start making a global consensus of what software can currently be built deterministically and out of those which are very likely to be true translations from source code.

EDIT: https://wiki.debian.org/ReproducibleBuilds is Debian's attempt at this.

To me it sounds like they hacked the editor/code signing tools to insert malicious code on save/commit by devs. Having iron-clad CI toolchains don't help you with that. Need to focus on how to defend the devs.

Based on the description, the source code wasn't modified, only the build tooling on the developer machines. As such a CI server wouldn't be impacted.

That's the point of a trusted build farm. Devs commit changes to git, and either request a build or the build farm polls for commits and builds the latest commit on trusted hardware+toolchain.

A malicious attack could change the code but it would be detectable because git would preserve the malicious parts in the repo, and further tie a specific malicious binary to a particular commit making it easy to find the malicious code itself.

As long as not all developers are compromised then whoever is doing the code review would see the malicious code when they pull the branch to review it.

> further tie a specific malicious binary to a particular commit

Git uses SHA1 for hashes, right? Aren't there demonstrations that SHA1 hashing is cracked, so you could craft a replacement commit that hashed to the same value, in theory.

The developers of git are working on moving git to use SHA2 and have already mitigated some of the concerns around using SHA1: https://git-scm.com/docs/hash-function-transition/

SHA1 hash collisions are hard, especially when the data you can inject needs to look like code to a human and compile correctly. But the concern is valid so it's good that git is improving in this way.

> starts sharing the hashes of their git trees before builds

Yes, and it'd be equally fine to share the hashes after the build? (Just so I'm not misunderstand anything)

You're right, I phrased that a bit ambiguously.

You have to segment and have monitoring tools monitored by people with a clue.

But that is very expensive to do. The average SaaS or software company does nothing.

> So how do we guard against this type of attack?

Looks like they compromised the editor. If so, then I imagine checking checksums for each component of the toolchain would work. Though if they compromised the filesystem or runtime then that would complicate things. But still, a hash tree or certificate of the OS and toolchain as part of CI would seem to be a good idea in 2021.

> So how do we guard against this type of attack?

Don’t give people running windows machines access to your source code/production

Especially if they are domain joined.

Just don’t. Inb4 “but Microsoft has most sophisticated security” crowd, every major hack of the last 15 years always starts with compromised windows box that allows attacker an outpost to move laterally.

Then you should go review major security events.

Attackers have been able to compromise major environments regardless of OS type. Windows having a larger market share is clearly going to have more compromises, but other OS based environments are just as prone to compromise. There also have been major incidents with standalone applications that are OS agnostic.

Heartbleed is a great example of an OS agnostic flaw that was used for major attacks. As noted earlier while the OS does play an issue, its more than likely just correlation rather than causation due to Microsoft having a dominant position in the market.

Reference: https://en.wikipedia.org/wiki/Heartbleed#Exploitation

Source of major leak/intrusion as a result of heartbleed? That was 2013 btw and I remember it clearly - since it was closed pretty quickly I don’t recall anything major coming out of it

Edit: i mean when it was 0day. Obviously if some idiot didn’t patch their servers it shouldn’t count

It's very possible there were several and nobody noticed them, or nobody attributed them correctly.

There are also the Spectre and Meltdown attacks which exploited Intel cpu pipeline bugs regardless of the OS running on them.

Spectre/meltdown imo is basically same category of security vulnerabilities - big monopoly cutting corners on security for commercial gain

It was not done by a bunch of amateurs that is for sure. Now everything points to Russia, but that is also the most obvious clues to leave as who would question it. However...we know the NSA wants to be in every system and this kind of operational security and evasion screams of the NSA to me.

> Now everything points to Russia

How do we know this is Russians? To my knowledge its very common practice to obfusticate origins before launching a campaign like this by washing through several different countries.

You could leave stuff like comments or references that would suggest it was the Russians, there's just no way of knowing, so I follow the fundamentals of political sabotage: whoever benefits most is the culprit. Who has the most to lose and gain here?

Yeah, no.

The various Russian APTs have tooling they prefer to use and are attributable to them. This is generally fairly stable because these are professionals who spend years learning specific toolchains, programming, and skills, and do not really change it up much, since they don't have to. Even if they're attributed, what is the world going to do? Toss a bomb into Russia?

And before you get started, yes, security professionals are aware that you can obfuscate that, but there are already techniques to defeat this second layer of obfuscation.

If multiple sources are saying this was probably Russia, they probably have a decent bit of proof.

Hmm I hadn't considered that but how do you find out what tools were being used to produce the payload source code? How can you be certain? Could another adversary simply use the same tooling or perhaps it is shared to an allied nation (enemy of my enemy is a friend) to do its bidding.

These people are incredibly smart. https://link.springer.com/article/10.1186/s42400-020-00048-4 https://www.blackhat.com/html/webcast/07072020-how-attackers...

TLDR.

> They highlight that not only malware samples and their specific properties (such as compiler settings, language settings, certain re-occurring patterns and the like) are useful, but also information available outside the actually attacked infrastructure, including data on the command & control infrastructure.

Yes you can obfuscate certain things, buts its hard to obfuscate EVERYTHING, and if you dig deep enough, you can make a decent effort finding the owner.

From reading the paper, it seems like it would be difficult for a private hacking group to manage but completely doable for someone like the NSA. They could outfit an entire team to work somewhere else for an extended period of time, making behavior profiles unreliable.

Why would the NSA bother with hacking American companies? The American security establishment is only one warrant or national security letter away from getting all the information they need from any of these companies.

The NSA has plenty of reasons it might want to infiltrate the DHS or other agencies.

Except the CIA and other actors have been known to impersonate the methods of other nation states, so attribution is never the smoking gun you're claiming it to be.

That was certainly my point...every nation state does their best to obfuscate their code and point somewhere else.

This is good in theory, until you yourself get owned by someone else, that now all of sudden knows exactly what you think is a tell-tale sign of some other actor.

From there on they can modify their payloads to look like they come from another toolchain.

Could the NSA just tell the rest of the government to pretend it was Russia?

It's not like any of us would know better.

Reply on news.ycombinator.com