The lesson isn’t that any particular victim sucks at security, it’s that well-resourced targeted attacks are generally unstoppable.
I recall that many companies have switched from a perimeter model of defense where systems are secured from the outside to a "defense in depth" model where each system is secured on it's own (plus the perimeter).
Perhaps folks should think about tightening the in-depth model and avoiding the consumer model of constant updates from a zillion providers. Or perhaps a single lab could verify updates of the zillion providers rather than leaving them on their own.
Yes, this is the current bleeding edge of cloud security, known as “zero trust.” Amongst other things, it usually involves provisioning mTLS identities in a secured manner to each service, with connections restricted to a whitelisted set of identities.
I found Evan Gilman and Doug Barth’s “Zero Trust Networks: Building Secure Systems in Untrusted Networks” [1] a pretty helpful read in understanding what modern/next-gen cloud security looks like.
Some modern implementations of varying depth and scale include SPIRE [2], Tailscale [3], and BeyondCorp [4].
-—-
[1]: https://www.amazon.com/Zero-Trust-Networks-Building-Untruste...
[2]: https://spiffe.io/
For example, quoting verbatim from the link above:
> [...] Modern OS's, routers, basic apps, etc aren't as secure as software designed in 1960's-1980's. People are defining secure as mitigates some specific things hackers are doing (they'll do something else) instead of properties the systems must maintain in all executions on all inputs. We have tools and development methods to do this but they're just not applied in general. Some still do, like INTEGRITY-178B and Muen Separation Kernel. [...]
There have always been lots of techniques that people claimed as magic bullets for security. They were there in the 1960s-80s, and there still are.
And yet Modern OS's, routers, basic apps are much more secure than the equivalents then.
I mean in the 1970s, there was the Ken Thompson backdoor and no one knew about it until he disclosed it in his ACM Awards acceptance speech[1].
Nowdays there are provably secure kernels[2] in real world deployments, and high assurance kernels[3] in billions more.
[1] https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html
[2] https://ts.data61.csiro.au/projects/seL4/
[3] https://en.wikipedia.org/wiki/L4_microkernel_family#Commerci...
Edit: It seems like a serious extension of the zero trust concept would involve something like "only allow in source code into the system from sources we trust and then we compile it ourselves". Limit trust to trusted identities and don't allow binaries in any more than people.
No, "zero trust" means you install it (or anything) but assume that is is malicious so take proactive steps to limit its access within your network.
"Zero Trust" means assuming everything (including your compiler) might be compromised, so try to limit the damage any one incident can cause.
But you have to keep in mind - the whole point of zero-trust systems is that even if a malicious third-party blob is present on some nodes, it's blast radius is limited by the whitelist of clients that it can communicate with, which makes exfiltrating information particularly difficult unless you have additional exploits that let you move laterally.
And sure, there exist things like Linux drivers that are binary blobs in source code form but at least directs one's eyes to such things.
Edit: Also, even in these environments, some people and some software has to run the network itself, like a sys admin tool. Which makes sys admin tools something everyone should look at. And brings us to the present hack.
The problem is that network security is a tiny, tiny subset of security in general. You can throw SAML, mTLS, or IPsec at your connections all you like, it would have done exactly nothing to stop the type of attack that hit SolarWinds and their customers.
Let me reiterate this: ZTN would have achieved zero protection.
The problem in general is that there are conflicting interests between what the developers want to do to reduce their work effort and operational application security. Even large vendors are hopelessly bad at producing secure and securable software. The documented and recommended easy path is not the secure one. The products are incredibly hard to secure from the perspective of some poor ops person.
Some random examples:
- Most vendors document their firewall ports, which is great! But not their direction or what roles they apply to.
- Very few vendors provide machine-readable firewall definitions that can be imported into firewall systems automatically.
- Vendors like AWS and Azure that do publish JSON lists of their endpoint addresses and ports use unique, non-standard formats, so very little effort is saved for the end-users.
- Public cloud vendors especially end up "blending" multiple customers into pools of shared IP addresses for all of their PaaS or SaaS services. This is a disaster for security, because it makes it virtually impossible to put a WAF or a WAP in front of certain services.
- mTLS support is a shitshow. Manually uploaded certificates that expire and break everything annually is the norm. CRLs and OCSP are almost never checked. Many products support only a single certificate, and will even crash if talking to products that do support multiple certificates and are in the middle of a rollover. Alerts and visibility of these certificates is poor. Secure certificate storage (i.e.: in a TPM) works better on laptops for securing WiFi than it does for servers processing billions of dollars in transactions.
- The rise of Docker, NuGet, Cargo, and NPM are a security disaster. Incident after incident and warning after warning have been ignored. Why? Because these technologies are soooo very convenient for developers, and they're the ones making the decisions to use these package management platforms. The ops people are then left with the mess. Have you ever tried patching a vulnerable version of .NET Core in a Docker container from a vendor that doesn't even exist any more? Good luck with that.
- Similarly, the plug & play nature of the public cloud with the extensions, plugins, agents, and marketplaces isn't a disaster waiting to happen. It did happen. That's what this SolarWinds thing was all about.
- Pervasive telemetry is largely indistinguishable from data ex-filtration. Not only does it pollute the logs, but vendors are deliberately obfuscating the traffic to work around customers blocking it. Did you know that Microsoft used "eu.vortex-win.data.microsft.com"? Did you spot the deliberate typo? It took me a while to realise why telemetry traffic was still getting through despite firewall blocks on "*.microsoft.com"!
- Back doors in products for "support" are another security disaster waiting to happen. Several SAN array vendors for example include these as standard now.
I could go on and on.
I'm watching some of our larger customers struggle with security, and to be honest it feels hopeless. These orgs have 10K+ endpoints, 2K+ servers, running tens of thousands of distinct pieces of software from at least a few hundred, maybe a few thousand orgs.
For every step forward in security, there's two steps back for someone else's convenience.
The most secure work is the work that doesn’t get done.
The whole thing is a cluster fuck. I keep getting hired as a person to help retain talent, but enterprises like to put all their cards on Solarwinds. When they can blame Solarwinds, their managers are still safe. There is no compartmentalization of access. You don't need to have all departments globally be in the same LAN to begin with.
We can hire all the "security experts" in the world, but if software and architecture are fatally flawed from the getgo, you can check all the checkboxes, and scan and produce all the pretty reports you want.
The end result is still paper mache
I'm thinking of properties like "for any api input the reply is independent of $SECRET".
Formal verification is an extreme cost amplifier, and the cost is super-linear with the size of the code base. I don't think my estimate is unrealistic, depending on exactly what you would expect to have running before vi (I was assuming a complete OS including a kernel, device drivers, a C stdlib or equivalent, and a userspace similar to the GNU tools).
This ignores the secondary problem which is that if the attacks being deployed are 100x stronger than the defenses, how hard is it to develop an attack that is merely 2x stronger than the defenses. If we lazily extrapolate this linearly, that would be 1/50th the resources to develop an attack that still outmatches the defenders. How many people do you think were on the SolarWinds attack? 10, 100, 1000? Even at 1000 that means you would only need 20 fulltime people for a year to develop a credible attack against nearly every Fortune 500 company in the world and most of the US government. That should be a terrifying concept. Obviously, this is lazy extrapolation, but it is not so off as to be non-illustrative of this problem.
Given this immense difference between real problems and available solutions, the only reasonable assumption for companies and people is to assume that they are completely defenseless against such actors and likely even largely defenseless against even poorly-resourced attacks as demonstrated time and time again. It is imperative that they act accordingly and only make systems accessible where the harm of guaranteed breach is less than the benefit of making those systems accessible.
But yeah, security is in a terrible spot, throughout the software industry.
It is not clear to me what the steps would be to improve it. It is not clear to me whether these steps would increase software costs (something that is quantitative) by "1,000% - 10,000%", and if so how society as a whole could possibly afford it, in this society where we have put (insecure) software everywhere.
We then need to determine what we deem adequate for intrusion detection. I contend that a competent intrusion extracts the vast majority of its value within 1 week. This is consistent with the results in the Verizon Data Breach Report from 2019 [1]. Given this, we would logically want to detect an intrusion in less than this time otherwise the vast majority of the damage has already been done. By this metric, 1 week or 7 days versus 240 days would require a 35x or 3,500% improvement in detection by FireEye to be able to detect the intrusion and materially impact the value gained from the intrusion.
In addition, this is not materially different than the standard in the industry as seen in the other graphs on that same page which shows that detection normally takes on the order of months which is far longer than the standard for exfiltration which is days to weeks.
I do not contend that all software everywhere must be made secure against all attackers. However, it must be evaluated against actual expected attacks with a proper understanding as to what it can actually defend against otherwise it is impossible to make an accurate cost-benefit analysis. It is up to the creator/user/people to make an evaluation if the benefits outweigh the risks. That is how you define adequacy.
[1] https://www.nist.gov/system/files/documents/2019/10/16/1-2-d... Page 10
Like sure, securing yourself against a government is hard. But when a security researcher told solarwinds an update server was accessible with the password 'solarwinds123' and they sat on that report for 3 days [1], my confidence levels that they were generally competent are not high.
> Security researcher Vinoth Kumar told Reuters that, last year, he alerted the company that anyone could access SolarWinds’ update server by using the password “solarwinds123” [1]
and
> When reached for comment by Newsweek, Kumar forwarded his email correspondence with SolarWinds. He first notified the company of the issue on November 19, 2019. SolarWinds' information security team responded a few days later on November 22, 2019. [2]
and
> Others - including Kyle Hanslovan, the cofounder of Maryland-based cybersecurity company Huntress - noticed that, days after SolarWinds realized their software had been compromised, the malicious updates were still available for download. [1]
Did the company with the above security practices get more important stuff right? Like sure... maybe? But probably not.
[1] https://www.reuters.com/article/global-cyber-solarwinds/hack...
[2] https://www.newsweek.com/solarwinds-update-server-could-acce...
Air-gapped networks are a pain to work with but are incredibly effective. I still haven't read anything on how the attackers actually reached the build server though, so it's hard to say if it would have worked in this specific case. I wonder if the lack of a clear statement of how they made the initial breach is due to
- bad searching on my part
- they still don't know
- they're still trying to mitigate it
- they're still trying to word it in a PR-approved way
One example: For many years before they were bought by HP, Compaq's mail system had a regular Internet-connected SMTP MTA that fed two back-to-back hardwired uucp MTAs that connected to another SMTP MTA that fed their internal network. This was the early days of the public Internet, and they didn't want to have to trust that their SMTP MTA was secure, so they just made sure nothing but mail could ever get through their mail pipeline.
It's awkward and a bit of a pain to set up and manage, but enterprise companies have the resources to do things like that. And it did actually provide state-of-the-art email security for its time.
(FWIW, I didn't work with Compaq, but compared notes with their network architect on an occasional basis...)