SolarWinds hack was 'largest and most sophisticated attack' ever: MSFT president
reuters.com
reuters.com
This is a fundamental shift in viewpoint.
It seems pretty well established that making secure software is impossible. Time to pivot to designing software systems that are tolerant of inevitable security breaches.
One example of this is compartmentalization. A single breach must not have access to all the sensitive data. Another is backups must be air gapped (or put on physically read-only media) so ransomware cannot compromise them.
Compartmentalization is used in battleship design, aircraft design, spy networks, etc. Time to use it in software systems design.
P.S. Just to be clear, one still strives to design airplane parts so they won't fail, it's just that one does not rely on them never failing.
P.P.S. It's really really hard to sink a battleship as they are so compartmentalized. See the sinking of the Bismarck and the Yamato.
1. What happens when the root password is guessed by a malicious person?
2. What happens when a trusted employee is really an enemy agent?
3. What happens when we download and install a malicious update from a trusted vendor?
4. What happens when the server room burns down?
5. What happens when a malicious USB stick is plugged into our secure network?
6. What happens when our CEO's laptop gets stolen?
And so on. I deliberately wrote "when", not "if".
Without providing the context of how expensive or cheap it will be to adhere to each of these best practices, it will be hard to convince those with decision-making authority to do the right thing, unless they are in a highly regulated environment to begin with.
An aircraft on the other hand is already very expensive to make, precisely because it involves transporting resources that are impossible to replace: human life.
I've argued this to well-placed people in two Fortune 250 companies. At the first place, the conversation led to a cleanup up the IT policies, so that they were more consistent and reasonable, but they still weren't grounded in reality. The reaction at the second place has caused me to realize that IT security -- like everything else at a company -- is just another political game of thrones. Aggressive people are trying to work their way up the ladder, and increase the size of their internal kingdoms. They use scary language to leverage fear in upper management to drive ever-larger budgets and ever-more inconvenience in the name of "security" which winds up being little more than security theater. If there's a Fortune-sized company out there which actually makes IT security decisions based on practical consideration of risk vs. cost, I'd love to know. My current belief is that, by the time you get this big, there's too much disconnection between the IT department(s) and everyone else to align the incentives.
Which trend towards infinity, because... well, they want the thing they're advocating for to happen.
This tends to attract toxic, political people who thrive in this sort of environment, and push out more reasonable, technical people who don't want to deal with that.
At some point, you get a critical mass of toxic IT security managers covering each other... and then everyone starts ignoring and/or hating security.
The real loss is that most IT security orgs I've seen are extremely deficient in developers. The kind of person you actually need to automate things, so that they're quicker, so that people use them, so that the organization is more secure. :(
The teams I really strongly respect, for example at Square/Microsoft, manage to align user interest and company interest very well - that's a very hard thing to do and requires top to bottom investment. Microsoft's approach has been to monetize security, which makes that easier.
Both organizations also have resisted the mistake that you and I both agree most orgs make - they've invested heavily into hiring engineers to solve security problems, and you generally have to pass a coding interview even for security roles.
edit: Reflecting more, I suspect that the described behavior of hyper-political teams that have a self-preserving structure is probably something prevalent, but that I haven't had the misfortune of being involved in. I would imagine companies like that are generally toxic across orgs, and not just in IT, but probably any area where investment is forced and it isn't revenue-generating.
> I would imagine companies like that are generally toxic across orgs, and not just in IT, but probably any area where investment is forced and it isn't revenue-generating
I'd agree this is probably a key contributing factor.
It's been my fortune / misfortune to work for a number of companies that qualify (healthcare, financial, insurance).
But it has also been true (in my much more limited experience) with healthier, IT-as-profit-center companies.
I'd hazard better framing (than profit/cost center) might be "incentivized to improve" vs "penalized for mistakes."
Unless it's the former, it's in no one's personal interest to go above and beyond or suggest change. So you get glacial, incremental processes, and lose people who are impatient with working that way.
As you note, I think monetizing security is a key step to establishing a healthier balance.
The defense in depths / layered defense principle is probably well known in most companies but a strong defense is usually more expensive (in every single metric) than a strong offense, and it only has to fail once. A budget to cover this is easier to justify in a military setup for the reason you mentioned: lives. A trained soldier is expensive to replace, and so it a warship. This justifies a huge budget. In the corporate world the balance is "can we expect to lose more money (image, etc.) from the projected number of hacks than we save by not implementing specific measures?".
And lastly, companies like to say how they employed "software engineers" but the reality is that they employed mostly "coders". Coders are no more software engineers than bricklayers are civil engineers. A lot of the infrastructure and software architecture is left to people who don't have the broader picture because it's assumed that the particular system is inconsequential. Turns out almost no link in a chain is inconsequential when you're faced with a very determined, very well funded attacker.
The New York Times had an article recently discussing how measures to prevent Covid have to take human nature into account in order to be effective. In other words, people only have so much capacity for adhering to strict measures, so it's important to stress the really important factors (avoid prolonged indoor exposure with large groups of people) and to allow for some relatively low risk behaviors (outdoor exercise with appropriate social distancing). Also cited HIV and teen pregnancy as similar public health problems (saying "don't have sex ever" vs "limit sex to these less risky behaviors").
So, coming back to security, I think any effective security policy will have to allow employees to get their work done efficiently without undue strain. Avoid "security theatre" while still mitigating against the most severe threats through training, compartmentalization, etc. Otherwise, you risk people going around the security measures so they can actually get their work done.
The military are not immune to this either. Case study, the 2006 Nimrod crash in Afghanistan [0], which killed all 14 of its crew. This plane was the flying equivalent of an unsecured server whose root password was 'password'.
[0] https://en.wikipedia.org/wiki/2006_Royal_Air_Force_Nimrod_cr...
And coming back to the particular Nimrod crash, the findings of the inquiry suggest a chain of issues slightly more complex than your analogy suggests. Setting a root password of "password" goes way beyond poor maintenance, it's designing the plane with a crack in the wing.
I agree that my analogy is an oversimplification. But I think my point stands - the plane had a massive set of issues that were tolerated because up to that point the operators had got away with it by sheer luck - there hadn't been an operational disaster, just a mass of technical and maintenance issues over the years.
In this case, the flight crew as military personnel were acting at a high standard but were fatally let down by the wider organisation that allowed them to fly. The military owned safety case was fundamentally flawed and evidence that suggested otherwise had been actively suppressed, as revealed by the Haddon Cave enquiry which found fault with a range of military and contractor organisations and individuals.
The point, as I said below, is that the framework under which the military operate does a far better job at forcing the people to more consistently adhere to stricter rules. Which means under similar circumstances there's a good chance the military can keep a system safer than most companies.
If a CEO or other high level management doesn't have a sense of how to answer those questions, they have no business running a company in the 21st century.
There are plenty of stories of ransomware literally holding a corporation's entire business hostage. Software security is an existential threat for pretty much every business these days.
1. make the root password unguessable and change it often
2. background check employees
3. audit trusted vendor's security procedures
4. install sprinklers (!)
5. jam all USB ports with glue
6. train CEO on laptop security protocol
Yes, water based sprinklers are not the best choice for inside the server-room proper. But that doesn't mean you can't have other automated fire suppression systems. In the days of yore Halon systems were mostly used for this particular application. Halon has some issues though and has mostly been banned / replaced with cleaner alternatives. These days you see things like FM-200 used for fire suppression in these kinds of applications.
All of that said, speaking as a former firefighter, I'd take a water based sprinkler over nothing, even in that setting. Why? Because at the end of the day, human life is more valuable than a data-center full of computers and data, and if stopping a fire early saves even one life, but ruins a room full of computers, that's an acceptable tradeoff. And of course sprinklers are absolutely invaluable outside of the server-room. I don't have all the numbers memorized now, but to summarize in a succinct form, "sprinklers are wildly effective at preventing incipient stage fires from growing large, and at saving lives and property."
And as @WalterBright continues to say below - you are storing backups off-site, right? And you have a DR data-center on another continent that can be spun up in a matter of minutes by toggling a load-balancer or DNS setting somewhere, right?
Even counting 9-11, as the plane crash cut the water pipes.
But still, relying on sprinklers to save your IT business is a bad idea. Offsite backups are needed.
Yep, totally agreed. Resiliency comes from having offsite backups, cold/warm standby sites, testing your backups and ability to switch to the DR site, etc. The sprinklers are mainly about saving human lives.
Sure you install sprinklers in a data center to sprinkle water on servers and other electric devices in a context where there might or might not be broken or other wise un-isolated wires due to the fire...
What you can use instead is to flood the room with CO2 to suffocate the fire.
The problem with that is that it's also deadly to humans.
Still e.g. for rooms with long term data storage it's not uncommon to have measurements like that.
Besides that there are docent of other reasons why a data center might go down (temporary).
So the better answer is make sure you system is distributed over multiple data center, i.e. eliminated any single point of failure.
And even a closed door in a residential building can slow things down enough to make the difference between life and death. This is why the fire service encourages people to sleep with their bedroom doors closed and to have a smoke detector near said bedroom door.
The downside to a heavily partitioned building, especially a very large building, is that it may become a maze that is hard to exit, and lots of doors between different areas slightly increase the risk that somebody will have a brain-fart and lock/chain doors that should never be locked, or otherwise do something that effectively traps people inside. Note to everybody: never chain/lock/barricade a door that must be used to exit a building in an emergency, in such a way that it can't be opened from the inside.
1. don't store everything on the machine(s) accessible via that root password
2. don't allow any employee unfettered access to everything
3. don't allow one piece of software to have access to everything
4. do not store backups in the server room, or even in the same building
5. buy computers that do not have USB support in any form
6. full disk encryption on laptop. Minimize sensitive data on laptop. Use smallest capacity disk drives on laptop. Don't allow any laptops to have access to secure network. Issue hardwired desktops to employees. No wifi to secure network.
Neither detection or prevention can be perfect like nothing human-designed, but rather detection should be used to hedge yourself in situations where prevention fails.
None of the mitigations that were described were aimed at preventing a breach or disaster. Instead all were designed to mitigate the damage that happens when a breach occurs.
These are not new concepts in the security industry. In fact very much of the opposite; manifestations of them like zero trust have been one of the main buzzwords for the last ten years or so in the cyber industry.
It's a different thing sketching something on paper and how it actually works when trying to apply it into the chaotic mass corporate IT usually is. I work in this industry and talk to a lot of corporations from all over the globe about their security postures. I don't see a huge amount of companies who have the resource to pull off a well segmented environment. Truth is it is expensive, quite complex to pull off, potentially disrupts business and it's still just an internal security cost that when you present it to your boss they will ask "why are we spending all that money on firewalls then?"
Anyhow the point was that if you build your security posture on the assumption that you can successfully lock everything down and you don't need to do any monitoring internally, you're setting yourself up for that massive disruption when the stars align for the attacker and they get a free roam environment in your internals. And it happens very often, as we can see.
As reported on "60 Minutes" yesterday.
The security company did not compartmentalize their own system. They relied on monitoring, which failed.
Furthermore, it was said on the segment that firmware on the hardware can be infected, so reinstalling the system software won't get rid of it. The obvious solution to that is to put the firmware in ROMs, so it cannot be electrically reprogrammed. But nobody does that.
Inconvenient? Yes. But sometimes appropriate.
1. Which resources are held by the company?
2. Which actors are interested in these resources (both benign and malicious)?
3. What could threaten the confidentiality, integrity and availability of these resources?
4. How likely are these threats?
5. What would the impact be if these threats would occur?
6. Given the likelihood and impact of each of these threats, form a method to handle these threats.
7. Execute on the plan.
Furthermore, the results of each of these steps should be documented and periodically reviewed.
That is what a security professional should ask.
An even more professional security professional should model the network of threats. For example, one disgruntled employee might have a relatively small negative impact on many different resources, but causing a large impact as a whole.
1) detection
2) detection
3) detection
4) sprinklers
5) detection
6) detection
unfortunately, most orgs outsource their internal detection or have no capability at all.
> Sprinklers
That is "we can prevent the server room from being destroyed" thinking, rather than "how do we survive the server room being destroyed" I'm proposing.
And you are wrong in labeling it "prevention". It is a feedback mechanism. Fast feedback is essential in almost ANY scenario. Most of the things can be fixed if detected early on.
The first thing is knowing. If you don't know something, you can't act on it. To take your battleship analogy, if I remove all torpedo sensors from the ship, how long will it last given it is compartmentized and whatnot.
That's still focusing on making components that will not fail, rather than a system that can tolerate failure.
But we have bunch of non-heart redundancy.
There is no intelligence behind our design. Your body takes its current form because no mutation which would change it has yet led to a statistically greater chance of you passing on your genes. That nature designed you to have pain receptors is no more an argument for their utility than it is for the peacock's feathers.
Sprinklers comment was made in jest.
The defeat was to chop the phone line where it entered the house, because the telephone company puts their box on the exterior of the house. (The book was printed before cell phones.) The sophisticated $$$$ detection system, defeated by the simple swing of an axe.
Relying on detection requires a perfect detection system, which is impossible.
Let's talk dead node detection, for example, and consensus. "Perfect detection" is impossible due to possible network partition, yet we can build reliable distributed systems on top of that imperfect reality, even in face of malicious nodes (B-F/T). We do this via redundancies; mutli-step 'actualization' of system changes (e.g. transactions); and the ability to revert to a known stable state.
Based on that analog, I think it reasonable to strive for security regime approaches that permit for security failures at component levels. The computational and bandwidth loads may be excessive (ATM) but ultimately there will be systems observing systems and a type of 'security consensus or quorum' theoretical work informing how to build such systems.
There is a thin layer of semantics that overlays the generalized and irreducible state transition due to input: one instance is preventing inconsistent action on data; the other is preventing insecure action on systems.
Like you said before, any solution is a mix of 2 paradigms.
How many people have a battery hooked up to the siren?
First of all, detection is methodologically bankrupt. We have almost no one out there saying how detection should be done with consensus - only in the last 5 years have we even started to improve here.
In my opinion, detection of attackers, which is what the industry focuses on today, is a huge waste of time and resources - it's the last step in the process that I would recommend.
I would personally say that detection should be staged as:
1. System inventory (can you attribute an IP or Hostname to a device identitiy, a user, etc)
2. Policy enforcement (can you detect when policies change, or are violated?)
3. Unexpected behaviors - go to the people building systems - ask them what's expected, what isn't, and build rules for that, or even better, have them build and maintain the rules under your guidance.
4. Attacker behaviors - finally, spend some time building rules for attacker behaviors.
Most organizations skip straight to 4, and then you have a team of defenders who have no idea how the network they're supposed to detect is supposed to actually work. This is throwing away the greatest advantage defenders have - that they know where the attack will take place, and they know all of the stakeholders for those environments.
Here's the chief of the NSA Tailored Access Operations saying this at USENIX four years ago.
https://youtu.be/bDJb8WOJYdA?t=83
"If you really want to protect your network you really have to know your network"
None of this is as simple as "detection" - it means working with the policy teams, with your infrastructure teams, with your product teams, to better understand your environment.
1. Computer stations use two types of fingerprinting at all times, facial recognition and typing biometrics. Also, login to the system requires password or pin entered after a card is inserted, followed by a fingerprint authentication, followed by the password of the day. Critical software/data must be accessed at an air-gapped machine inside a Faraday cage.
2. Employees only access based on what they need for their job that week, access controls are fine grained, employee access is logged, and that log goes via data diode to an otherwise air-gapped computer inside a Faraday cage.
3. Users can't download and install. Only trusted professionals can, and then only after the software is vigorously tested and approved.
4. Hot site goes fully active, personnel are immediately transitioned there, and a root cause analysis is performed to figure out just who screwed the pooch to allow the server room to burn. Repairs are made as quickly as possible by vetted personnel, then checked for security by different employees, and then checked by a third team.
5. Physically disassemble the computer to the point where you can unsolder the USB ports (some come with support on board for USB ports but none in the case, enterprising bad actors could open the case and install their own USB port). Also, have anti-tamper cases with anti-tampering turned on after that. Have OS protections preventing media not whitelisted.
6. Step 1: Phone home and wipe procedure on drive activates if the computer boots up and authorized use does not log in with X minutes. Disk has full disk encryption and is reencrypted with new password each month. Step 2:Fire the CEO unless they were mugged, or a K&R family situation.
The above steps are extremely expensive though, and only very large organizations will be able to afford them. For startups, I have no idea..some of them are implementable, but most aren't. Also, you have to accept that 5m-10m of every hour is taken up by security measures.
None of the above prevents a computer that a bad actor has physical access to being compromised, but it makes it hard enough that generally it's only going to be state-level actors that will take the trouble, and you'll likely know it's happened so you can take steps.
People don't learn from mistakes if that's the response. They'll also hide their mistakes, making things worse. You want a CEO to report the loss ASAP, not cover it up. Having a "no fault" culture encourages people to quickly report mistakes.
>1. Computer stations use two types of fingerprinting at all times, facial recognition and typing biometrics. Also, login to the system requires password or pin entered after a card is inserted, followed by a fingerprint authentication, followed by the password of the day. Critical software/data must be accessed at an air-gapped machine inside a Faraday cage.
Is the air-gapped machine machine seeded with all future passwords of the day, as well as trained for all potential operators' biometrics (facial, typing, fingerprint) before being put into service?
Nice way to filter $$$$$$$ through consultancies though.
> It seems pretty well established that making secure software is impossible. Time to pivot to designing software systems that are tolerant of inevitable security breaches.
This is the nature of "zero-touch networking". More generally, we talk about "security boundaries" as the equivalent of what you'd consider "compartmentalization". For example, the virtual machine is a security boundary (infect the OS but the VM should keep it contained to not affect the host hardware), the network firewall is a security boundary (treat everything outside as potentially hostile unless authenticated), the authentication system is one, and so on, because each one imposes substantial security barriers that you depend on to some degree, but you also model the "what if it fails?" scenario.
> One example of this is compartmentalization. A single breach must not have access to all the sensitive data. Another is backups must be air gapped (or put on physically read-only media) so ransomware cannot compromise them.
All of these things are already commonly known practice, and both data compartmentalization and airgaps have been in practice for decades.
I'm not saying that it's perfect everywhere, because it's a shitshow in general, but rather that the field has been aware of and has been implementing these practices for a long time when people have the incentives to do it.
I think there are two major differences. The first is that in cyber security the failure mode is adversarial, not coincidental, so tolerances work differently. It's less about the stress the system can take, more about how long it'll take the adversary to figure it out and how useful the exploit is. The second difference is that there isn't remotely as much market/government infrastructure around making sure that you don't screw up the security. That's both good and bad: good because it means lower costs to entry. Bad because it means plenty of people will play fast and loose.
Instead of building a giant moat and assuming that everyone who got past the moat is trusted, assume that everyone is untrusted by default, and build a capability system that's expressive enough that you can give everyone just enough capabilities that they can do their job without going through a bunch of pointless checks.
In practice that model is impopular because corporations tend to screw up the later part.
WE are the weakest link! The most technically secure system would be nearly unusable by humans without enormous inconvenience. So long as software systems are built by humans, for humans, they will always be vulnerable at the interface.
The budget and architecture changes are not popular, though.
I partially blame this the security community as well (of which I'm a part of). The common tendency to think exclusively in risk controls and absolutist statements on secure vs. insecure means the incremental improvements needed to hit eventual ZTA are difficult to event start. In short, security teams suck at intra-company sales sometimes.
That said, the rumor/article I'm pretty sure I read detailed how the SolarWinds CEO was an ex-CFO type, and shredded their "cost centers" the last few. "Cost centers" mean security teams as a rule. Their security team was tiny, just like every SaaS vendor skating by only through passing audits with important vendors and getting breached (which you can do w/o a security team).
Fwiw, anyone at places fighting cloud migrations, cloud migrations get you several easy long jumps into ZTA almost by default.
I've been at this a long time. My POV is that this is true, but doesn't matter. The leadership almost never actually cares about security. It's the correct choice for most orgs. https://hbr.org/2015/03/why-data-breaches-dont-hurt-stock-pr...
Even the most impacted companies only take a hit in the medium term (1 yr) at best. https://investorplace.com/2019/03/equifax-stock-investors-ar... Right after that article, the stock took off and is still doing well.
Leadership is sophisticated these days. They aren't ignoring security due to lack of salesmanship from their security team.
I think there are two levels of security sales, if you will. One is junior//mid level to junior//mid level, and one is CISO<>CIO, and they both handle different pushes that play their own, sort of non-overlapping role in company risk management.
The CISO<>CIO thing falls right into the bucket that you're mentioning... big architectural changes need much more going for them than good salesmanship from security. If there's not a major benefit for the enterprise in non-sec areas by doing changes like ZTA, or a lot of precedent at this point about a SIEM/SOC just being something one has, it's an uphill battle.
That said, a lot of these breaches, like the Equifax admin:admin, isn't so much a "for the stock price, this is a net positive we'll run the risk for" type of decision that security loses. A lot of this is just patiently needling/selling to dev and IT teams about finally rotating a pw to something complex, for instance. That happens successfully.... attackers try another IP door on the internet and perhaps Equifax doesn't happen (using the theory that Equifax wasn't from an APT, which is of course a different situation). Of course insane change management boards exist, but in many cases it's just that easy of a fix.
My theory is even more relevant when you look at SaaS vendors, like SolarWinds, as really driving govt's innovation edge and by extension being the vector into the environments. In those cases, the security team's political capital is even stronger, as smaller companies mean much more cross-org influence is possible. Sometimes, it's just flipping "min pw complexity" as a radio button in AWS's IAM console, and no one is the wiser. When so many attacks source to these very easy wins to fix, that's where the salesmanship is lacking (or can really help) for sec.
To your point about "12 months and everyone forgets," that's not the case for the SaaS vendors working in a crowded space providing generic products. While the pool of possible vendors is shrunk by like how many SaaS vendors bother to pass FEDRAMP, and/or you stack on contractual inertia, there is still a pool. SolarWinds doesn't really provide that exceptionally unique of a product, and they got the entire Fed Govt pwned. That'll be >12 month impact for them. If it's not, I owe you a beer. This is even more pronounced of a motivator across the legion of SaaS vendors who have their first handful of serious clients, but a sec incident w/o a seriously embedded platform blows them up.
> flipping "min pw complexity"
I couldn't disagree more about this, but that's a particular sore point of mine. Otherwise, yeah, cloud services in general give you abilities like this. I mean, so does on-prem -- AD DS has the same (actually, far far far better) switch, but cloud is where the action is.
> selling to dev and IT teams about finally rotating a pw to something complex [...] just that easy of a fix.
Yes, and that isn't selling failure so much as security incompetence from the top. If you have a solid core, it's easy to throw switches when you see a blemish on the surface. If the core is rotten, that surface defect is not going away, even if you do catch it and even if you do fix it! The security team is only as good as the leadership.
I firmly agree with what you're saying, violently agree perhaps! But I think the scope of a smaller SaaS company means the sec team has an amount of technical and people agency that's sort of unheard of at the bulk of companies.
That's true, that's perhaps "exceptions," but a not unimportant amount of SaaS vendors are at that headcount/company profile .
Their password was "SolarWinds123".
Everyone who gets pwnd tells a story about sophisticated state actors to make it sound like some unstoppable force has hit their impenetrable defences.
2/ For SolarWinds itself, the password is one tiny step along the way. I can guarantee you that having that password won't allow you in any way, to deploy persistent malware on developers's machines (first you'd have to work around Windefender) and getting knowledge of the company's architecture, its internal Repos, etc. You'd have to bypass MFA too at some point, which the attackers did.
3/ If having malware than monitors developers VisualStudio console in realtime to inject a few lines into it so that it gets secretly compiled into their day-to-day work, without breaking the project, while also providing C2 capabilities; if that, is not "sophisticated", i'd be curious to see an example of "sophisticated" malware.
SolarWinds recommended to proactively add exceptions to your AntiVirus such as Windows Defender for their stuff[0]. They probably followed their own advice for their own systems as well. Just saying...
I don't think I'll believe such guarantees about a company whose update dissemination server password for a write-privileged account is CompanyName123.
I mean, you could be correct, but still.
There's also effort at making the airplane survive malicious cargo.
The very next step of this that so many people starting out forget is:
When the part has failed, and the redundant systems taken over, how will the original be repaired? If a repair does not happen soonish, then other parts will fail, and the system as a whole will fail.
Too many systems are designed to be redundant, but with no self-test, monitoring and alerting system, redundancy is useless.
In the case of the compartmentalization proposed, this means there is no point in compartmentalizing your data if nobody will do anything when a compartment is compromised.
What I think might be a good idea for big organizations with sensitive data is to download a portion of the web and run their own intranet and isolate the network. Maybe Google can license their search engine and deliver some of the indexed web on prem as a service. If there is nothing coming in and going out, that might be a good foundation for good defense.
I think this is the point. Software is about operating at the top level predominantly, packaging together stuff other people wrote using only simplified API's which abstract away their internal complexity. So you can zoom along at light speed producing "solutions" before the other guy and make the most money the fastest and move on.
Engineering is about starting from physical principles and building a product that fits the understanding of those fundamental principles at work. The engineer generally stays with a set of fundamental domain principles their entire career (e.g. sticks with bridges as opposed to transmission lines) while programmers tend to stay at the top layers while the technologies below the surface change radically.
It doesn't have to be that way for all software, presumably. Critical software and hardware can be designed together, for example, from the bottom up. I wonder if software patents is at fault for allowing those at the top layer to capture almost all the value in computers, and commoditize the technology below.
That's not how engineering works at all. The vast majority of engineering is working at a high level to take systems other people have long since perfected and slapping them together to solve new problems. If you need a solenoid valve, you don't design one, you buy one. And engineers do not stick to narrow domains - I've personally designed a seismic rated tv mount and a machine for testing helicopter flexbeams in the same day. How many bridges would need to be built for people to do nothing but bridges for 40 years?
No, "I did not care enough to read the documentation and understand what I was working with" is not a valid excuse for making something that fails catastrophically. Of course there are going to be little idiosyncrasies that you would never expect if you weren't down in the weeds, but that's true of every field, and the solution is to assume that such issues will inevitably exist and design with appropriate factors of safety and redundancy.
I didn't say build things from scratch, I said use fundamental principles. Your solenoid operates directly on your fluids or whatever and you presumably need to be able to understand how. Is it not possible to choose incorrectly without having the right expertise?
Neither the tv mounts nor the testing machine were remotely close to statics class. For the seismic TV mounts, I had to make an economical laser cut sheet metal assembly which could be assembled in a field and still survive earthquakes. For the flexbeam tester, I had to design a machine that could survive millions of cycles while placed in an oven at high temperature to simulate the effects of 3 years of flight time in less than 1 year. To say it was just statics would be like saying software is just assigning variables.
Conversely, if you are a database person, that is an incredibly narrow skillset. Creating a database isn't a profession, it's not even a job, it's just a task. Yes a database guy can create a database for practically any kind of organization, and the structures guy can analyze a truss for any kind of organization. But the engineer who can only analyze trusses is fired.
Yes, an engineer must understand fundamental principles, but that is true of any profession. How can you set up a database without knowing the fundamentals of how a database works? How could an accountant manage money without knowing the fundamentals of transactions? How could a ditch digger dig a ditch if they don't understand the fundamentals of a shovel? Any job that requires no understanding whatsoever was automated away long ago. That said, most professions, engineers included, can treat many of these things as black boxes during day to day life. I don't know how any given solenoid valve is implemented, and I don't generally care beyond it being rated for my use case. Of course this can lead to problems - I once designed a big machine for scanning cars and the power cables wrapped around it made it act like a giant antennna interfering with the data cables to the cameras - that took a while to debug.
The world is a fascinating place, you would do well to learn about it instead of resting on assumptions.
Are you really snapping at me rudely over a disagreement about the differences between engineering and programming? To answer your question, no. I'm saying that it's the same technical domain.
> Yes, an engineer must understand fundamental principles, but that is true of any profession. How can you set up a database without knowing the fundamentals of how a database works?
Excellent point. Because it's at the heart of my reasoning too. Identify those fundamentals and you can apply my argument. In my prior post I noted the fundamentals in engineering are specifically physical principles. In the case of databases they are not, instead they are completely at an abstract level.
And I am saying that these things are radically different, and thus the only way you could classify them as the same technical domain is if you are ignorant of those differences, or define domain so broadly that everything affected by physics is the same domain. You specifically refer to engineers remaining in a narrow domain and specifically gave an example of bridges and transmission lines as distinct domains, and it would be oxymoronic to talk about a narrow broad domain. You not only admitted that you have no idea what you're talking about, you used the fact that you were ignorant as evidence in support of your claim.
Going back to the example, both things employ statics. Statics is the most basic tool of engineering - it is the analysis of forces on something which isn't accelerating. There is no way to completely avoid statics for anything that physically exists. Statics is to engineering as wood is to carpenters. It is equally absurd to say that two engineering tasks are in the same domain because they both use statics as it is that two carpentry tasks are in the same domain because they both involve wood. The engineers designing bridges and transmission lines both employ statics, as do those designing kidney dialysis filters and rocket engine combustion chambers and acoustic panels and laser cutters, all six of which could very well be done by the same engineer (I've personally been employed to do five of the above so far in my career).
Your original claim, which I argued against, was:
> The engineer generally stays with a set of fundamental domain principles their entire career (e.g. sticks with bridges as opposed to transmission lines) while programmers tend to stay at the top layers while the technologies below the surface change radically.
Your argument had nothing to do with the difference between principles rooted in physics vs mathematics. You were saying that programmers use high level, general abstractions while engineers make more limited use of abstractions and remain highly specialized. I am saying that engineers work on a wide variety of problems over their careers, involving radically different physical principles, and that they abstract away many of the details of these various narrow domains so that they can work on the top level.
True. Where did mathematics come in? Math does not abstract away the fundamentals in the way that I am talking about. Whether you use messy coordinates or elegant tensors to describe a fundamental property, you are still working with the fundamental property.
Also perhaps the only way to get through to someone like you is to point out I have a doctorate in electrical engineering, and teach in a EECS department, working with both engineers and programmers daily and even helping to design the curricula (fyi, a person who specializes in databases is a real thing, we have no less than 4 grad-level classes on them, and there are no CS classes entirely on "variables" analogous to classes on statics). I have spent literally decades pondering the mindset differences between them. You should give up trying to argue me down because you won't get there by trying to twist my own words against me.
Oh and in EE we generally don't use statics. Pretty much ever. Though I did teach a dynamics class that included some one time. By the way that tongue-in-cheek reference to statics was an failed attempt at using self-deprecation to tiptoe around your oversized ego. Nice job weaponizing it.
I don't think there is a fundamental difference, just a difference in degree. Software enables far more layers of abstraction and far quicker development cycles than other engineering disciplines.
As for people building design tools, you mean engineers? When I say "top level predominantly" I mean things like this. The top level is the highest level for the software at hand. Practically everyone uses software.
As for research, when it comes to engineering they are generally doing either applied science (of one of the hard sciences) or applied math. Certainly if you go to research conferences in CS or the right niches of engineering, you meet lots of people who are basically just applied mathematicians.
I find that a lot of software people have a highly idealized notion of how the "traditional" engineering disciplines work. Generalizing strict safety procedures found in some niche industries like aerospace to other fields where they don't exist. The truth is that working under time or budget pressures and without fully understanding what you're doing is extremely prevalent outside of software too.
It certainly can be. But you've got to design it in from the beginning.
But the first step is pretty hard - convince programmers that writing perfect software is impossible and to stop relying on it being perfect. Fire anyone in charge of security who certifies "the system is secure". :-/
But you cannot compare them because the context is different: software is less regulated than aviation so while anyone can write of very bad software in 5min and sell it, this is not possible for a plane. And not everyone can pilot a plane, while there is no mandatory licencing and training to use a software.
In addition, no safety feature on a plane can prevent the pilot from crashing it willingly. Setting such an easy password and not adding additional protection (due to lack of training) is self sabotage. So far, anything related to SolarWind points towards a bad usage of the tools.
This is no longer true. There are procedures where no crewmember is allowed to be alone in the cockpit.
In the security world we refer to this concept as "Assume Breach" and to throw in a buzz word you might hear often these days "Zero Trust".
Joking aside, Assume Breach, Zero Trust and what I call "Homefield Advantage" are the main strategies to help secure the modern workplace in my opinion.
The design failure was "assume the seawall would not be breached". But it was, which destroyed the backup electric generators, and the rest followed from that.
A better design would assume the seawall is overtopped, and so would have put the generators on a platform. Simple, cheap, and effective.
This is also the principle behind Qubes OS, security-oriented OS based on security through isolation.
There are variations on this theme. An under studied and under used approach is extreme compartmentalization, where components are e.g. isolated in processes with only one simple input and one simple output allowed: for a codec for example, the only thing you can do if the codec code has a security bug and if you manage to control the stream and are able to inject a malicious one, is to change the output; which is not very useful given you could always change the output to whatever you want if you have arbitrary control on the input stream... Well at some point you may want to avoid CPU hogging too, and limit max memory allocation if that primitive is available, but having that kind of limits can be systematized in such an architecture.
So this variation makes the hypothesis that the codec can fail, but that the container is reliable. And there is no sound security architecture anymore if the container is not secure (or let's say not simple enough to be rated secure with high confidence). So you will always have to put some root(s) of trust in your system. Although you can get some benefits from mere mitigations, and sometimes they are more realistic esp. if you try to encapsulate legacy software, and we know a lot of SW activity is management of legacy software. But that's quite weak compared to a real security architecture, where it is reasonable to strongly trust the trusted components because they are simple enough, and where hard security properties are logically derived from those reasonable hypotheses.
Now you may want to imagine russian dolls of containers with decreasing restrictions (and then even more complex network of components), but if you dig it is likely to rely on the same techs as your extremely restricted component, and when not the case you only have very few tech-distinct layers (let's say process, then VM), so breaking out of extremely isolated ones would likely be the most problematic cases anyway. => you really can't do sound designs without some trusted components (or in a network: computers -- etc.) Of course you can add random mitigations everywhere on top of that, esp. if you are sure they won't decrease the security.
TLDR: mere mitigations are not a security architecture, and you won't necessarily be able to stack tons of them to be sure to resist in case just one fail. Some highly reliable components are still a must if you want confidence in the resulting system. Evidence: see the complex JS->browser->OS->VM escape chains in capture the flag or even real world attacks.
I've been working in this industry for nearly 45 years now. I still see endemic vulnerability to single points of failure, and little recognition of that.
Heck, the SolarWinds hack was first discovered by a security company because it had compromised their own internal systems and gone undetected for some time.
But a key aspect of this is that one does not develop software for airplanes the same way and with the same constraints/goals as other areas of software engineering.
If anything, aerospace engineering is a prime example of how software can be made more reliable by tolerating failures instead of relying on it not to fail, to come back to GP's point about failure-tolerant designs.
Exactly. I often have a hard time getting this point across, glad I succeeded.
Boeing really aren't a good example for anything besides negligence and how to game regulators.
But the concerted marketing effort pays dividends. Solarwinds is almost the only choice for government. For potential attackers its a big red arrow. Exploit a rent-seeking company that writes mediocre software and exists mostly to cash in on "best practice" lock-in with government contracts.
as far as i know nobodys really addressed the elephant in the room. Solarwinds is still a preferred government purchase for monitoring. every company that was affected by it still uses it (or at least hasnt publically refuted it) and no government official has come forward to admit they will discontinue it. Solarwinds hasnt offered any remediation or major changes in their development, leadership or code. just patch, rinse, and repeat and try not to pay too much attention to the issue.
But Microsoft's "system of trust" was undermined. The attacker was even inside Microsoft and Azure. No anti-virus, no "defender", no amount of basic or advanced telemetry caught this.
FireEye alone caught it... By accident.
The real "elephant in the room" is that software security continues to be marketing theatre. Closed-source software should not be trusted and consumers should not believe that any due diligence has been done to protect them.
Overall I tend to agree with your general points, but, this statement.
Would you say that seatbelts are "safety theatre" because some people die in car accidents while wearing a seatbelt?
I really don't think it's fair to attribute all software security as BS, especially given the resources state actors throw at breaking software. It's like arguing your seatbelts don't work because your 4 door sedan was hit by a bus.
Seatbelts are real security measures. Still, more than 60K people die every year in motor accidents in North America and in the EU. Seatbelts cannot save life in all cases, such as outrageous speeds, but they are a sane measure for most -- if not all -- use-cases of driving.
In this compromise, the major purveyor of OS and security products itself was compromised. No company, no $agency did due diligence. Some of these victims are supposed to be in the business of high-assurance and due diligence.
None of the systems that we are conditioned to believe to be effective actually worked. And it was a standard use-case for all the victims.
I think the elephant in the room is actually a more general issue that is cross platform and independent of the product implementation.
It’s 2021 and we are still ignoring the fundamental “best practice” that we’ve known about for at least 20 years.
Systems should be isolated from each other unless there is an overwhelming need for them to be connected and everything should run with the least privilege.
Centralised credential vaults and binary artefact repositories are an obvious example.
The two computers are developed by two independent teams who are not allowed to talk to each other. Two different CPUs, two different algorithms, two different programming languages.
The idea, of course, is a design problem with one does not propagate to the other.
If one program is used in all your infrastructure, all your infrastructure is compromised when that program has a bug.
As a marine saying goes, never take two chronometers with you to the sea. Take either one or three.
1) Build servers: Maybe by keeping it possible to build one's software on a laptop — then, when it's time to release something, one can build the software at the build server plus some people's laptops and compare hashes.
2) ???
3) Different people who don't talk with each other, do security code review, independently of each other? Hmm does that make sense
And we should have checks built into the build pipelines and checksum the files that go into the build. We have Merkle trees for a long time. I think, it would be possible to detect injected code.
Not accurate re: leadership. A new CEO just started in January, and their CTO was either terminated or he left just after the hack news.
Second, with such negligence? They deserve to be shut down. This isn't a highly sophisticated zero day exploit, there were multiple huge failure at Solarwinds that allowed the attack to happen.
A far better system is "no fault" where the company has incentive to be open about problems and finding solutions.
Calmly, our CEO said: 'No. If there's anyone in this company who will never make that mistake again it's him.'
I was in the room when the CEO told that (one of the nice things about small companies) and have had contact with the person responsible for a few years until he got another job. True story ;)
There were changes though: the 'two pair of eyes' principle was enforced a lot stricter from then on.
On April 28th 2021 they will announce Q1 2020 results. THAT's when we'll know for sure how bad this hurt them.
They may release less than sanguine guidance on Feb 25th, we'll see.
The reason why I know this stuff I will leave as an exercise to the reader. :)
The malware keeps a "breadcrumb" trail of each machine it infects, and based on investigations by Symantec [1] it was determined that every Stuxnet sample from Natanz originated from outside suppliers.
It was so sophisticated at not being detected that it went under everyone's radar for 5+ years.
Stuxnet behaved in exactly the same way.. neither did anything that would be detectable unless certain criteria was met and a secondary payload sent.
Stuxnet was very sophisticated to hit a relatively narrow target by comparison.
So I think they are both sophisticated, but not comparing them directly.
I would love to know how many machines got hit by SOlarwinds - however, its not about the machines - its about the value of the data it slurped... So even if it was one exchange server, with 10,000 treasury/nsa/whatever accounts, that one machines intel value is quite high.
inflitrate an airgapped network -> silently spread within -> silently destroy complex unique equipment
compromise central infrastructure -> silently spread to customers via compromised updates -> silently exfiltrate useful data/create an advanced persistent threat (APT).
The APT/Exfiltration work is much more close to the sophistication of the virus that attacked the centrifuges than the bad password. For all we know the hack could have levereaged information gained via kaspersky, telegram, or whatever other software to do sophisticated things much the same way we asked Siemens for their help.
We don't have enough information to make an assessment yet.
That's not fair. It should sound like this: "the initial attack vector was a Windows 0day requiring 0 clicks, delivered by a USB stick in a parking lot."
https://arstechnica.com/information-technology/2020/12/18000...
“Our analysis indicates that these compromises are not self-propagating; each of the attacks require meticulous planning and manual interaction. Our ongoing investigation uncovered this campaign, and we are sharing this information consistent with our standard practice.”
https://www.fireeye.com/blog/threat-research/2020/12/evasive...
You are saying my comparison is not fair because the USB stick actually had a 0 day which implies sophistication, but the supply chain build server was producing code and was also acting as a 0 day, in fact the actual payload appears to be significantly technically complex and custom.
Where do you draw the line between sophisticated and not?
USB in parking lot -> 0day -> horizontal movement 0days -> vertical movement 0days -> Payload delivery.
Bad admin credentials -> compromised build system -> supply chain 0day -> horizontal/vertical movement -> exfiltration of sensitive data.
[USB in parking lot -> 0day] to [Bad admin credentials -> compromised build system -> supply chain 0day] or
[USB in parking lot -> 0day] to [Bad admin credentials -> compromised build system] or
[USB in parking lot] to [Bad admin credentials -> compromised build system]
The chinese successfully pulled this off at Lockheed in the early 2000s
They would get lists of employees who attended various conferences, then email them a trojan saying "hey we met at conference X - here is something for you to click on"
And employees would fall for it.
The malware would TRICKLE out data very slowly so as not to be noticed.
They did this for a LONG time.
At the time, lockheed only had 3 egress points to the internet - and they had 110,000 laptops.
Another attack vector was the same as stuxnet - airgap; attack a supplier in taiwan, infect any USB stick put into the suppliers machines, then transfer once that USB stick was put into the Lockheed machine.
When the ruse was discovered - the chinese removed the throttle and they attempted to firehose out as much data as they could before they were cut off.
Stuxnet "worked by first causing an infected Iranian IR-1 centrifuge to increase from its normal operating speed of 1,064 hertz to 1,410 hertz for 15 minutes before returning to its normal frequency. Twenty-seven days later, the worm went back into action, slowing the infected centrifuges down to a few hundred hertz for a full 50 minutes. The stresses from the excessive, then slower, speeds caused the aluminium centrifugal tubes to expand, often forcing parts of the centrifuges into sufficient contact with each other to destroy the machine."
The solar winds hack is an otherwise unremarkable trojan that spread exclusively due to the bad security measures of solarwinds.
>The solar winds hack is an otherwise unremarkable trojan that spread exclusively due to the bad security measures of solarwinds.
We don't have sufficient information to say what happened after the trojan landed is unsophisticated. All we have is the idea that microsoft benefits from making their attacker seem more competent than they are, which is true.
Which makes me think: A bad actor could create a really good open source library or package, wait for everyone to use it and then introduce malware into it. Or they could "accidentally" add a security vulnerability and exploit it.
"There is another theory which states that this has already happened."
This often happens with Chrome browser extensions that change hands and the new owner then injects malware or crypto mining or keylogging etc. into the newly acquired extension.
You are vastly underestimating the lengths that this attack when to avoid detection. They didn't just hijack Solarwinds updates and inject malware. They did things like concealing the commands of malware in legitimate looking Solarwinds packets. When they infiltrated a place, they had the malware sleep for months. They carefully avoid standard malware detection techniques and you can only do that if you have very sophisticated engineers. This isn't just some scripts someone pulled from the internet.
The US does not need to hack SolarWinds because they probably could use a court order for the same outcome to distribute rigged binaries for political enemies.
A thousand engineers is a huge project. You have no better knowledge than Microsoft about the attack.
https://web.archive.org/web/20061215050427/http://www.matasa...
Agent-based endpoint management is super convenient and is mainstream in modern IT management. But nobody grows up wanting to write the communications software for enterprise server inventory software, so the code quality on these products, which are ubiquitous, is awful. But 99% of enterprise buyers aren't even slightly motivated by software security when making purchasing decisions, and everyone knows it, so vendors buy fig-leaf audits from vendors who will sell public-facing documents for 2p1w engagements, and IT purchasers, whose performance reviews are based on completing projects and not protecting their companies from $500,000 purchase orders for what is effectively malware, accept those bogus reviews and get on with their lives.
Nothing is going to change about this, because nothing is going to alter the incentives. Reporters like Nicole Perlroth will try to blame it on ex-NSA hired guns (as if you needed the exploit chains NSA buys to break any of these systems), but I can't think of any realistic policy that could be applied to stop these kinds of attacks, not without massively disrupting the technology industry at the same time.
Best get used to it, is I guess my point. I mean that sincerely, not as a sort of appeal to right the ship; the ship is already upside down.
Why wouldn't Dan Geer's proposal to attach traditional products liability to closed source software improve the situation? Over time, source availability and reproducible builds should make this kind of thing a lot more difficult without wrecking anyone's budget. No?
Just very simple things, like reimplementing non-performance-sensitive C software from the 1990s and 2000s in simple memory-safe languages; it can't happen, the budget to make it happen would totally disrupt P&L at large companies; repeat with every well-known risk this kind of code is exposed to. I don't think we know how to solve this problem, which is why I tend to recoil from policy proposals to "solve" it.
Humiliating the competition is good marketing.
It just shined a light on how shaky the foundation is we’re standing on. This is relatively tame considering its just some back-doored bolt on software.
What happens when it’s the build compiler for the Windows OS? How about the intel NIC firmware?
Our only answer seems to be even more automated updates. Yet that means a slip up in chrome build security could compromise tens of millions of people in a week.
It’s mind boggling how fragile all of this is. It’s barely safe with insiders behaving correctly.
How many people do you think have to sign off on a code change driving the Qualcomm chipset in your phone? My guess is 1, and that’s if it’s a “modern” development process. So that’s it, two people colluding and you’ve got a back door worth a fortune to a nation state.
What a fucking mess.
No, exactly the opposite.
Automatic updates are insane. If software updates itself automatically, you need to assume it is compromised. If software requires manual updates, you need to assume that the more frequent the updates, the less secure it is.
Security companies benefit from having adversaries be big and scary
Hacked companies benefit from adversaries being big and scary. Nobody blames you if you get hacked by the russians. Getting hacked by teenagers looks bad.
Edit: that said, i feel like i should add, i personally dont feel, based on what we know, that this particular attack was a bunch of kids.
https://www.fireeye.com/blog/threat-research/2020/12/evasive...
No high school kid is doing that.
We’re Geekbuilt.®
Developed by network and systems engineers who know what it takes to manage today's dynamic IT environments, SolarWinds has a deep connection to the IT community.
The result? IT management products that are effective, accessible, and easy to use.
This is every website of any company selling to enterprise customers. It's incredibly frustrating as a reader.
I imagine anytime someone actually answers the question "Yes, but what do you do?", they get fired.
[1] https://www.solarwindsmsp.com/blog/do-your-vendors-take-secu...
It's a more complete picture of how much modern businesses value security or pentesting audits. The answer to that is exactly zero effs.
As long as businesses think there is no ROI in security, this will stay the same.
The only thing sophisticated about this was that from code to deployment there wasn't any single audit of anyone. And that's a bad joke by any security measurement.
If any intruder can stay undetected THAT long, I don't wanna know what's up with MSFTs infrastructure. Must be an open door for everyone, and probably still uses the Perl pre-release based pipeline.
I would not be surprised to find hair saloons with better security practices than our military.
Pretty sure texans would not like to be caled balkans, even though balkan culture is somewhat similar to texan
On December 8, 2020, the cybersecurity firm FireEye announced that red team tools had been stolen from it by what it believed to be a state-sponsored attacker.[106][107][108] FireEye was believed to be a target of the SVR, Russia's Foreign Intelligence Service.[27][109] FireEye says that it discovered the SolarWinds supply chain attack in the course of investigating FireEye's own breach and tool theft.[110][111]
After discovering that attack, FireEye reported it to the U.S. National Security Agency (NSA), a federal agency responsible for helping to defend the U.S. from cyberattacks.[1] The NSA is not known to have been aware of the attack before being notified by FireEye.[1] The NSA uses SolarWinds software itself.[1]
The sophistication of the malware required state powers and assistance of the industrial company that manufactured the control systems. Those systems are not cheap and reverse engineering the control system software required detailed knowledge of the embedded hardware and software.
Since Microsoft itself was compromised via SolarWinds angle I'd take the president's statement with a grain of salt and probably less objective than it would be otherwise.
“We have been actively looking for indicators of this actor and can confirm that we detected malicious SolarWinds binaries in our environment, which we isolated and removed,” the company said in a statement.
https://en.wikipedia.org/wiki/2020_United_States_federal_gov...
This has a simple fix I've advocated for years.
Put the firmware for disk drives, USB sticks, embedded systems, etc., in ROM. Or at least provide a physical write-enable switch for updates.
I have no idea why people responsible for security do not demand this. I would expect them to be really sick and tired of "we have no idea if our firmware is infected or not, because we allow any piece of software to modify the firmware."
As an alternative, a 'security card' slot; similarly easily replaced or with an internal switch depressed re-'paired' to a new key.
It must be out of the ordinary. The ceremony must have weight and differentiation from typical activities. True going inside and moving a jumper is fine for those willing to open the case, but it must work on Gradnma's Dell; while under warranty, without voiding the warranty.
Was the attack otherwise sophisticated and just relied on an easy entry point? So the breaking in was easy, but the plan to steal once inside was sophisticated?
https://www.sec.gov/Archives/edgar/data/0001739942/000162828...
The amount of effort and different techniques they've put in to remain undetected for months (years?) while infecting a lot of actors is pretty impressive.
Were any in use at all? No
So, nothing really sophisticated... just exploiting the bad design of existing operating systems.
This will continue for the foreseeable future.
Anyone have insights on what part of the attack might have taken a thousand people?
To call it the largest & most sophisticated ever seen sounds like ignoring some that others have mentioned here, and the fact that we don't know what others have not been publicized yet, including things like this (https://www.schneier.com/blog/archives/2021/02/chinese-suppl... where we don't know everything, there can be debate; but given means, motive, and opportunity, I think at least keeping our eyes open and attempting to adopt wise practices based on realistic trust levels and long-term track records, could be a good idea.
Plus I question the credibility of MS on security or anything they say, given their long track record, and when compared to others (like maybe OpenBSD), and things like this as just one very recent example: https://www.schneier.com/blog/archives/2021/02/on-vulnerabil... ...?
My take is that this is mostly on the heads of all the many gov orgs (I'm ignoring corporate land right now), who have become so dominated by gui-ninjas with stockholm syndrome for MS and other black-box closed source systems that they honestly just don't think about the underlying system fundamentals at all. Not only have I contracted and seen some of these systems first hand (and the bad management that engineers them), but I have a family member is is the head of IT for a state gov org, and I've heard about how lackluster the response has been internally. There is just a response to update systems. No fundamental reconsideration of sourcing practices, no fundamental reconsideration of security practices, just lazy hope that it doesn't affect them. I highly suspect most of these orgs are completely compromised already on the tech side (and other ways too, cough Epstein cough), and because of lack of accountability, nobody really gives a fuck.
Then at the very high levels at the federal level, those "leaders" don't care because they are playing the classic good ol boy backscratching contract and bid game, and they get their cut, so they could care less that a few years down the road something like all of OPM is compromised. Congress, the one entity that should be in a position to start passing legislation to address these issues, is instead completely compromised on both sides of the isle by the same kind of realpolitik, where K-street writes the legislation along with a few staffers they will hire later for their "help", congresspeople don't even read the bills, but vote according to donation dollars and influence gained. The whole system is completely disconnected from reality, and the number one thing that is lacking is the largest four letter word in DC or in any government org...
Accountability.
Unfortunately it's an idea that is going to be a very hard idea to get rid of (Right now my computer is running CarbonBlack, F-Secure AV, EgoSecure, CyberARK, and at least a couple of others I forget. I assume many of you on corporate machines have the same problem).
https://a16z.simplecast.com/episodes/solarwinds-anatomy-of-h...
It is as if a group of mates who codes a hack like this for a hobby/interest are disorganised have no experience in delivering software? Such ignorance for a self confessed expert. Scary.
The word “seen” is doing a lot of work there.
(Think, probably, and fair, are doing some too.)
For more info see: https://www.linuxfoundation.org/en/blog/preventing-supply-ch...
[1] https://www.microsoft.com/security/blog/2020/12/18/analyzing...
It looks like they are trying to limit their liability by claiming there was nothing at all that they could have done.
Reality: 'solarwinds123'
Not trying to bad-mouth MSFT, just saying that presidents are PR.