SolarWind, enough with the password already | Hacker News Reader

SolarWind, enough with the password already

gru.gq

219 pointstroydavis132 comments

I do think a lot of folks are missing the main point that no matter what security theater is in place, a state actor with enough motivation is going to breach it. They're not going to send someone after the password protected parts, they're going to send your recruiter the most irresistible candidate--the perfect background, right out of your favorite school and with expertise in exactly your tech stack. You'll get pages and pages of glowing recommendations from people inside and around the industry. They'll ace your interview loops, be loved by all your engineers and managers, and they won't bat an eye at the lowball first offer you give them. They'll move up the corporate ranks with ease and be everyone's friend... and then the best security in the world doesn't matter one bit.

care to add examples? has there ever been a case when state actor sent a rogue employee and succeeded? because in serious organizations there are robust defenses even against potentially rogue employees

It is _far_ more common than you think. A few years after I started at MS this guy was caught: https://www.theatlantic.com/international/archive/2010/07/wh... I knew a lot of folks internally that worked with him, knew him, etc. and had _no idea_ he was a spy or agent. He was the model of a perfect employee or hire for MS at the time.

wow, so this guy worked as senior developer for infosec vendor NeoBIT (neobit.ru/partners openly cites FSB as their #1 client) and then gets a QA tester job at MSFT, no wonder DHS tracked him since he applied for visa.

see, these people are not that smart, actually

Sure, but those are the people that have been caught. The ones actually making a difference will likely never be known.

Stuxnet is the one that comes to mind first, where a Polish mole was able to be hired at an Iranian nuclear enrichment facility.

Could you cite that? Why would they need to infect flash drives all around the world if they had a mole directly inside the facility?

Edit: grepping Wikipedia for “mole” it does appear that there was an Iranian one working with the Dutch government.

https://en.wikipedia.org/wiki/Stuxnet#cite_ref-156

I thought China does this all the time.

https://www.fbi.gov/news/speeches/responding-effectively-to-...

economic espionage is different thing and yes it exists. for example Huawei equipment - is cheap and low quality Cisco ripoff. But it is no different than employees in Silicon Valley changing companies and taking their knowledge to the competitor. In the former, people relocate from one company in the valley to China, in the latter, people go from one SV company to another. Same thing for me

Has Lucid Motors's founder (former Tesla employee) conducted economic espionage or is it simply capitalism working as intended?

> In the former, people relocate from one company in the valley to China, in the latter, people go from one SV company to another. Same thing for me

You may wish to ask Nortel about that:

* https://www.bnnbloomberg.ca/did-a-chinese-hack-kill-canada-s...

* 7m ago: https://news.ycombinator.com/item?id=23743869

There's a long list of criticisms against Huawei:

* https://en.wikipedia.org/wiki/Criticism_of_Huawei

I read the Nortel thread and apparently the Nortel management killed the company.

https://www.nytimes.com/2019/11/06/technology/twitter-saudi-...

Not exactly what OP mentioned, but something like that did happen in the past:

https://www.theguardian.com/technology/2010/jul/14/russian-s...

Are there? I've done some hiring and I don't recall any procedures in place for the detection of foreign intelligence assets. I suppose it is possible that I have never worked for a serious organization.

Security clearances would help, but obviously that's not something that's going to be tenable for all companies.

I think the main way to prevent issues is to just assume at any time you could be infiltrated. Don't mistrust all your employees, but don't live with lax security policies that allow a person to get away with something undetected.

Google learned this the hard way, between the Snowden revelations and China's cyberattacks.

Audit trails are important, as is, um, not giving read access to user data without really really good justifications. Even beyond espionage, employees could stalk personal contacts (as happened at Uber in 2016 and Facebook in 2018).

My reading of the comment was that the protections are in the form of limiting what insiders can do / auditing what they do, rather than in detecting them during the hiring process.

You don't receive insider threat training at work? There's plenty of examples given during that sort of training.

The FBI gave some examples in their published literature on the topic.

https://www.fbi.gov/file-repository/china-risk-to-corporate-...

I have yet to come across such an organisation, actually, and I'm a security consultant. The scenario is usually if an employee's systems get hacked, very few "serious organisations" seem to have procedures for rogue employees, at least seen from an EU perspective (perhaps the culture is different in the USA or in poorer parts of the world).

it is hard to catch expert rogue employees, but there are systems like conducting security clearance, DLP, UEBA, airgapped system separation (low security, medium security, high security), and ton of other security layers/controls that together can give some assurance

While this happened to me in 2004, I can still see it working even today: http://boston.conman.org/2004/09/19.1

That's an excessively long game--much easier to compromise someone already in a position.

I don't know that getting a job is anywhere near "an excessively long game" in the context of publicly known instances of espionage. Have you heard about the group of Russian spies that spent decades blending in (somewhat) to US lives and doing nothing much?

https://en.wikipedia.org/wiki/Illegals_Program

They were arrested around 2010; one had allegedly been a spy for ten years, and another since the 1970s.

States and powerful, well-funded adversaries have all the time in the world. Look at the 9/11 attacks, the hijackers were taught how to fly planes by US flight schools!

why not both?

It seems like there are two (or more) issues:

1) Any inviting target will be ... targeted and so likely breached at least at some points. The article is correct that expect a single target to invulnerable is a mistake.

2) US (and other corporate) systems download untestable binary blobs from third parties as standard operating procedure, not because no one understands the risks but because they aren't concerned with the eventual costs (and happy with the immediate benefits). Similarly, everything gets connected to everything in a whole variety of systems (the sabotaged water system and the cars sabotaged by auto-maker-server-sabotage are just things coming through in the last few days or weeks).

It seems like a critical analysis should look at point two so the article's nothing can be done implication is here not appealing. Here, the problem seems hypothetically addressable. This may also be unsolvable but it seems closer to solvable. If the US imposed cyber security standards, a solution would seem more likely, etc.

An engineer with a great pedigree and work background and social finesse who passes all interviews and interaction without detection... Willing to take on a high risk role... All for the good of the homeland?

I think you're describing a work of fiction. It's practically a James Bond or Bruce Wayne-like character. This particular person would be extremely hard to find/hire/compel by some competing nation, if they even exist.

Meet Jack Barsky.

https://en.wikipedia.org/wiki/Jack_Barsky

Doesn't quite fit the image of perfection described above

You may be underestimating the recruiting efforts and budgets of major nation-state intelligence agencies - Russia and China are doing exactly the same thing the US has been doing for a long time. Spending some tens of millions of dollars to accomplish inserting an advanced persistent threat inside of software used by some huge percentage of the US federal government is a tiny drop in the bucket of the budget of such agencies.

The main difference is that the US has historically accomplished it through a role in many cases as a vendor, or a supplier of essential high tech stuff (CIA/NSA and Crypto AG for instance).

The good of the homeland and money. You're essentially being paid to do two jobs. Go work for a good salary in a foreign country, and then collect a good salary from your intelligence agency.

Hell if you're targeting the US, why not do it? The worst that can happen to you is that you'll be deported and go back home to work a great job within the government.

> The worst that can happen to you is that you'll be deported

This is demonstrably not true.

> I think you're describing a work of fiction. It's practically a James Bond or Bruce Wayne-like character.

Diane Feinstein, the _Chair of the Senate Intelligence Committee_, had a Chinese spy as her personal driver _for 20 years_. So yes, slipping engineers into tech companies is not that complicated and in fact happens all the time.

Isnt that the FBI's job to vet people?

And then they move up the corporate ladder, loved by all? Read OP's original description, not the same. Driver spy is not a counterpoint.

So... does anyone use Kaspersky anymore?

I mean any CTO or underling in any company could be an unknown "asset" --maybe mitigated a bit if they are scrutinized because of official contracts with GOV. These types may by simply naive and sympathetic to some politics elsewhere. All governments should be weary of these types, lest they have undesired outcomes.

But you don't have to run faster than the bear, you just have to run faster than the slowest person. All of this is theoretically possible but the cost has to be justified by the intelligence gain and there's very few security targets that are simultaneously worth the cost and not achievable via simpler means.

> Security theater is the practice of taking security measures that are intended to provide the feeling of improved security while doing little or nothing to achieve it.

https://en.wikipedia.org/wiki/Security_theater

What I find really funny about Solarwinds - is that in the medium to large ISP sector it's always been an absolute joke. Nobody of any consequence or real size has ever used it for network monitoring.

Once you get to the scale of ISPs that have 50,000+ customers, or are supporting more than that through other smaller ISPs that are downstream of them - the monitoring and network automation tools are almost entirely open source, and some combination of GPL/LGPL/BSD/Apache/MIT license. Combined with custom things written in house to tie together different tools for a company's specific business needs.

What you'll have typically is a collection of network equipment that may have closed-source operating systems (cisco, juniper routers and switches and similar, optical transport platforms from vendors like Infinera, Ciena), but everything managing and monitoring them is open source and runs on a *nix platform.

If you have the in-house Linux/BSD knowledge to run the world's most powerful and popular open source networking tools, there is no need to ever touch solarwinds.

My job interacts on a regular basis with all of the different pieces of the puzzle that make up solutions which are, in my opinion, vastly superior to Solarwinds.

In the serious ISP business, if you ask the persons who admin the monitoring tools what they think of solarwinds, the answer you'll almost universally get back is "Windows GUI button pushing tools for enterprise end users who don't have the knowledge or motivation to really understand what's going on under the hood of their network".

I don’t know what you consider a “serious ISP”, or what serious people do, but I’m confident that 10/10 of the big ISPs have Solarwinds on their corporate network.

The customer facing stuff may be different, but once you own the LAN, you own the company.

Their play was a cheaper, easier, multi-vendor toolset for enterprise networks. You’d pay half of whatever the Cisco dreck costs, and not need an army of consultants to tend it.

I’d argue that the vast majority of network people do not demonstrate strong Unix skill sets. Windows tools FTW in most enterprises, as dumb as that may be.

> I don’t know what you consider a “serious ISP”

Something that's big enough and has a wide enough enough reach that other network operators with presence at major IX points know its AS number by sight - the same way people will recognize AS174 as Cogent or AS1299 as Telia, for instance. Or an ISP that is big enough that its wholly-owned/controlled fiber network spans most of a state, or several states, and has other major ISPs riding on it (whether as lit 10/100G customers, or dark fiber IRUs, or whatever).

Something big enough to have a whole team of guys with bucket trucks and fiber equipment running around building the physical internet, while at the same time there's an office/work-from-home environment with 4 or 5 people whose job title has some form of "network engineer" in it, building the network at OSI layers 2/3.

Or for an ISP that is not middle-mile/last-mile focused, and is rather a hosting/colocation company, something with significant datacenter presence at or near major IX points, as measured in square feet of space leased, kW of electrical power and cooling.

> I’m confident that 10/10 of the big ISPs

Which ASes would those be? If you can find a documented instance of a top-50 (by CAIDA ASRANK size) ISP using solarwinds to run its core stuff, please provide a reference to it...

https://asrank.caida.org/asns

The most likely source of said claims would be something like SolarWinds's customer list, which they took offline in December.

http://web.archive.org/web/20201214030038/https://www.solarw...

"Our customer list includes:

...

All ten of the top ten US telecommunications companies"

I see... AT&T, Sprint, Comcast, Level 3 (now CenturyLink, still AS3356) for US-based ISPs. Telecom Italia made the shortlist, too.

(And an honorable mention for Cisco, which was also apparently explicitly targeted: https://tools.cisco.com/security/center/resources/solarwinds...; "While Cisco does not generally use SolarWinds for its enterprise network management or monitoring, we have isolated and removed the Orion installations from a small number of Cisco assets.")

That said, it doesn't say anything about _how_ the ISPs were using SolarWinds, just that they were in some capacity. But with any infiltration, it doesn't matter if it's widely used, so long as it's used somewhere that can be used as a launching pad for a follow-up attack.

edit: apparently CenturyLink rebranded as Lumen last year. They're still AS3356 (and its subsidiary networks) to me.

> I don’t know what you consider a “serious ISP”

I understood this as referring to their definition higher up in their answer:

> > ISPs that have 50,000+ customers, or are supporting more than that through other smaller ISPs that are downstream of them

a lot of CIOs are very cheap and prefer to keep their IT teams "lean" (understaffed, underpaid, underskilled) as they see them as cost center, and rather hand over couple mills to a vendor like SolarWinds to install their "automation/AI" pixie dust.

So instead of investing into own employees who have the best interest of a company in mind (because you know job satisfaction and job security) -> they prefer investing into third party vendor whose interest is only to keep renewing multi-mill contract year after year while keep delivering barely above what's required to keep things afloat

That's true with pretty much any enterprise-aimed tool, isn't it? With enterprise IT generally expected to run a very mixed pile of different stuff, without having the resources to specialize much on each of these things. Whereas an ISP, SaaS, specialized hosting company, ... puts much more emphasis on specific stacks and mastery of them, and often more recognition that investing in these things is not just a cost center.

Yes - though a big enough ISP with 40, 50 staff or more also needs a very wide range of common enterprise software tools, which the people running must have full mastery of.

You've got stuff going on like billing/accounting systems, call centres, GIS systems for outside plant fiber construction and aerial+underground utilities work, HR software, VoIP systems, IDS and NAC systems. Lots of things that support the ordinary office-worker environment of the ISP in addition to all of the tools that automate and monitor the network.

>Nobody of any consequence or real size has ever used it for network monitoring.

At least one of the big 3 telcos uses it very extensively for network monitoring, inventory, device configuration enforcement and alert generation.

I am much less surprised to hear that in the context of a company that is a "baby bell" / ILEC such as Verizon, Frontier, Centurylink (Former embarq/uswest/qwest/whatever), than I would be if I heard that they were using solarwinds inside NTT or Telia.

Without going into a whole lot of personal opinion and detail, the business practices and management methodologies in a ILEC are very different from other ISPs.

Out of curiosity, can you give some examples of open-source monitoring tools that large-scale ISPs use?

I'm not a large scale ISP but I do have to monitor quite a lot of stuff.

You'd be amazed at how much you can monitor with Nagios/Icinga(1,2). They are written in C but call a lot of external stuff written in whatever you fancy and that's the power, right there. Bodge upon bodge! There's no single technology in these beasties. The interface between the system and the plugins is very basic to say the least, so you can throw whatever at it as you require. I'm a sysadmin not a programmer and need to get jobs done.

We currently use Icinga 1 with a dash of Netdisco and I intend to migrate to Icinga 2 with DIrector etc.

That said, I have dallied with OpenNMS many times ever since the project began - it's too good to ignore. Zabbix also turns my head quite often.

opennms is simultaneously

arcane

weirdly laid out

a massive java memory hog (thankfully, RAM is cheap, giving an opennms VM 16GB of memory isn't a big deal anymore)

extremely powerful

something that has 450+ pages of documentation

totally open source

extensible to support monitoring of massive international-scale networks

the first and most important thing is to have the correct network architecture and engineering to make effective use of the tools, and not have a network that needs a lot of babysitting in the first place. after that's taken care of as an over-arching and continual business process:

there is no one single god box piece of software that is the be-all and do-all of network management/monitoring for an ISP. Some things come close, such as LibreNMS when used as the sole tool for a small ISP. But most often it is a patchwork quilt of many different things, each used for a discrete purpose.

in no particular order:

opennms

a combination of (influxdb + telegraf + grafana)

librenms

provisioning and automation tools like ansible

various in house things built on traditional RRA files and rrdtool

tools like netbox for keeping track of datacenter customers/hosting environments

phpipam or nipap for IP address management

various self-hostable wiki software packages for internal documentation

various types of self-hostable ticketing systems, monitoring systems that integrate with a customized asterisk system for NOC phone workflow

4 or 5 different tools that fill the same role as smokeping

wireshark

lots of different things for analyzing netflow data (Elastiflow or other)

ELK stack stuff, elasticsearch/logstash/kibana, customized as needed.

in house setups for openstreetmap tile servers and map presentation, to pull data from back-end mariadb databases and present them on monitoring displays.

GIS software like QGIS and a PostGIS backend

lots of different possible things done with custom code and postgresql, mysql/mariadb, or similar

if you go through the PDF slideshows for the powerpoint decks at the last 4-5 years of the NANOG, RIPE and APNIC conferences you'll see discussion of some of the most popular network automation and monitoring tools.

Claming the hackers would have been able to compromise SolarWinds even with good security pratices does not absolve the company of having actual good security.

I know there are people that can pick a lock. Still gonna have them on my doors anyway.

I think the point is that if you accidentally leave your door unlocked, you can’t truthfully say “well, if I didn’t do that the burglars couldn’t have gotten in” while you also have ground floor windows.

If we're using the door analogies... it seems like the solar winds attack came because someone figured out how to sneak-in the "doggie door" (hacked the auto-update function, maybe call it the "delivery door" like homes had 50 years).

And problem wasn't so much that this happened (cause indeed, shit happen). The problem was all these key enterprises (Microsoft, government agencies, etc, etc) had "doggies doors" when really they should have only had the most secure doors themselves.

And sadly, unlike the retail situation, where a bank can decide they want only a secure door to their vault, today's enterprises have basically decided the benefits of giving multiple access to other enterprises trumps the security costs, since they never pay the costs of bad security anyway.

The article notes that none its analysis absolves SolarWinds, and repeatedly goes out of its way to knife them.

My problem is not with SolarWind but with the analysis that keeps raising the password as the one and only problem. “If only they had good password hygiene then the company would’ve been totally safe against the Russian intelligence services!”

That is just not how things work.

To go further. The password was on GitHub from 2017 to November 2019. The first test build to see if they could backdoor things was in October 2019. If the password was the problem, why wasn’t SolarWind hacked in 2017 or 18?

The only explanation is that it wasn’t an operation that existed back then. It wasn’t a target for the SVR at that point in time, or they weren’t able to service it with their operational capacity. But regardless, the critical factor here is that the RIS started this operation, not that the password was bad or available on GitHub. (Or whatever the issue is with the password.)

Let’s discuss whether the operational concept (CONOP) of hacking a civilian target to get into the supply chain and hit other targets is acceptable in cyber espionage. It seems to be acceptable because that is a methodology that everyone uses.

My point has not been that you just can’t win against Ho Chi Minh, or that SolarWind was particularly negligent (or not, their security posture was abysmal but also irrelevant)... my point is that we should focus on what actually matters — the CONOP. Because if the US sanctions Russia for this operation then the US is locking itself and it’s allies into a position when this CONOP is off the table. If that is what everyone agrees with, fine. But it’s the real discussion to have. Not what sort of security SolarWind did or (realistically) did not have.

Why is it not possible that they didn't discover it or pay attention? Or what makes you think if someone gets pushed to github it gets immediately used? I'm sure it gets scanned and indexed by all sort of actors fast but they must work with incredible amount of information.

Keep in mind that there's a ton of institutional shorting on $SWI and that's where a lot of these attack articles are coming from.

Having said that, SolarWinds is garbage software even without the security vulnerabilities and I hope it goes the way of the dodo. Source: I've had the misfortune of using it on multiple contracts.

Because anyone who is smart or has competent security teams have been looking to drop them like a sack of potatoes. I work with a ton of customers that have/had SolarWinds and the general sentiment with them is that they are ripping it out and replacing with alternatives.

I would be willing to bet that the majority of SolarWinds sales are relationship based, not based on technical wins.

It is extremely dangerous for a company that is based on relationship sales to require their economic buyers spend a non-insignificant amount of political capital defending bad practices. TBH most buyers aren't going to stick their neck out for them unless they have a really good reason. Even then, security teams and legal teams might poo-poo the purchase and for most people in IT it is easier to find an alternative.

Being reliant on relationship sales rather than technical wins is not by itself a bad sales or growth strategy... But as a company who has taken that approach, you have to ensure you don't do stupid stuff, or the stupid stuff isn't something that requires people to stick their neck out for you on.

I imagine this is why institutional investors are shorting $SWI more so than anything. Their customer churn is going to probably not be pretty and they are going to have to work really hard and hope that in 2-3 quarters people have forgotten about it.

FIFY: I would be willing to bet that the majority of Enterprise sales are relationship based, not based on technical wins.

Their Q4 2020 Earnings were pretty good. They issued downward guidance on their Q1 2021 Earnings but it looks like they don't expect to lose THAT much business.

Most of their subscriptions are yearly so most people are stuck paying them for the duration of the contract regardless of whether they ripped out their Solarwinds stack or not.

Are you familiar with Sturgeon's Law?

The underestimation of the offense is done to ridiculous extents. The NSA pwned most of the world and exfiltrated data for years before it ever came to light (from one of their own!), yet all non-technical and even some technical people talk about security as if it's a bit you turn on or off. Unless you intend to spend as much money and resources on defense as USA/Russia/China does on offense, it's an uphill battle you will only seldom win. And you only have to lose once to lose almost everything.

But that doesn't absolve you from trying! I totally agree with the point that a TLA has the resources to get into the network if they throw everything they have at you. A strong password might have changed nothing. But it is still a total failure on basic security on SolarWinds side. And honestly, the fact that they're now blaming an intern shows that this was not the one small weak point the attacker found, but a cultural problem.

I think I remember seeing solarwinds123 as a password around 2011 (perhaps as a default password within Orion?) but couldn't find a web search for it.

But I did find this example which I find amusing:

"For example if your account name is 'orion@mycompany.com' and the password is SolarWinds123, that's what you put in for the authentication." https://thwack.solarwinds.com/product-forums/network-perform...

Did some more searching; Google's filter by date is broken since companies update old urls to include recent news headlines.

https://www.google.com/search?q=solarwinds123&tbs=cdr:1,cd_m...

People want an easy fix. It is easier to blame a single person for now follow a single rule, than to set the standard that executives should put controls in place to ensure that policies are being followed.

I understand the spirit of your comment, but I want to point out

> It is easier to blame a single person

There is only one person to blame. The CEO.

Huh? But the board appointed the CEO. So why isn't it the board?

Or wait, the board was appointed by shareholders. So why isn't it shareholders?

And so on...

The board can't be expected to know as much as the CEO, the shareholders can't be expected to know as much as the board, and so on.

As far as I'm concerned, the #1 responsibility of the CEO is to take blame for fuckups.

Yes, I consider that a higher priority than making profits. Because if the CEO is unable to make profits, then the CEO has to own the fuckup of not making profits.

But the CEO can't be expected to know as much as each VP, just like each VP doesn't know everything each manager knows, etc.

Also, when profits aren't made, it's not the CEO who suffers. They already got their salary. Its shareholders who suffer.

Sorry if it's not clear but my overall point is that accountability has to exist at all levels. The CEO isn't the position where all accountability emanates from or where it all stops. The CEO is held accountable to the board; VP's are held accountable to the CEO. The CEO is just one cog in the chain.

I agree everybody has some accountability, but the CEO's job is to be accountable for everything that happens in the company. It's part of his job to get the information from the VP, and if he can't do that then he can't do his job. He also has the ability to fire the VP, something the VP cannot do in return.

> Also, when profits aren't made, it's not the CEO who suffers. They already got their salary. Its shareholders who suffer.

The CEO shouldn't have a golden parachute. And honestly, if the CEO's job is only to make money, he should be paid directly proportionally to how much money he made for the company! This isn't like an engineer where it's difficult to measure their value. You can just look at the stock price (assuming the company is public, it's slightly fuzzier for private companies) and pay based on hitting predetermined metrics or proportional to stock price.

What I hear from these complaints is that no good CEO would agree to something like that. As far as I'm concerned this is a fundamental cultural problem similar to how the police reject body cams. If we pay them, they should agree to our terms. And if they don't, let's find replacements who actually want to work with us.

> Could SolarWind have been too difficult for the KGB to use them in an enablement operation? Yes, it is possible to achieve that level of security. Creating a strong fast detection capability with rapid remediation and incident response will make it hard for attackers to dwell for any length of time, or persist on the system after they gain access. It requires vigilance and some effort, but it can be done. Of course, SolarWind wasn’t close to reaching that level.

Who is responsible for vetting these partners? What’s the process look like? Surely it’s more than “trust us, we gotz great securties”.

"Everyone has been doing it this way, don't break stuff."

did we ever see post mortem from Solarwinds? How did attackers enter the network? Even if build server used admin:admin, this server was in the intranet. How did they get inside the network?

we need post mortem and to understand the entire attack chain, rather than sit and speculate about the abilities of KGB/SVR.

Truth is KGB/SVR employees are very dumb and routinely leave traces. Their best hackers are actually civilians with commercial interests who do black hat campaigns for them in exchange for cover/protection on russian soil, but are not officially employed/enlisted.

I have a hard time believing these were russian state sponsored hackers, unless somebody provides the hard evidence

> did we ever see post mortem from Solarwinds?

Other than to provide misinformation (to lead people astray) what is the advantage to solar winds to do a post mortem? Why educate people? Upside vs. downside?

A businesses purpose is to act like a business. Not to educate (for lack of a better way to put it) the 'peanut' gallery, pundits, news outlets, bloggers or to improve security for others. Or to seem like 'a good company'. Nobody will deal with or not deal with Solar Winds based on what they say afterwords in a public and open forum. Privately and maybe under NDA sure but why broadcast this to everyone? (Answer is not 'well that's what you do').

You are absolutely wrong on your approach to this. If there is no transparency into what happened, there is nothing but the companies word that it won't happen again. In the case of SolarWinds, their word means less than nothing.

When you have a breach of this magnitude, people need to understand how the attack happened and what technical or process controls you have put in place to prevent it going forward.

The biggest issue with SolarWinds breach is they have done nothing but try to obfuscate what happened. When they did press release, they said it was a "Security Vulnerability".

WRONG!!!! There was a backdoor intentionally placed in their product and sent out as update to tens of thousands of customers. At the very best their response has been uninformed, but knowing what we know about solarwinds as a company it seems intentional. To date they have not corrected that release and still often times refer to it as a vulnerability.

> If there is no transparency into what happened, there is nothing but the companies word that it won't happen again. In the case of SolarWinds, their word means less than nothing.

True, the silence is damaging. But what if the answers would be more damaging than the silence?

Agree but nobody wants to believe that. Since many of the people who see upside have a vested interest in that upside remaining even to the detriment of others.

Plenty of companies seem to think that seeming like "a good company" is a sufficient business reason to do a post-mortem. That's the thing about expectations: even if you can't come up with a rationale for doing something, other peoples' reactions to not doing the thing can be a sufficient justification for doing it.

> well that's what you do

I assume what this phrase is supposed to get at the sense that doing port mortems is the (morally) right thing to do and part of our duties as engineers to each other and to the public that has an interest in security. If that's the case then I have to disagree with you, that's an excellent reason to do it. (The fact that something would be the right thing to do means that you have a good reason to do the thing: namely, that it would be the right thing to do.) You can be cynical and say that "doing the right thing" is not going to be a good enough motivation to convince business X to do it, and that's fine, but it doesn't sound like that's what you're saying here.

> Plenty of companies seem to think that seeming like "a good company" is a sufficient business reason to do a post-mortem.

'good company' to what audience? Hackers and security people and 'nerds'. How big is that audience? You don't make business decisions based like that. It's business after all.

Look at all the (for lack of a better way to put it) 'dicked off' things Apple has done. They pissed off (and continue to piss off) very diverse groups of people and companies that interact with them. In the end the vast majority of the Apple customers (the people who pay them money) don't care about this at all.

Note I didn't say that things couldn't be shared with direct and important groups of people. Just that it didn't have to be open and public and broadcast in any way as entertainment or help for the community at large.

doing post mortem will be responsible thing to do as it will allow other companies to strenghten their defenses against this vector. if it was rogue employee who used his credentials - well that's one vector. if it was cheap CIO who kept security team understaffed and underpaid => underskilled -> well that's another lesson to learn for all other CIOs/CISOs around the world. Try to cut costs on your IT people, try to outsource talent - this is what you get. It will earn a goodwill for Solarwinds and will help everybody else to be aware of the attack vector

A post-mortem allows for the company to perform an analysis of what exactly went wrong and what needs to be fixed etc...

It also shows that they have a CSO role and they are trying to instill faith in their customers...

Simple: credibility. Right now, the default position is that they're insecure and they don't know what they're doing. A post-mortem with a solid RCA shifts that to "we've identified this, fixed it, and put in place systems to ensure it can't happen again."

It's the aviation industry playbook; air travel is perceived as safe (partially) because of the big song and dance they put on about safety analysis after an incident.

Not only was it a bad idea to throw some unnamed intern under the bus, it reveals how broken the culture is at SolarWinds that they would even think that this is relevant. It smacks of delegating blame to someone no longer their, and not accepting responsibility to a very bad series of decisions.

Grugq makes a few good points here:

Sophisticated hacks employ a kill chain - think of it as what aviation calls a "cascade of failures". There's no single cause for the awful outcome, but instead a series of events where intercepting any of them could have mitigated the crash or in this case, the hack. For example, sure they got in, but they also remained undetected. If they didn't get in or if they were detected, the whole thing may have been mitigated.

I also like how he's breaking away from just labeling the thing 'APT' and instead he describes the entity behind the attack, who they are, where they come from, what motivates them and how they are goal oriented rather than opportunistic. In other words, they didn't pick the target because of a weak password, they picked the target for strategic reasons.

And finally the point of how well resourced and experienced these operatives are - or to use his phrase, they're pretty fucking metal. To unpack this a bit, the operatives targeting these kinds of attacks are well funded, experienced, patient, persistent, have large teams and once they've picked you, it's really hard to consider the odds stacked in your favor unless you truly understand what you're up against and have prepared accordingly.

Reminds me of events like Pearl Harbor or even DDay where multiple signals existed to warn of the attacks and by some obnoxious sequence of ignorance they all went ignored.

That's perhaps a better analogy than my aviation one above. Exactly this. Break a link in the kill chain and it's prevented. So that's a slight advantage the defender has, but the trouble is most businesses (and individuals) have such a large attack surface that a determined attacker can usually find an alternative chain.

Thanks

Related ongoing thread:

SolarWinds CEO blames intern for password leak - https://news.ycombinator.com/item?id=26284782

> There is no rule that would prohibit the SolarWinds espionage campaign which the US would be willing to abide by itself.

Of course!

> misunderstanding of the hack in the public sphere

Hopefully that part is not misunderstood, right?

I accidentally clicked on a link in the article and the headline made me laugh.

> Former SolarWinds CEO blames intern for 'solarwinds123' password leak

> There is no rule that would prohibit the SolarWinds espionage campaign which the US would be willing to abide by itself.

Amen brother.

Thanks Reuters for propagating that bug bounty hunter's speculation. Cause this is where all of this goes back to.

tl;dr: thegrugq argues that it's possible for SolarWinds' security to have been inadequate, yet that security posture to have made no difference in whether they were hacked by the SVR.

thegrugq argues it's very likely:

> I’m perfectly willing to believe that their build servers were using “admin:admin” and that’s how the Russians gained access to inject their code… but, this was a clandestine intelligence operation. They did not succeed merely because SolarWind had poor password hygiene.

I have mixed feelings about this. I agree with him that if the SVR specifically targeted a particular organization with a specific goal in mind, it wouldn't really matter if they had weak passwords or not.

OTOH, I think it's not out of the question that one or more organizations (intelligence agencies, criminals, etc.) found the password and took advantage of it more as a "let's see where this thread leads" type of opportunistic attack, and all of the downstream consequences only happened because of that.

I've never worked for an intelligence agency, but I've been a professional penetration tester for about a decade, and when I go after an organization, that's typically my approach: find the weakest links and start following them to see where they go. In a complex environment, usually that leads to control over everything sooner or later.

Edit: just to clarify that last paragraph, what I'm getting at is that if I imagine myself in the shoes of an intelligence agency, the "organizations" I'd be going after would be foreign countries. I'm sure in some cases it would make sense to go after specific businesses of interest, but in the absence of legal restrictions, I'd be looking for the weakest links in entire industries that supported those countries in some way, not necessarily picking a specific business and targeting them.

There are quite a few companies out there that make systems monitoring and administration software that would provide similar levels of access to a wide range of organizations if their build chains were compromised. The one that was compromised was the one that also had a publicly-exposed update server with a password that could have been obtained in at least two different ways.[1] Coincidence? Perhaps, but I don't think it's fair to just take it off the table.

[1] Accidentally exposed in a public GitHub repo for many months, as well as being easily guessed. Either alone would have been enough. Both being true seems to me to make it more likely.

I think that's a good summary. Also, it's easy to imagine the SVR poking at all companies which sell high-trust applications[1] to many government agencies, and running with the ones[2] that worked.

Regarding whether the password thing was a coincidence, I wouldn't be surprised that, if other large enterprise software companies were severely hacked, similar stories surfaced. That doesn't mean it's a coincidence, of course, but may mean that this is average among enterprise software companies. One takeaway here is that software companies shipping trusted software to third-party networks have an exposure more like Google or the Federal Reserve, not like other software companies. That's not how (a lot of) the software industry has acted.

[1]: NMS, systems management, facilities management, possibly CRM

[2]: Where one was detected (because the attackers chose the wrong target in FireEye), others have probably occurred and/or are active now. While this was very sophisticated, it wasn't Stuxnet (https://blog.erratasec.com/2021/02/no-1000-engineers-were-no...).

Not quite. Rather, they argue that it's theoretically possible (though not necessarily realistic) to have good enough security to resist targeted hacking by the KGB, but this requires vastly more than just better password hygiene.

> it's theoretically possible (though not necessarily realistic) to have good enough security to resist targeted hacking by the KGB

It is possible to have a security level where the cost outweighs the benefit. If the KGB really wants to go all out, they could buy employees, burn zero days or even hold sysadmin families hostage - but that would be extremely expensive and risky and they'd really need a big reason to go that far. If your password is "admin" or "solarwinds123", on the other hand, the biggest expense is probably the employee time spent laughing in the coffee room.

I agree that it is vastly expensive to have a security level high enough that the KGB realistically can not take over your network, but it's far cheaper and more realistic to have a security level where it's not worth the expense.

> I agree that it is vastly expensive to have a security level high enough that the KGB realistically can not take over your network, but it's far cheaper and more realistic to have a security level where it's not worth the expense.

Yep. The former is possible in theory but not likely to happen. The latter is somewhat difficult, but ought to be table stakes for any company dealing with security-sensitive anything.

Nonsense argument. What’s the point of securing anything if THE KGB can cast a hack spell and get root on my server?

No, read the article. What can they do if someone from KGB will apply to a junior dev position and get the job? They will hand them all their keys.

And there are teams, a very few of them large, where internal segmentation is strong enough to survive a compromised developer machine. But that's an extraordinarily high bar to clear and very few companies, even those with strong security teams, really manage to clear it.

I’ve been hearing this argument for over 10 years now about nsa, idf, china and now the kgb. The truth is internal audits are quite effective at catching these. If nsa could just place a plant why did they spend so much effort to tap into companies’ inter-dc fibers?

Because mass surveillance is useful to identify targets and moles are useful to dig up the rest of the information on targets.

It's not an XOR.

There's a lot of baseless conjecture in this as far as I can tell.

>The SolarWind backdoor was deeply integrated into the code, it was injected during their build process, and there is no way that the server having a weak password was the pivotal factor. As if Russian Intelligence would just give up if there were a strong password instead!

If this was a wikipedia article there'd be a [citation needed] every other word.

I also wonder if they're not overplaying the skill of Russian cybersecurity agents. I'm sure they're good, I'm sure some of them are very, very good but the idea that basically they don't care about passwords is going too far IMO. The main advantage of being a state-sponsored hacker is that you have access to resources most other black hats couldn't dream of (like sending a team of burglars to ransack somebody's house, or physically threaten an employee) but that doesn't mean that they can stop obeying the laws of physics and algorithmics.

>There is practically no chance that the server’s password was in anyway relevant to the hack overall.

Source: my behind.

I think the author has a good point that it's probably best to have a holistic approach about these hacks instead of focusing exclusively on some details, but the details do matter. After all, the big picture is nothing but a long series of details, isn't it?

>Close does not count in security. In offensive security you’re either successful or not. When you’re dealing with access then the only possible states are: did it work? Yes or no. Whether you need 5 minutes or 5 weeks to get a shell, once you have that shell, it is the same level of game over. That’s what we’re talking about here. The technique used to gain access is a minor issue.

Reductio ad absurdum. So what does the author want us to do then? Set our passwords to qwerty1234 and just give up?

It's especially weird when the author a few paragraphs earlier states:

>Could SolarWind have been too difficult for the KGB to use them in an enablement operation? Yes, it is possible to achieve that level of security. Creating a strong fast detection capability with rapid remediation and incident response will make it hard for attackers to dwell for any length of time, or persist on the system after they gain access.

So it turns out that 5 minutes or 5 weeks does matter after all?

I find very little of substance in this entire rant. Also the KGB doesn't exist anymore, I don't know if the author doesn't know that or decides to keep using it for stylistic reasons but real life is not an 80s American B movie.

I call them the SVR, an agency formed from the FCD of the KGB. The first chief directorate was responsible for international espionage. The internal security responsibilities, along with everything else, was made into the FSB. There are some cross overs which reflect official Russian policy, such as the use of FSB officers to conduct espionage in the near abroad (such as Estonia) because the state believes they’re oblasts not sovereign nations.

That said, KGB is KGB. They call themselves Chekists, we call them KGB.

> I find very little of substance in this entire rant. Also the KGB doesn't exist anymore, I don't know if the author doesn't know that or decides to keep using it for stylistic reasons but real life is not an 80s American B movie.

I agree with you; not sure why the HN community finds this interesting.

Obviously the FSB now.

Social Engineering is a part of hacking and let's not underestimate the skills of spying agencies in this field.

Read what grugq says above.

It's not clear what you're really trying to argue here. Obviously, he's not suggesting we use "qwerty1234" as our passwords; it's you who's reducing arguments to absurdity here.

I'm pretty sure Grugq is aware of the current names for the Russian IC agencies.

I think your argument is way off base, and it seems to me that you are arguing with the one person that I would find reason to believe.

The article has good points about the nature of attacks by a determined adversary. The concept of "The One Critical TTP" that companies tout to divert blame and that observers use to justify why they are not vulnerable to the same thing is utter nonsense. If somebody shoots a bullet at a bulletproof vest and it goes through you should not conclude that they just so happened to hit the only weak spot and if you just fix it everything is okay. You should instead assume that the bulletproof vest might have lots of problems at least against the gun you were using to test it. Successful breaches and attacks do not show you where your only weak points are, they show you the level of quality your process provides. To actually fix the problems the process, not the product, needs to be improved so that it is able to deliver higher quality outcomes.

In the case of SolarWinds, we now know that the level of quality their process provides is insufficient to stop whoever attacked them. If we assume that it was a targeted attack by a nation-state actor, then we now know that they can not protect their customers against an actual adversary who had reason to attack them, willingness to attack them, and the ability to attack them. They are completely unable to defend against actual threats who will actually attack them. Lots of people will say: "Of course if a nation-state wants to attack me then there is nothing I can do, but why would they attack me?" Well, in this case, that is an actual threat. To provide an actual solution they do, in fact, actually need to be able to stop a nation-state.

So, how does SolarWinds fare against a nation-state? They are not even on the same continent. Everybody thinks it is completely and utterly laughable that they would have had any hope of stopping them. Not just that, it is a forgone conclusion that if a nation-state wants to attack any commercial system they can with utter ease. It is not even viewed as a possibility for any currently deployed system to stop any nation-state from getting what they want.

How far are these systems from stopping a nation-state? Well first we need to figure out what a nation-state can do. How valuable do you think the specs for the $1.5 trillion F-35 project would be to a peer adversary [1]? $100B? $10B? $1B? At the very least I would state that if a peer adversary could get the specs for the F-35 they would be willing to spend at least $1B on that project. So, to stop a nation-state you need a system that can protect against an attack funded to the $1B level. Assuming $500k/engineer-yr that is an attack with 2,000 engineer-years of development on it. There is no organization in the world who would even dare to claim that a team of 400 engineers working for 5 years could not completely and utterly compromise their systems. Even at 1/10th that nobody would dare to claim they could stop 40 engineers with 5 years. Even at 1/100th you would be hard pressed to actually find anybody who would claim they can stop 4 engineers for 5 years and you could probably count on one hand somebody who could actually deliver. The systems that are being deployed that are actually attacked by and must protect against nation-states need to improve by at least a factor of 100x before they can actually do their job. So, these systems are multiple orders of magnitudes away from achieving the minimum standard of functionality.

What can be done about this state of affairs? Either we must do 100x better than the best deployed systems, or if that can not be done, then we must assume that these systems can be 100% guaranteed compromised and act accordingly. Either we must disconnect these systems since they can not be defended, or the benefits of their use must be greater than the worst-case outcome of failure.

[1] https://www.idga.org/archived-content/news/pentagon-admits-f...

this guy makes a lot of unsubstantiated assumptions about how kgb works

Grugq is well known in the infosec industry and more experienced in the area than you realize. He's been writing about opsec for years, among other areas.

This is true. Grugq on a very short list of infosec people who have my implicit trust.

I would wager anyone with intimate knowledge of how the KGB works, and has the evidence to prove it probably isn't going to be writing publicly about it for very long...

Forensics and attribution are big parts of the infosec world. Researchers study attacks, tools, payloads, etc. and get an idea of different threat actors, their levels of sophistication, and who might have done which one. I’m sure it’s never ironclad - the conclusion that it was a particular intelligence agency is just an educated guess, and sophisticated attackers might intentionally ape the signatures of others. But it’s not completely hopeless.

Of course,

because hacking other countries in Russia is "legal" and they do it.

The trade off is that cannot really go on good vacations with all that stolen money, because they might have a unexpected visit

People focus on the password because it's the only part of the story they can relate to or understand. Orange County Rep. Katie Porter:

> "I've got a stronger password than 'solarwinds123' to stop my kids from watching too much YouTube on their iPad ... You and your company were supposed to be preventing the Russians from reading Defense Department emails!"

Words fail.

Is she that wrong? I don’t think so.

Do I think most private companies could defend against Double Dragon or Lazarus or Fancy Bear? No, if a state level adversary is attacking you and the payoff is that good, you are going to get popped.

But a strong posture makes it harder, which means they throw more at you and you have a chance of picking up on the attack. Best case, anyways. Worst case, you get to testify to Congress that your security measures were top notch and industry leading. That sounds a shit ton better than “we left a screen door open and didn’t notice for months.”

She's wrong to imply that if only SolarWinds had followed her iPad password policy, the attack would have been stopped. And she's mistaken about Orion's use case, which has nothing to do with email security.

And while Russia conducted this attack, I'm tired of the Russian scarecrow: SolarWinds' job here has nothing to do with Russia.

But mostly I'm jaded by ambitious SoCal pols neglecting their districts to score easy points on national issues.

> She's wrong to imply that if only SolarWinds had followed her iPad password policy, the attack would have been stopped.

I don't think she was implying that at all. She was highlighting that if they couldn't even do a basic thing like employing stronger, more complex passwords - how could they defend against Russians reading DoD emails.

> if a state level adversary is attacking you and the payoff is that good, you are going to get popped

So we should assume Windows, Linux, every CDN, every major firewall, switch and router, etc. are all owned by Russia?

And by China, and by the US and probably a bunch of other actors.

I mean, software is far too complicated in our current rube goldberg tower of abstractions, and the asymmetry favours the attacker (only have to be lucky once, etc).

Until a few generations have grown up with software, I'm not sure this is going to improve (although in that case, we've probably solved climate change, so that would be good).

Depends on how you want to slice that.

My laptop? My OpenBSD router? Very unlikely anyone has attacked it. I’ve had boring jobs and have boring interests.

Do I think the Russians, Iranians, or any major foreign adversary have a 0-day they could use against my systems if I suddenly got a top secret clearance and clocked in as more interesting? Absolutely.

I don't think this cliched "I'm not interesting" logic makes sense. At scale, a lot of "non-interesting" stuff becomes interesting. Or a way to find a needle in a haystack. Why wait until it's urgent to focus on a particular person? We all are aware that the US intelligence services operate this way, right? I can't think of a reason why others wouldn't.

The discussion above focuses on targeted operations by state intelligence. The CIA/FBI wouldn’t run around using 0-days on everyone’s box because the risk of discovery would be too high.

I do, however, agree with you in part: I’m sure that I have a lengthy profile built from passive monitoring. Heck, I’ve googled “tor project” so I know I’m in a database.

https://daserste.ndr.de/panorama/aktuell/NSA-targets-the-pri...

[I deleted everything I wrote]

Everything is about relationships. It makes no sense to "target" someone for being suspicious up front, because when they know they are interested in you, what they want to find out is who you interact with and how. So ideally, they (any data analyst) want everybody in their database. Then they do queries when they are looking for something.

And looking at what has been public, in the news, it seems like it isn't that unusual to break into and scarf up someone else's database in its entirety, without any fancy "0-day" exploits. Case in point, the US Office of Personnel Management had everything compromised, basically all the information the US government possessed about everyone with a security clearance. Probably it will never be publicized how many spies were lost, let alone other damage.

Is that even a question? There are many known markets for zero-day exploits against most OS's and software; that means whoever has the money has the ability to own whatever they want until it's detected and patched.

Reply on news.ycombinator.com