SolarWinds CEO blames intern for password leak
edition.cnn.com
edition.cnn.com
So let's analyze the various scenarios under which the "intern" might have been responsible, and why each one is bullshit.
1) The intern exposed a company password on their own account.
Counter: What kind of "password policies" allowed such a weak password in the first place?
2) The intern came up with the weak password themselves, thus violating "password policies" not just for secrecy but strength/security. This password was then used for several critical, production applications.
Counter: Why was an intern in charge of deciding a password used for anything critical?
3) The intern came up with the weak password and exposed it, but it was only used for the intern's own corporate accounts (e.g. their Windows workstation).
Counter: Why did an intern have a level of access such that their account being breached could lead to this level of compromise/exfiltration?
Conclusion: There is no conceivable scenario under which this makes sense.
I have an employee who I thought I could give more responsibilities, but he keeps not locking his computer when he walks away. He has very limited access to everything and it would impede his career if he didn’t also have the same attitude about other issues. (My question is, how do I make him diligent — It’s real potential wasted).
It might be that in your organisation this is a cargo-cult security practice that he's not bothered by because he knows it's not an effective practice.
Or it could be that he knows he doesn't have enough permissions to do any damage, so he doesn't bother locking his computer. Trusting him with a little responsibility might change that.
I wish I had an answer for the second part; it's hard to see someone with talent be the one to get laid off after months of asking them to be more diligent in their work.
Oof, this hits close to home. I don't have an answer either.
Edit: Also singing over the top praises of employees using the victim's account. "MonkeyButton is truly the best coworker I have ever had "
All this fun has gone away now with Covid and remote working.
We had a similar thing at one gamedev place that I worked at where and email would go out to the team if you left your computer unlocked(I forget the exact phrase but it was fairly silly).
We had shared offices and one of the programmers had the office right next to the kitchen. One day we all heard the senior programmer shout "WHAT THE FUCK!" and all ran over to see what had happened.
It turns out one of our engineers had walked into the kitchen and left his computer unlocked. The senior developer seeing this had opened up outlook, started a new message and began typing in the subject. What he didn't know is the developer had hand-rolled a keylogger with a match pattern for the message that everyone would send and dispatched Windows+L via key injection to the main window loop.
The trap was sprung and the machine locked right in front of him as he typed the last letter unable to send the email.
There was all sorts of other shenanigans at that place(like a fake "April 2nd" firing, they got the person who did that back with an annoy-a-tron over a 6 month period) but that was one of the more memorable ones.
If nothing else at least it promotes awareness (and cake!).
Physical access is considered game over, no?
For example, posted yesterday: https://tailscale.com/blog/rotate-ssh-keys/ (https://news.ycombinator.com/item?id=26249380)
> One company had an employee they fired for bad behaviour. A few months later all their production servers got wiped out. ... A month later, it happened again! Everything gone.
> What happened was that the rogue employee, months before they were fired, had waited until everyone went to lunch, walked around from one PC to the next, and collected the ssh private keys from any of them that weren’t locked. ... Then, a while after getting fired, he used a co-worker’s ssh key to login and destroy everything.
Of course, screen locking would not prevent a determined, disgruntled (or desperate) employee from pulling off something like this, but it would stop the casual collection of secrets while everyone is at lunch. Add full disk encryption to PCs and you have a reasonable chance at defending.
Oh, and if you're doing anything with credit cards, PCI DSS requires you to enforce a workstation locking policy. Failing to do so opens you up to massive liability in the event of a breach.
I think that there's sometimes a tendency to overthink management topics - there's no need to overthink this one.
It's reasonably believable that they leaked their password because. Interns.
And the second factor?
1) access restrictions such that even malicious interns, and certainly careless interns can do little damage *when* the inevitable leak happens
2) Actively scan everyone's online presence, and let them know that this is a requirement of employment.
3) Require 2FA
4) Much better training so it is reduced
5) Internally firewalled and airgapped systems
I could go on... but none of these were done
The fact that they blame the intern shows that they are insanely unqualified for any job related to any sort of security. These CxOs are active hazards in the industry.
this should be something that is implemented in any organisation beyond a very small scale, mainly because even if not malicious, people should not be able to make critical mistakes in systems they have no know how off.
The intern leaked the password, but how was he able to know this was critical information? Not to mention he should never have been put in that position in the first place.
One morning when I came in and sat down at my desk, all of the old-timers were having coffee and discussing the fiasco. I was very happy to hear all of them talk about how mistakes happen, and the last person to be blamed for such an outage is the poor guy or gal that hit the ENTER button. Rather, blame falls (to various degrees) on: the engineers in their orbit who should be backing them up; the managers helping to onboard them; the chain of command; the entire system that is in place to prevent inappropriate access.
One of my best early-in-career lessons was that it takes maturity to own up to your mistakes (no matter how bone-headed), and it also takes good managers and a good company to foster an environment in which you can own up to them without fear of losing your job. Any company that wants to hang a weight around an intern's neck for something like this is not a company I would want to support in any way.
[0] https://netflixtechblog.com/a-closer-look-at-the-christmas-e...
owning up to your mistakes also gives you an incredible amount of credibility and respect in my opinion. Mistakes happen, especially in complex systems. Owning up to mistakes and explaining your reasoning about your actions makes you and your compatriots better engineers.
Excluding malicious action, most people make a (semi) critical error sometime in their career, especially if you work on the ops side of things, these can often be disasterous. Engineering who claim they never have made a mistake that usually either not working on anything that has value or are just lucky in my opinion. engineering who are afraid to say they made a mistake are a cultural issue aswell, because it delays troubleshooting during incidents.
Something we tell employees during our onboarding in a technical comes down to this.
- reason about a problem by yourself first - think about impact before you do a change, if in doubt, ask and doublecheck. - admit mistakes when you realise them, explain your reasoning and why you did the action. - learn from your mistakes, but accept that being error-free is simply not possible.
Third option: they don't realise when they make a mistake, either because they are not smart enough or too full of themselves.
The only way to guaranteed never have mistakes in your code, is to not have any code at all.
If you are in the right company. I made a big mistake at one point. We had all of the people responsible for one of the payment methods away on holiday (this one payment method was managed by another team in another country, as it wasn't using our normal payment gateway and provider that my team was maintaining).
Huge panic, someone needs to fix this, you can't use X any more for completing your order. I'm in the right team that's handling payments and fulfilment, only one at work at that point so I'm told to fix it. I do, I fix it, send it over to testing, get the green light, fix is deployed, everyone is happy.
2 hours later, we figure out the payments are working, but the orders are not being finalised and are still in unpaid. We realised that that single payment method done by this other team in another country was not using our standard payment processing workflow and it has a different way of actually getting the confirmation from the payment provider. This was quite a big company, we had around tens of thousands of Euros blocked in those 2 hours. I own up to it, I admin I made a mistake, go deeper (we did not have anything about this documented) fix it again, we unblock everything, all good. Until 1 month later I got fired (there were layoffs because of larger financial issues, but I was on the list because of the incident), it was the only time in my career this happened to me.
In the same time, someone else in my team made a mistake, hid it from management, even though we knew about it internally, fixed it and bragged about fixing the issue (no mention of him being the one who caused it) and got somehow (didn't even know we had such a thing) employee of the month and big praise in the next department meeting from management.
My lesson from this? Screw these companies and the people running them. I was asked to help, I did it, I made a mistake, fully aware of that, but then I'm the only one thrown under the bus for it.
To give you a counterexample.
The same company my prior example came from, also had some other "silly mistakes" made by an intern. He had to do inventory of a couple of old servers and remove hard disks from these servers. The servers where to be sold.
Sadly, no one told him we had additional servers in the back of the storage room which he forgot to check because they where not on the same pallet as the batch he was told to check.
Result, an couple of servers got sold with disks still in them. Luckely the company we sold to was friendly enough to give us a headsup about it and it resulted in no further issues, but still. Our company director personally took this as a reason to spearhead a plan about improving operational security and change processes (Aka, remove the hard disks when the machines are put out of service instead of half a decade later when their sold).
The intern felt pretty bummed and thought he was responsible for the mistake, but in my opinion he done the job that was asked of him, he just got incomplete instructions. This was also explicitly communicated with him by his direct supervisor.
In my experience, not throwing people under the bus to hide organizational or process failure, but simply admitting the processes could be better and striving for improvement does absolute wonders for morale and team building.
Being perfect is impossible, organizations should keep people to impossible standards, especially to hide incompetence.
Bingo. I’ve made mistakes that have taken down systems or caused them to silently fail. The worst mistake I’ve made took down basically my entire company for about 20 minutes. This turned out not to be critical, because our site was still operating, and it was just external data feeds that weren’t getting updated, but I freaked out about it for a minute. After that minute, I went and got help, and we fixed it. Had I not, I probably could have fixed it myself, but it would have taken much longer and cost much more than it did.
If you’re in an environment that doesn’t recognize that, you aren’t in a place that actually values and understands engineering work.
Any customer that can see through the BS will immediately turn around on their heels. And SolarWinds will be happy that they just lost that problem customer.
SolarWinds is looking for an Equifax not a Netflix.
Every individual contributor in that company has just learned that they need to cover their ass for any action that could possibly go wrong.
The cultural outcome is that accountabilities will be spread across managers so that blame can’t be assigned to an individual.
That outage is what drove us to rearchitect all of Netflix to be multi-region.
If there were any singular behavior I wish every tech company would adopt, it is that level of perspective on what makes a system reliable.
At the time, I had several years in the industry under my belt but was new to big tech companies, and especially the tech that Amazon had built. I lacked confidence in myself but I was super conscious of actively seeking out the help of others. And it failed spectacularly - one SDE3 in my team with whom I was sort of paired would quite literally back away slowly when I was showing him an issue I needed his advice on.
Big companies like that have such variation in teams, and I think I just got rather unlucky in my experience.
If a mistake happens, don't blame the individual, blame the process then find a way to fix that process. If a company has a blame culture people spend more time covering their own arse instead of building safer processes.
Imagine the size of a rainbow table of SHA512 or SHA3-512 hashes for a 60 character password. A 60 character password could be as simple as:
* I enjoy driving my tiny white car with a standard transmission.
* My big cat, Ace, really sleeps a lot during the day while I work!
* Growing up my favorite song was Time by Pink Floyd about regret :(
For the few that you need to manually enter, something long is good, but it should ideally use characters or different classes, and ideally not be comprised solely of dictionary words (which your first example is), otherwise the search space is greatly reduced.
Also, manually entering a 60 character password is not going to be fun :) I think the longest "manual enter" password I have is 25 characters, and it's a PITA to enter in a password field!
You only need that to impose a greater character width against brute force attacks. That is the only value in high entropy.
The actual reason people think they need this is because it was written into a NIST publication a very long time ago and it just became common practice. As a proof of concept what is the published standard that imposes that practice? I bet you think you need this but cannot find the written standard guidance suggesting it.
The guy who originally wrote that standard later came out and said it was a mistake. Bad advise that he wishes he could take back, but it’s too late everybody thinks they need it and they don’t know why or where that guidance even comes from.
> and it's a PITA to enter in a password field!
Only on a touch screen.
My first example also contains uppercase and punctuation. Think of it like IPv6. When the key space is large enough you don’t need a bunch of bullshit and gimmicks to ensure uniqueness.
That's exactly the reason I meant - to make brute forcing take much longer.
> Only on a touch screen
Yep, certainly worse on a touch screen, but I hate entering long passwords even on a keyboard - it's OK if it's a field that let's you see what you type, but if it's just asterisks, then I find it hard not seeing that visual feedback.
"you can delegate authority, but you cannot delegate responsibility." [1]
This CEO is a very poor leader.
1. https://www.theleadermaker.com/you-cant-delegate-responsibil...
I've worker with interns whose major redeeming quality was that their internship was fixed length and would be over soon. I've also worker with interns who demonstrated ability and responsibility sufficient to get the same access that I had (and, clearly, an offer letter). That it was an intern doesn't mean the level of access was inappropriate; of course, if it were my intern, I would take blame for them leaving their password on github.
1) Every password policy allows for dumb passwords in certain places. Because password policies are only enforced on systems that integrate with the password policy enforcement mechanism. Which never covers everything. Even with a password policy, it’s easy to make dumb passwords.
2) It doesn’t say that the intern chose this policy or that it belonged to something critical. There has been no link established between that password and the breach. A random researcher said they found the password a year ago and reported it. It could have been used, but there’s no reason to believe it’s relevant.
3) Nothing suggests that they did.
Conclusion: It’s better to not be an armchair quarterback after a breach, especially when it’s still under active investigation by actual professionals with access to actual data, and they aren’t even making the claims that folks here are making.
Failure to realize this is shameful.
More recently, our politicians simply “don’t recall”, or worse directly lie to us.
For C-level folks, its simpler. Take no responsibility ever, unless forced to by the courts. Even then, taking actual responsibility is so rare that I have no examples.
That this is common in most organizations shows that most larger organizations end up being selection systems for filtering sociopathic to the top.
It is not because their are deliberately designed this way, but because this is what sociopaths seek, and the organization fails to actively filter against it.
The sociopaths are the ones designing the organization and creating the legal and bureaucratic frameworks ostensibly meant to filter them out. That the end result nurtures and rewards them and allows them to use their subordinates as a bullet sponge seems entirely deliberate.
The management team should be the first to be blamed when such incidents arise.
Throwing an intern under the bus is silly and disgusting.
How do we solve this though? Who is the CEO responsible to - who do they care, if anyone, about how they appear to? How do we shift this behaviour for the examples of CEOs and politicians who maybe haven't been outed yet by such a security breach but are vulnerable to such lacking of a rigid structure of command/responsibility?
Edit to add: I'm beginning to think HN is full of really lazy people.
Because of downvotes? That's probably just because you brought partisan politics into the discussion.
Wut? Mentioning a politician’s name isn’t partisan politics. He said nothing about the politics of Cruz, he compared the Senator shifting blame onto his kids with this CEO blaming an intern. Don’t be like that.
(Edited to correct typo)
A much simpler explanation is that people who read and comment on HN do not want to see every conversation degenerate into yet another political fistfight, since it seems to have infected every other part of our lives. One of the core values of this community is that we mostly manage to avoid that.
It's interesting how I can reference, with no personal opinion attached to it, a factual, recent situation that's comparable to the thread - about a well-known person who happens to be a politician - happens to be a politician, and as you said, because you're exhausted of hearing about them, you'd prefer they be censored/avoided in conversation. It's interesting to say the least.
P.S. Triggered is a valid word, I used it in the correct context - and you were in fact triggered by using it and are trying to gate keep its use.
Of course it is, why would you think otherwise? By calling out a particular politician by name, especially a controversial one, OP introduced a political slant to the discussion. Could just as easily have said "this is a problem we commonly see with politicians, or other people in leadership positions" without calling a particular one out.
> Don’t be like that.
Like what, exactly? OP made a comment, and when it started to go grey he maligned the HN commentariat as a whole as being full of lazy people. All I did was help him understand what aspect of his comment was the likely trigger for the downvotes. HN by and large does not like political discussions, it's even right there in the moderation rules.
> Ted Cruz claimed it was his children's fault for pressuring him into their trip...
I'll bite. (Stick with me; I'll get to the point after some analysis that may be objectionable or seem unrelated.)
He's got responsibility to two parties: the citizens of his state and his children. Both are important in different ways, and the needs of each conflict in certain circumstances.
Did he want to get somewhere warm? Probably.
Did his children want their father with them? Probably.
Was he unable to advise while in Cancun? Doubtful.
Was he unable to legislate while in Cancun. He was not.
So, despite the inappropriate optics of his trip (and reflection of poor character for a leader of the state), it appears as though he made a judgement that favored his family and his self, presumably because he judged no apparent harm to the state by his actions.
I bring all this up because it's relevant to the parallel you bring: he blames his children for the choice. The problem is, neither you nor anyone else I'm aware of have demonstrated harm by his choice. So his "blaming" his children appears to reflect his choice between two conflicting priorities.
The intern, on the other hand, caused demonstrable harm and the CEO blames his own decisions on said intern.
Cruz did not blame lack of preparedness on his children. He blamed bad optics on his choice to acquiesce to his children. Far as I can tell, you're bringing a straw man to a knife fight.
I don't think your assertion is incorrect, that incompetence and lack of integrity are responsible for the CEO's choice, but I don't think you demonstrated that properly with your parallel. I find Ted Cruz to be pretty unpalatable, but I believe the hype around his trip is exaggerated, and your use of it to affirm your assertion is misplaced.
However, in an effort to make this more interesting, let's say the parallel is appropriate. I would, for the sake of argument, challenge that the fault, instead, lies in the hands of the people of Texas. If "the buck stops here", then let us remember that there is an entity above that of Ted Cruz, and that is the people of Texas.
If we truly believe that mistakes are not the fault of the individual who made the mistake, but rather the person at the top of the chain, then it stands to reason that politicians are not at fault, but rather the citizens that voted that politician in.
Since voters have ultimate responsibility to decide the way in which their community is governed—just as a CEO makes choices that bubble down—then it seems to me the people are ultimately responsible.
And in this context, your parallel seems even more misplaced, because the people of Texas (and the rest of the country) are blaming the "little guy" when those in charge of actually deciding who's hired and who's fired are the responsible parties.
Personally, I find the ultimate lack of integrity in "the people", because everyone blames "government" when "government" goes wrong, but fail to acknowledge that they are, in fact, said government.
I charge you to challenge my assertion and justify your parallel.
Great! where do we sign up to vote on legislation?
I'd like to clarify one thing that I think was nebulous in my post: while I think "the people" are the responsible party, I meant to suggest that choosing who is the responsible party at all is close to arbitrary.
That is to say, arguments over who's held responsible more reflects who we want to be responsible than who may actually be responsible, which is in fact shared among several people. (The CEO, the intern, the person who gave the intern the password, the person who configured the system to require a password that could be leaked, etc.)
We choose the CEO (or Ted Cruz, or whomever) because of our own biases, in spite of the fact that responsibility is shared, and in spite of the fact that who's "in charge" is nebulous at best.
We do the same when blaming the 12 US billionaires (or however many there are) for income inequality in the country, or blame all white people for racial injustice, and so on.
That doesn't mean there aren't responsible parties, or that people and groups should be free from criticism. The truth of the matter is that—no matter how justified it is to assign blame—it's exceedingly rare for anyone to take responsibility for their own failings.
This shouldn't necessarily contort to whom and how we assign blame. But, rather, it should inform us of the breadth of the problem, our own contributions to it, and help us become better citizens, instead of shift the blame as the CEO or Cruz did.
If we find that unpalatable, but don't acknowledge our own complicity, then all we've achieved is a pitchfork-laden witch hunt, and not actual direction towards resolution.
He chose 1.
1) He's a pushover and his children, that we can assume he loves, he doesn't have control or authority over them for whatever reason; and in fact they'd have control over him.
2) He's using his children as an excuse to avoid taking accountability, and the problem down the whole line is a lack of accountability.
I agree the necessity of being in-person is debatable, however regardless, the lack of common sense to understand how it would be easily perceived is quite astonishing.
You're blaming the intern here - in fact it was the policies and protocol that allowed that whole shenanigans to unfold how it did: it shouldn't have been the interns responsibility to begin with - protocol should have prevented it, protocol and process that in a chain of command/accountability should reach the CEO if the protocol is adequate.
Of course the hype about his trip is exaggerated, that's to be expected - perhaps we've all been attempted to be gaslighted by Trump for so long that this doesn't seem like a big deal, voted in representatives not being actively engaged in military fashion to protect and help "his" people. We're so used to unqualified, inadequate, incompetent politicians being voted into power - that this seems okay - how many people died so far, how much damage was caused leading to economic damage and reducing productivity of those harmed - especially if they have to fight insurance companies for damage; it speaks to Eric Weinstein's commentary that it's only during wartime that the true leaders appear out of necessity.
Indeed, the people of Texas have been manipulated, arguably controlled subtly for many decades, directed by the duopoly - crafted heavily by industrial complexes - to put relatively bad people into the spotlight, for the mainstream media complex, controlled by a handful of conglomerates, to then amplify the two main narratives of either side of the duopoly. So is it really the people's fault? Yes and no, of course with the right policy proposals all of this can be countered strongly, and the change will cascade quickly, exponentially - policy proposals such as Andrew Yang's core policies; it's all the yin-yang dance of the universe, people need to suffer to learn - to pay attention, and hopefully we're at a part of a cycle where healing and creation is rapidly approaching, hopefully this isn't just a foreshock - where Trump's wave is still building to a tsunami - and we're yet awaiting the major earthquake before everything destabilizes and perhaps we're going to lead into a civil war - and perhaps even global war will start in parallel, if democracies can't be stabilized enough to create multi-lateral trade agreements to economically funnel resources away from the known bad actors who aren't behaving or falling in line to peace and freedom.
It's complex, there are systems that most don't know or understand or consider - and these systems self-perpetuate like a virus to keep counter attacks at bay; the autonomous immune system of America and the world is failing, inadequate. Luckily the purpose and value of distributing power, control - separating Federal government from the States - and further diversifying risk via having many States with their own varying rules - allows neighbouring States to shine light/the truth onto the States that have been darkened, influenced, lead too far astray by bad actors (domestic and foreign).
It's a lack of integrity in the systems - these flows that lead to multi-generational trauma, karma - action, consequences - passed down from one generation to the next. It's why my focus for all my projects is health: there are still ripples passed through stress-trauma of parents of war onto their children, and onto their children, and so on. We are generating new active trauma while not doing enough to support new people born or migrating to our more/relatively stable countries - and not enough to heal the existing population.
Also, I'm not sure I understand the parallel question. I do however think it's an elected person's responsibility to counter any claims - arguably by showing he was doing everything in his power he could help the people, and that there was nothing more he could have been doing + with proof to back it up; let alone traveling during a pandemic, where most people are prevented from traveling - that alone is terrible role modelling vs. AOC doing a social media effort to raise $5 million for the people of Texas. The issue is really the contrast is sickening - from the weak inadequacy of doing what's ideal and right vs. not - we need to purge the system ideally through non-violent means, America needs to throw up - clear all of the shit - the whole "clear the swamp" idea, the mantra that Trump misappropriated to rally and align with mainstream media narratives of demonizing the unknown - when in fact it's the devil himself who's managing and rallying the swamp; allowing those not blinded by or manipulated, indoctrinated into their control and lack of critical thinking and reasoning, and skills for integrity.
Looking at your other posts, it also seems that you easily show a pessimistic attitude, similar to how old people will complain about younger generations. No one responds positively to such generalizations.
Mentioning a political figure doesn't necessitate baiting one into a conversation - however yes, it's clear that people are tired, exhausted - and no longer actually read to comprehend, stunting critical thinking throughout what they're reading - and then making an assumption as to what they're reading, and then doing a lazy downvote for a dopamine hit to feel fulfilled - as if they were diligent, reaffirming their assumptions.
And I disagree that "no one responds positively to such generalizations" - it doesn't make them untrue, and usually I also reference the solution, policy proposals, to solve the problems.
When will IT companies learn not to use telnet, ftp, and easy to guess passwords on their intranet.
It is often the case after these breaches that HNers start assigning blame in ways that indicate their lack of practical experience in the security world. After 20 years working in security, from startups to fortune 500s, including the most well funded security teams in the world, EVERYONE has stupid password problems in some corners of their company. The password leak was absolutely not the CEOs mistake.
"The final component in our GAAP figure – that ugly $11 billion write-down – is almost entirely the quantification of a mistake I made in 2016. That year, Berkshire purchased Precision Castparts (“PCC”), and I paid too much for the company. No one misled me in any way – I was simply too optimistic about PCC’s normalized profit potential. Last year, my miscalculation was laid bare by adverse developments throughout the aerospace industry, PCC’s most important source of customers." . . . "I was wrong, however, in judging the average amount of future earnings and, consequently, wrong in my calculation of the proper price to pay for the business. PCC is far from my first error of that sort. But it’s a big one."
WOW. I rarely see that level of accountability at senior organizational levels.
SolarWinds: “the intern revealed our ‘solarwinds123’ password.”
As a CEO you've got nobody to blame after your first 90 days of employment.
In small companies I can understand that. But I've had it also happen in government jobs, in big companies and startups that collect a lot of sensitive personal data.
In one case I was with a startup that had names, addresses, DOBs, phone numbers and debit/card details in their DB. When they hired me they didn't ask for references or ID. I was given full admin rights to their Azure account on day one.
If I were in charge I would at the very least want to validate a new employees ID before giving them any form of access to IT systems, and only elevate access privileges once they had an established track record within the company.
Critical Password Policy
Rule 1: All critical passwords must contain the company name, lower case, with no special characters or spaces.
Rule 2: All critical passwords must have between 3 and 1024 randomly selected sequential integers appended, each of no less than 1 and no greater than 3.
When you are creating your critical passwords, it is critical that you follow the rules in the exact sequence as stated above, and that you do not introduce any external sources of randomness or entropy.
Failing to follow this Critical Password Policy may result in your dismissal and later blame before a Congressional committee.
I’ll type these notes up in a PowerPoint and we can deliver a presentation to key stakeholders.
https://twitter.com/vinodsparrow/status/1338431183588188160
This tweet has a screenshot of an email from Vinoth Kumar (named in the CNN article) to SolarWinds saying:
Hi Team,
I have found a public Github repo which is leaking ftp credential belongs to SolarWinds.
Repo URL: https://github.com/
Downloads Url: http://downloads.solarwinds.com
FTP Url: ftp://solarwinds.upload.akamai.com
Username:
Password:
POC: http://downloads.solarwinds.com/test.txt
I was able to upload a test POC.
Via this any hacker could upload malicious exe and update it with release SolarWinds product.
My thoughts:
(1) I assume this means when it comes to technical measures to prevent a weak password, SolarWinds would have to rely on Akamai.
(2) The researcher was able to upload to the root directory of downloads.solarwinds.com. As an educated guess, this may have been a shared account and many people knew this password. When many people share an account, they tend to choose passwords that are easy to convey to someone else. If so, the intern probably didn't create the password and was only responsible for leaking it.
Still there's some interesting things that could help a (much smaller, less critical) software vendor decide where to focus their security efforts. Perhaps near the top of the list should be: 1. Who in your organisation has access to your build and distribution toolchain, and how secure are their credentials? 2. How good is your record keeping? Are all your builds traceable back to a specific revision in your source control, and are you keeping build logs somewhere they can't be tampered with?
[1] https://www.bloomberg.com/opinion/articles/2021-02-26/deepen...
If your least experienced employee can accidentally topple the entire company, it is the entire company at fault. There are cases where individuals can undermine a whole organization and be at fault, but that requires sophistication and corruption which go above and beyond solid safeguards.
Turns out in an unfortunate and unforeseen turn of events, an intern wiped their production Hadoop cluster just a week later with the backups having some issues.
He was fired pretty quickly, but I heard he hasn‘t been too bad off since...
Don‘t hire any interns, they can do quite a lot of damage!
So in terms of specific people that's Bill Bock, the chairman of that board at the time.
They did indeed replace their CEO. I would assume after agreeing to pay off the old one as this is the usual practice and, unlike some no-name intern, a handsomely compensated CEO can afford expensive lawyers if you try to kick them out uncompensated for incompetence based merely on the evidence that their inadequate oversight cost you billions of dollars.
The last time an intern made such a disproportionate impact was in 1996.
Security breaches happen - fix up the issues and show people your reform. Blaming the intern just makes you look like an ass whos company I never want to do business with again.
Amazing.
Any company that continues to rely on Solarwinds after this, if the CEO is not fired, is accepting that their security is only as good as a SolarWinds intern.
[1] 'First-Order Measurement', Quality Software Management, Volume 2, Gerald Weinberg, Dorset House Publishing, 1993
Imagine a bank where the CEO says "the problem with all the money going missing was that an intern dropped the keys to the security vault and we had told the guards to never question anyone who had the keys, just let them take whatever they want without question". Seems to me that you can't pin anything on the intern. The problem was the extreme lack of security practices at the company, which ultimately falls on the CEO, who's trying to blame his own incompetence on a intern.
This means that the CEO, CTO, and all management down the chain failed to create any kind of robustly secure system.
Their entire system was never more than a single password leak away from complete failure for the security of the entire USG.
With literally thousands of opportunities for such leaks per day, it is inevitable.
So, a system must be designed such that when the inevitable occurs, the consequences are minimal. The most basic of fail-safe designs.
These people, from the CEO on down, utterly failed to do this.
Yet they made millions from false claims to have succeeded in creating a secure system, when they created a highway for global espionage. They literally could not have provided our adversaries a better avenue for espionage if they tried. The free world would literally be better off if all of that company and it's management had never existed.
Meanwhile, as a small manufacturer who has some govt-related work, the blizzard of new security certification requirements for Controlled Militarily Critical Technical Data and the like coming down the pike is like nothing I've ever seen before.
Sure, some of it is due to increased threats, but much of it is definitely because of a*$holes like those CxOs who so utterly and deliberately failed in their most basic duties, and should never work in the industry again.
The SolarWinds execs have provided us with keen insight for the root cause of the SolarWinds attack, but the insight they conveyed is probably not the insight they intended to convey.
Conversely, if I delivered something of noteworthy benefit for the project I will talk about it in terms of how 'we' achieved the success, and if it was someone else I will call them out by name and congratulate them on the solution.
This ethos is probably career limiting as a dev, but it does help me sleep soundly.
Rep. Katie Porter: "You and your company were supposed to be preventing the Russians from reading DoD emails!"
Microsoft President Brad Smith: "There is no indication, to my knowledge, that the DoD was attacked"
That's accountability! dilligence! Microsoft at its best.
SolarWinds CEO blames intern for password leak - https://news.ycombinator.com/item?id=26284782
Minimum number of days allowed between password changes.
Number of days warning given before a password expires.
Says that you have to use any password within 0+x days - otherwise it would have expired - the older the posting the more unlikely it would be valid. Would you risk detection with a password without mix case and consecutive numbers. Most would avoid traps or suspect a trap
It's a shame really, to blow a totally manageable crisis where solar winds could emerge stronger as a result.
Fwiw this is a good archive of mostly well done post mortems: https://github.com/danluu/post-mortems
Are we all this blind?
Let's even say the intern was malicious and trying to do harm... that is still your fault. One person in the lowest position in the company can break everything? Again, you are setup incorrectly.
Good times.
I wouldn't. I'd blame senior technical leadership for putting processes into place that allow failures like that. Most especially, the CEO.
Who would want to work for a company that throws them under the bus for the largest hack in recent history
"12345? That's amazing I've got the same combination on my luggage!"
The fact that he is blaming the intern shows that he knows his systems suck and is attempting to divert.
The entire management chain from CEO down to the intern's manager is at best too incompetent to work in the industry, and more accurately, a major threat to national security.
Anyone who trusts these people to do so much as run out for a pizza is a fool.
It is the fault of both of you. ¯\_(シ)_/¯
checks notes
... a single intern! then you're doing it wrong.
Whatever that means.
This would never have happened in the first place had they used an encrypted complex password and a simple password manager.
The whole company takes the hit with blunders like this. It's everyone's fault responsible for the infrastructure allowing this to happen in the first place. Those who pass the blame on others very quickly are equally to blame which means the CEO is just as to blame as the 'intern'.
Clearly the whole company doesn't train their interns.
The git repository just happened to be public.
It's entirely reasonable to think that the person in question possibly didn't even stop to think that Solarwinds123 was an actual secret that needed to be kept, as it is the equivalent of common passwords that are published publicly in manufacturer documentation.
I’d say in all the ways that matters this was basically everybody BUT the interns fault.