Results of technical investigations for Storm-0558 key acquisition
msrc.microsoft.com
msrc.microsoft.com
If I were an advanced persistent threat attacker working for China who had compromised Microsoft's internal network via employee credentials (and I'm not), the first thing I'd do is figure out where they keep the crash logs and quietly exfil them, alongside the debugging symbols.
Often, these are not stored securely enough relative to their actual value. Having spent some time at a FAANG, every single new hire, with the exception of those who have worked in finance or corporate regulation, assumes you can just glue crash data onto the bugtracker (that's what bugtrackers are for, tracking bugs, which includes reproducing them, right?). You have to detrain them of that and you have to have a vault for things like crashdumps that is so easy to use that people don't get lazy and start circumventing your protections because their job is to fix bugs and you've made their job harder.
With a compromised engineer's account, we can assume the attacker at least has access to the bugtracker and probably the ability to acquire or generate debug symbols for a binary. All that's left then is to wait for one engineer to get sloppy and paste a crashdump as an attachment on a bug, then slurp it before someone notices and deletes it (assuming they do; even at my big scary "We really care about user privacy" corp, individual engineers were loathe to make a bug harder to understand by stripping crashlogs off of it unless someone in security came in and whipped them. Proper internal opsec can really slow down development here).
Let's assume a crash dump can be megabytes up to gigabytes big.
How could a vault handle this securely?
the moment it is copied from the vault to the developer's computer, you introduce data remanence (undelete from file system).
keeping such coredump purely in RAM makes it accessible on a compromised developer machine (GNU Debugger), and if the developer machine crashes, its coredump contains/wraps the sensitive coredump.
A vault that doesn't allow direct/full coredump download, but allows queries (think "SQL queries against a vault REST API") could still be queried for e.g. "select * from coredump where string like '%secret_key%'".
So without more insight, a coredump vault sounds like security theater which tremendously makes it more difficult for intended purposes.
This attacker probably (it's unclear, since the write-up doesn't tell us) scanned compromised machines for key material using some kind of dragnet scanning tool. If the data wasn't on the compromised filesystem, they wouldn't have found it. Even though perhaps in theory they could have sat on a machine with debug access (depending on the nature of the compromise, this is even a stretch - reading another process's RAM usually requires much higher privilege than filesystem access) and obtained a core dump from RAM.
Security is always a tension between the theoretical and the practical and I think "putting crash dumps in an easy-to-use place that isn't a developer's Downloads folder" isn't a bad idea.
Tear down the environment after the debugging is done.
Keeping the crash dumps in a vault presumably allows more permission/control that an internal issue tracker (usually anyone can access the issue tracker). At least a vault can apply RBAC or even time based policies so these things aren't laying around forever.
Given that stack traces, crash addresses, and most register contents are considered to be security insensitive, most people don't really need access to the raw dumps.
It's far from perfect, but it would be unfair to call it "security theater". It seems like a pretty decent balance in practice. Admittedly, we have the slight advantage of several hundred millions installs, so the actual bugs that are causing crashes are likely to happen quite a few times and statistical analysis will often provide better clues than diving deep into an individual crash dump.
From my understanding, this is more or less how the Microsoft system was designed with credential scanning and redaction over coredumps, but a chain of bugs and negligence broke the veil.
> but presumably the attacker here had to know about the crash, and the layout of the crash dump
another statement from the article:
> Our credential scanning methods did not detect its presence (this issue has been corrected).
The article does not give any timeline when things happened.
Imagine the following timeline:
- hacker gets coredump in 2021, doesn't know that it contains valuable credentials.
- For data retention policy reasons, Microsoft deletes their copy of the coredump — but hacker just keeps it.
- Microsoft updates its credential scanning methods.
- Microsoft runs updated credential software over their reduced archive (retention policy) of coredumps. As that particular coredump doesn't exist anymore at Microsoft, they are not aware of the issue.
- hacker get scanner update.
- hacker runs updated credential scanner software over their archive of coredumps. Jackpot.
... but they're definitely not strictly necessary.
The article also says that Microsoft's credential scanning tools failed to find the key, and that issue has now been corrected. This makes me think that the key was detectable by scanning.
Overall, my reading of this is that the engineer moved the dump containing the key into their account at some point, and it just sat there for a time. At a later point, the attacker compromised the account and pulled all available files. They then scanned for keys (with better tooling than MS had; maybe it needed something more sophisticated than looking for BEGIN PRIVATE KEY), and hit the jackpot.
And how often do you hit the jackpot? For larger lotteries, it's less than once in a million. So that leads to two equally unpleasant alternatives:
1. The attacker was informed where to find the key.
2. The attackers have compromised a large part of Microsoft engineering and routinely scan all their files.
I think the most likely explanation is that this actor routinely attempts to compromise big-tech engineers using low-sophistication means, then grabs whatever they can get. Keep doing that often enough, for long enough, and you get something valuable -- that's the "persistent" in APT.
'After April 2021, when the key was leaked to the corporate environment in the crash dump, the Storm-0558 actor was able to successfully compromise a Microsoft engineer’s corporate account. This account had access to the debugging environment containing the crash dump which incorrectly contained the key.'
So either the attacker was already in the network and happened to find the dump while doing some kind of scanning that wasn't detected, or they knew to go after this specific person's account.
1: https://github.com/openssh/openssh-portable/blob/694150ad927...
Yes, although the article we're discussing shows that you can't rely on that, the dump could be subsequently moved to a developer machine for investigation, and unencrypted key material left in could be compromised that way... defense in depth would make sense here.
The redact_key_from_crash_dump function would be a good place to look for juicy bugs. Any case it misses tells you what to look for.
Think about it, first you need a race condition, and that race condition has to result in the unexpected result. That right there, assuming this code has been tested and is frequently used, is probably a less than 10% chance (if it was frequently happening someone would have noticed.) Then you need an engineer to decide they need this particular crash dump. Then you need your credential scanning software (which again, presumably usually catches stuff) to not be able to detect this particular credential. Now you need an account compromised to get network access and that user has access to this crash dump and the hacker happens to get to it and grabs it.
But even then, you should be safe because the key is old and is only good to get into consumer email accounts...except you have a bug that accepts the old key AND a bug that didn't reject this signing key for a token accessing corporate email accounts.
This is a really good system engineering lesson. Try all you want eventually enough small things will add up to cause a catastrophic result. The lesson is, to the extent you can, engineer things so when they blow-up the blast radius is limited.
Why isn't it masking before writing to disk ? God only knows.
Crash handlers don't know what state the system will be in when they're called. Will we be completely out of memory, so even malloc calls have started failing and no library is safe to call? Are we out of disk space, so we maybe can't write our logs out anyway? Is storage impaired, so we can write but only incredibly slowly? Is there something like a garbage collector that's trying to use 100% of every CPU? Are we crashing because of a fault in our logging system, which we're about to log to, giving us a crash in a crash? Does the system have an alarm or automated restart that won't fire until we exit, which our crash handler delays?
It's pretty common to keep it simple in the crash handler.
With a caveat that when it comes to security the eventual succession doesn't come as a random process but will be actively targeted and exploited. The attackers are not random processes flipping coins, rather they can flip a coin that often lands on "heads", in their favor.
The post-mortem results are presented as if events happened as a random set of unfortunate circumstances: the attacker just happened to work for Microsoft, there just happened to be a race condition, and then a crash randomly happened, and then the attacker just happened to find the crash dump somewhere. We should consider even starting with the initial "race condition" bug, that it might have been inserted deliberately. The crash could have been triggered deliberately. An attacker may have been expecting the crash dump to appear in a particular place to grab it. The attacker may have had accomplices.
I thought a good security rule was to reduce the attack surface. But ok, we are talking about ... Microsoft. /s
Public RCAs are nothing more than carefully curated PR stunts to calm customers. You can be sure the internal RCA is a lot more damning.
Does it say that?
> the Storm-0558 actor was able to successfully compromise a Microsoft engineer’s corporate account
The Microsoft ecosystem looks like a Lego car built by the neighborhoud kids, everybody bringing something from home and smashing it together.
Which should probably mean that half (or more) of the Western business world relying on Outlook.com is a very wrong thing to have in place, but as the current money incentives are not focused on resilience nor on stuff like breaking super-centralized Outlook.com-like entities down means that I'm pretty sure we'll continue having events like this one happening well into the future.
"reducing your blast radius" is never truly finished, so how do you know what is sufficient, or when the ROI on investing time/money is still positive?
* July 11 2023 this was caught, April 2021 it was suspected to have happened. So, 2+ years they had this credential, and 2 months from detection until disclosure.
* How many tokens were forged, how much did they access? I'm assuming bad if they didn't disclose.
* No timetable from once detected to fix implemented. Just "this issue has been corrected". Hope they implemented that quickly...
* They've fixed 4 direct problems, but obviously there's some systemic issues. What are they doing about those?
I would argue, it's time to decentralize inside a wider security perimeter.
... can't identify nation state actors within their own company.
I suppose that would be illegal. Whereas using it to improve AdSense CTR or selling it to brokers is perfectly acceptable.
The real advantage is that MS has to play at least somewhat inside the legal system.
A nation state has effectively unlimited budget in money, but more than that, their incentives are different. Any defender has to maintain an increasingly complex system with an evolving attack surface. A nation state attacker has to maintain ongoing access to any parts of that system. Access grants opportunities. They can wait, and can afford to do so. They have a massive time budget to tap. A company who does not prioritise or budget ongoing maintenance will eventually reassign their expensive resources to projects that do produce visible or at least measurable results. And in doing so, they neglect the unmeasureable outcomes from the prior projects that are now starved of proper resources. (Or even just attention.)
Compromises will happen. That's the ground truth. The important part is the blast radius.
In this particular case the impact was magnified by string of failures. Missing or ineffective revocation of a signing certificate was a big factor, but the failure was further compounded by the applicable scope of what that certificate could sign things for. Those two process failures caused this incident - everything else is attributable to bugs.
In short, MS dropped the ball on an organisational level.
Why do you think this is true?
Let's look at the other side of the equation and see what they are up against.
Now, obviously Mossad won't be spending all of their $2B on offensive stuff in this space, but on a ballpark estimate I'd say their spend on offensive cyber[tm] is between $300M and $500M.
From vulnerability equity programs we know that the going rate to acquire a reliable exploit against a high-value, hard target is between $1M and $3M. So we can safely infer that it would cost at least that much to develop one from scratch. Let's be charitable and say that on average it costs around $2M to develop a bespoke exploit against a hardened, niche target. These exploits also have their shelf-life and will eventually get burned. Again, we can be charitable and say that on average an exploit remains useful for maybe 18 months. Over that time the adversaries will either have developed their own parallel methods, or will be buying another one to replace their now-expired product.
That puts the expense floor, before operation staffing costs, to somewhere around $750k per year just to stay in place with regards to access technology. For high-value, bespoke targets where no exploits are readily available from the brokers, you can still expect to spend about $2M per year to develop and maintain a matching capability.
Then let's consider the operation personnel costs. A well run intelligence operation is probably not a sweatshop. With our venerable Stetson-Harrison estimation method we can put an average headcount per operation to 9 people. An operation lead, two analysts, three software engineers, one project manager, and two support staff. Let's say the fully loaded cost for lead and manager is $300k each, for software engineers $250k each, and for analysts/support $190k each.
So a single operation could expect to have annual, ongoing costs just a hair under $4M. Around half for maintaining the access technology and the rest to keep the operation going. Let's also say that the intelligence agency maintains a discretionary buffer budget to absorb occasional one-off cost runs, so that if a project for some reason generates an extra $1.5M charge in one year, it'll be expected slippage and already accounted for.
At $4M per year, per project, and a minimum of $300M to spend on such projects, you can maintain a lot of access operations. For just one intelligence agency.
When I said that nation state adversaries have effectively unlimited budgets, I meant that they can keep paying those sums just to maintain their access. For them, the ongoing access itself is an essential means to an end. They don't attack systems because they need to get through a system. They do that to achieve their objectives. Those are not "breach systems X and Y" - they are more along the lines of "collect and exfiltrate information of type M". How they do that is irrelevant. And as long as they can maintain their access to relevant source(s) of valuable enough intelligence data, they can keep paying for it, year after year.
Not just in money. They can keep the operation staffed. And they can afford to wait.
As a defender against such adversaries, you will ALWAYS be at a disadvantage. You need to keep a complex, ever evolving system secure and shut attackers out, plus you can not afford to make mistakes. Your adversaries only need you to make one. If not this year, then maybe one after the next. They can sustain their ongoing operations, while you, as a defender in a corporation subject to quarter-to-quarter thinking, have to keep justifying your work (and the ongoing expense) on systems that only show measurable results when they fail in their purpose.
So while my original statement may not be technically true, for all intents and purposes nation-state adversaries do have unlimited budgets. And no amount of expenditure will make you invulnerable.
> The biggest weakness and strength in security is loyalty and ego.
Nation states can do things that private corporations can't. Appealing to ego is a big one. National loyalty is another. That, in addition to blackmail, bribes, offering a save-haven, etc... are hard to compete with.
That being said, defense mechanisms can be build that are make insider compromise difficult to fight against. For example, HSMs are one key tool that make insider compromise much more difficult. I've worked at a non-zero number of companies that hired external firms (think the Xerox and ATTs of the world) to perform security critical activities, or audit employee requests -- that didn't work so well, but there are tools here. Most of the time they aren't just "Limit access to the people that need it" -- in fact, if you rely really heavily on ACLs, and not an inherit built-in security model to your system, I'd say the risk of something going wrong is much higher.
Would you be suspicious of a co-worker who recommended the adoption of the widely used log4j logging library? Of course you wouldn't!
> Microsoft provided an API to help validate the signatures cryptographically but did not update these libraries to perform this scope validation automatically
The phrasing as "some obscure bugs were carefully exploited" seems a bit off, it looks more like a comedy of errors where none of the security systems served its purpose at all.
You can't start out with something unconstrained and expect to patch all the holes in it. Mathematically speaking, you start with set A which is all possibilities and set B which is all the things you know to remove and you end up with A \ B not {}.
You have to start out with something constrained and allow only the good bits through the holes.
Having a similar discussion at the moment. Which is the correct solution:
a) Buy some software to redact all PII from logs
b) don't put it in there in the first place
Plus, they are the ones who put the security of their system behind this detection, letting developers access the dump they believed to be redacted. Whether they made a massive mistake at the design or implementation phase doesn't really absolve them.
Reading that MS were able to fix the crashing system’s race condition that included the key, it’s likely to have been a long-lived intermediate key for which the private key was held in memory (with a HSM backed root key for chain of trust validation, assuming MS aren’t completely stupid).
The challenge is the sheer scale these servers operate in terms of crypto-OPS… it would melt most dedicated HSMs.
[1] https://www.futurex.com/download/excrypt-ssp-enterprise-v-2-...
It's probably a better idea to pursue short lived private keys, rather than HSMs. If the timeline is accurate, the key was saved in a crash dump in 2021 and used for evil in 2023, monthly or quarterly rotation would have made the key useless in the two year period.
A certificate chain is a little too long to include in access tokens, IMHO, but I don't know how Microsoft's auth systems work.
The key expired in April 2021. Short lived keys only work if you actually check for expiry, which it appears they weren't doing.
That way, the HSM only needs to do one transaction per hour per auth server. If auth tokens need to be valid for 24 hours, then the certificates from the HSM need to be valid for about 25 hours (plus some leeway for refresh delays maybe).
If someone compromises the auth server and gets the private key (or gets in a position to request a cert from the HSM), then it is still quite bad in the sense that they have up to 25 hours to exploit it. But if this is only one of many controls, it still provides significant defence in depth, and cuts off certain types of attacks, especially for APTs who might not have any available TTPs to gain persistence in a highly secure auth server environment and who only briefly manage to gain access or get access to stale information as in this case.
Bingo, this is the most worrying bit to me. Pasting from a comment I posted earlier:
> Microsoft's own developers failed to implement a secure authentication check on top of their own libraries and infrastructure.
If Microsoft can't use its own identity platform correctly in a flagship Microsoft product (Outlook), what chance does anyone else stand?
This is something that's incredibly obvious in a code review.
If a Chinese fighter jet shot down a FedEx plane flying over the Pacific, that would be considered an attack on US sovereignty and the government would respond appropriately. Certainly we wouldn't expect FedEx to have to own their own private fleet of fighter jets to protect their transport planes. No one would be like, "Well it's FedEx's fault for not having the right anti-aircraft defenses."
But somehow, once it hits the digital domain we're just supposed to accept that Microsoft is required to defend themselves against China and Russia.
The real threat is a cascading power grid failure due to undersupply, e.g. coordinated forced plant shutdowns. A few days without electricity at a large scale means reduced availability of medical and emergency services, no running water, failing refrigeration, no stoves/ovens for cooking for most of the population, no working gas pumps, no electronic payment, no banking (no way to get cash) etc.
In a world where Stuxnet took out uranium centrifuges, and we've had actual PoC's of exploits that resulted in generators fragging themselves, I find your statement to be of the most shocking form of naivete I've heard in a while.
And in point of fact, the network connection would probably be for disabling alarms and control systems in order to mask work done to weaken the integrity of the structure itself. Physical and digital is inextricably linked.
It happened very fast, and was very unsubtle. Probably not enough time for site personnel to respond before damage was done.
Key: "It seems they were used to the high levels of vibration" - Diane Vaughan wrote an important book that introduced the term "normalisation of deviance" as a factor in the Challenger Launch Decision, a more famous complex accident.
Another way of putting it is that "all complex systems operate in a degraded mode all the time", paraphrasing Adrian Colyer: https://blog.acolyer.org/2016/02/10/how-complex-systems-fail
But if a bunch of Chinese people robbed a US bank, let's say the federal reserve, causing enormous financial damage but not loss of life, the response would be similar. Especially so if their link to the actual Chinese government was suspected couldn't reliably be proven.
Governments catch foreign agents somewhat regularly, and those captures don't lead to an all-out war.
Vs. any "cyber" crime? All that nice stuff about "...establish Justice, insure domestic Tranquility, provide for the common defence, promote the general Welfare..." falls on the floor, and...YOYO.
They kind of do, no?
Also, preventing crime and carrying out criminal justice are two very different things.
https://www.justice.gov/opa/pr/four-russian-government-emplo...
In fact, inadequately protecting their assets from mistakes or attacks can lead to SEC fines on top of losses,
https://www.sec.gov/news/press-release/2013-222
If there is an attack in progress, the police will intervene, of course. But if it leads to a financial institution collapsing because all their money was in the one place and they weren't insured, then that's the fault of the bank.
Secondly, the US does this all the time, even to friendly countries, so it's hard to justify harsher measures.
Value was destroyed in both cases. Users having their private data stolen have been harmed, the company's brand value is harmed, and they may lose users over this.
> or lives lost
Lives can be lost and real people can be harmed if their private information is stolen and used against them. There are dissidents and journalists in repressive countries whose safety depends on information security.
https://en.m.wikipedia.org/wiki/Malaysia_Airlines_Flight_17
As far as I know Malaysia is not at war with Russia.
Espionage is a dirty game.
What I always find interesting is how the US has taken on a strategy of indicting individual Chinese/Russian hackers for acting in the interests of their countries, whenever they can be identified by DoJ.
This policy is interesting, because, as we all know, turnabout is fair play.
How long before retired NSA operators are advised to never travel outside the US lest they be at risk of being picked up on international arrest warrants from China?
And I presume there has been no known dump of e-mails exfiltrated during this attack?
That's my understanding.
Those email accounts could have had multiple authentication factors enabled, other conditional access policies applied (geo-location, device trust, time of day etc)… all of which were skipped over.
I will say, I’m not very satisfied with just “improved security tooling.” My gut is telling me that there is a better solution out there to guarding against credential leakage, but I feel wrangling memory dumps to have “expected” data is a fools errand.
- Not storing key data in an HSM.
- Exporting crashdumps outside of the production account.
Redacting private keys and other sensitive data from these will always be failure-prone. Keys are also only just one problem, there will be personal data, passwords etc. in those dumps depending on what process it was.
- Corp environment infiltration not detected at the time (presumably, since this part is pure guesswork)
- Not enough log retention in the corp environment to track a 2 year old infiltration.
- Not assuring that key validation correctly denied requests with the wrong scope.
Fails-with-a-valid-key-with-wrong-(scope/date/subject etc.) are the kind of cases that always deserve test coverage, especially for a dedicated key-validation endpoint. Also concerning that this wasn't found by manual-testing/red-teaming/pen-testing in the time since it shipped.
- Slow detection, slow response, poor communication
The lack of scope checking seems especially egregious. It sounds like any number of keys would have been incorrectly trusted. If it was an RSA key that signed JWTs which it sounds like or similar, Microsoft has an issuer endpoint for all customers and it's critical to check the issuer/scope for those since any number of things can create a token with a valid signature.
>Not enough log retention in the corp environment to track a 2 year old infiltration.
The two are in conflict. Redaction is a problem for logs as well. A longer retention period implies a more damaging vulnerability.
It's very unlikely anyway a software company will fix an old crashdump - the software probably moved on a lot. So if there's no specific report/problem it's attached to and it's not a sensitive area, we're better off having it deleted. Same thing for old logs.
It didn't say that Microsoft couldn't identify that infiltration had occurred just that they didn't retain the logs to prove to exfiltration. That makes a lot of sense, maintaining access logs is one thing but to retain the detailed logging of every file action by every user on a 100k+ user corporate network long-term would be a massive amount of storage, of fairly limited value.
Even in this case, it might be nice to have but it wouldn't change any of the major findings you care about if you are Microsoft: that a bug allowed a key to be written to a dump file, that the scanning tools didn't detect the key in the dump file, and that the authentication process didn't properly check the keys.
Common guys. This requires at least one 3rd party investigation to be credible.
It is possible the crash that contained the race condition was planned. But how would anyone know the race condition existed? We know now so it is likely, based upon the successful attack, it was known prior to the key being exfiltrated. How many of our organizations have the same race condition and our keys are/have been vulnerable? No one else has keys that are valid for more than a year, right?
Let's say the race condition was unknown to the attacker. How can the attacker find the key in the dump? Did they have a scanning tool looking for key data? If so, how long was that running on MS network before it found the keys in the dump file? How many of our organizations' keys are in dump files? Are they all expired?
That’s something I would fix
Then, if deletion is required, simply erase the row that holds the mapping.
And finally, be sure to not store that mapping table in the same place as your backups or your logs.
Also when you don’t know how a Chinese threat group got into your network that’s a major issue which will cost more than theoretical legal risk
These days I wouldn't be surprised if they are running 100GigE or 400GigE networks just for managing logs throughput at aggregation points.
I assume that’s a way smaller scale. However the document doesn’t go into detail which kind of logs exactly they were missing, so maybe these were network logs
Text compresses extremely well and long-term archive space is generally inexpensive.
The hard part is all the work of deciding what is important, what the processes will be, and implementing a proper archive system.
Most people just don't dedicate the resources to things like this until the need is demonstrated.
Now hackers have it even easier to find valuable keys from otherwise opaque core dumps: Microsoft's corrected detection software will tell them as soon as it finds one.
Or it was crashing so often they didn’t have to.
Race condition to scrub the crashdump sounds fishy. When the system is crashing it’s hard to make assumptions or have any guarantees any cleanup and scrubbing is going to happen.
MSFT decided to move their APAC support center for Office365 to China. If there is an issue in APAC engineers usually respond from there. It makes sense regarding time zone and cost.
However, people change jobs etc. Hence this setup will result in a number of good engineers with very detailed knowledge of those systems who live in China.
On the more tinfoil end of ideas: if the office for this work is in a particular country it brings corresponding risks for physical opsec of the infra.
Correction is good, but why can't they go one more step and allow everyone to scan their server minidumps for crash-landed keys?
No “this issue has been corrected” for this one. Are we still budgeting storage like it’s the 1990s for logs?
> Are we still budgeting storage like it’s the 1990s for logs?
Retention policies are not necessarily about storage space; sometimes, they are there to avoid being required to provide that old data during lawsuits.
There are no legal considerations here. That’s what lawyers are for, and big tech has a ton of lawyers.
If you work somewhere that people are deleting things to keep them from being discoverable, run away as fast as you can.
I do work at big tech and see a lot of logging policies, in fact. Having lawyers involved doesn’t make it “no legal considerations”, it makes it properly assessed legal considerations. It is default practice across big tech (and other large professional industries) to delete eg email as soon as reasonable in order to reduce discoverable records. It is also required for most services that they be able to delete logs and archived material to comply with the laws about keeping personal data around for no purpose - most places will never see this brought up but if they do, it can be a big deal.
Can someone explain a bit about what kind of race condition this could be and how it exposes the signing key (in this case seems to be exposed without encryption)?
Also it feels strange that Government doesn't have its own signing key and they just use the same as everyone else. Which they didn't address and apparently do not intend to change.
I'm saying that Microsoft should have a separate private key to sign government auth tokens with.
So instead of failing safe, lack of new code to address additional scope features "failed open" and granted access to keys that didn't actually have the right scope.
I smell like gross negligence here.
2. Minimize workflows that cross security boundaries
Are my takeaways
Turns out when you write APIs that access security related things, you have to treat everything coming in as a threat, right? Shouldn't that be table stakes by now?
We need a professional gatekeeping organization because the vast majority of us suck at our jobs and refuse to do anything about it.
What training do you think could have prevented this? Microsoft deals with complex software at enormous scale. Bugs happen. And in this case, it was a severe one that's already been dealt with.
If the first instinct is to punish, people will not be helpful in identifying their own mistake.
Also, this is why companies like Microsoft have processes and systems to avoid such mistakes. They obviously failed here, but can be improved independently of the people involved.
IIRC, airline safety investigations run in that way quite successfully.
Where concern should occur is if one sees repeated mistakes of the same preventable kind.
Our industry can't even manage the OWASP top ten with new grads. Surely we can try something slight different?
M&M conferences are "non-punitive" to encourage people to accurately report what happened, even if what happened was a fatal medical mistake.
Those are the only people on the planet you can trust to never make this mistake again.