Microsoft warns thousands of cloud customers of exposed databases
reuters.com
reuters.com
- Microsoft introduced Jupyter notebook support for Cosmos (who even asked for this?) and it was turned on for all customers automatically February 2021
- Security researchers find a way to break out of the container running the notebook and get long lived credentials to other people’s databases
- The credentials can be used off network. I assume with the public Azure API.
Don’t let your intern projects go into production without thorough security review.
This is my early and potentially slightly erroneous understanding of the situation:
- Jupiter Notebook support was enabled if you used the SQL API and not the Gremlin or MongoDB API. What is unclear is whether breaking out of the Notbook container gave you access to keys for just instances using SQL API or any API.
- CosmoDB is very weak in enforcing identity perimeters so keys are a weak point. Enforcing something like hourly key rotations is left up to the customer to build.
- Azure is pretty much an "it's public IP" unless you explicitly make it private. And even when they add controls like Private end points they have weird routing mechanisms that can result in traffic bypassing controls like hub firewalls.
- Using things like CosmoDB firewalls, private endpoints, automatic regular key rotations, and only enabling features you need should have helped if this was a hostile breach.
Take everything I said with a pinch of salt :)
Lots of users. The functionality itself is great. CosmosDB is a nice OLTP datastore with a flexible schema, and the ability to run OLAP queries over that data with SQL makes it a powerful tool.
> So you can imagine our surprise when we were able to gain complete unrestricted access to the accounts and databases of several thousand Microsoft Azure customers, including many Fortune 500 companies.
This is basically worst-case scenario for a database provider.
https://www.wiz.io/blog/chaosdb-how-we-hacked-thousands-of-a...
Microsoft gets a temporary mark on their reputation until their pr and marketing departments make us forget about it. That’s it. Nobody is even going to lose their bonus over this.
https://www.google.com/amp/s/www.nytimes.com/2013/09/04/tech...
s/ran by/ran into the ground by/g
Azure is a little messier than the rest but the end result is the same.
Seems like their resources (billions in cash) aren't allocated correctly.
Incompetence. It’s not the exclusive domain of small companies. I’ve dealt with some very incompetent people at Fortune 500 companies. As a matter of fact, the very people responsible for the mishap we are discussing now work at a Fortune 500 company.
I just wanted to point out that big businesses don’t necessarily know what they are doing. People who don’t work for one assume they are well oiled machines, and people who do work for a big business assume every other big business is a well oiled machine.
For some reason they will keep millions on the books specifically for lawsuits, but shelling out $ for proper security is unfathomable (see- T-mobile).
This is how everything works in pretty much every industry. There are layers upon layers of complexity in everything, and no one has enough oversight to take full responsibility for some mistake that occurred somewhere down the rabbit hole.
Somewhere, somehow you could probably call someone's mistake incompetence, but doing that relieves everyone else of their little part in developing a chain of tools and applications that enabled that incompetence. If we heap all the blame on a single individual then no one else has any reason to improve it. We can just say "OMG, Chris was terrible! I'm so glad he's been fired now everything will be perfect!" until the whole sorry mess happens again.
Instead if we accept that mistakes are inevitable, and we accept that anyone can make one, then we're driven to build systems and processes that include guards against mistakes. Applications that check and validate things automatically, even if it's hard and expensive. That's how you get to robust software that doesn't fail like the thing in the article. Blaming individuals will never get you to that point.
Then you still need to be competent enough to to assess that this is the case. You can't judge code doing something correctly automatically if you don't even know what that thing is and can't do it yourself.
Such management and snake oil people need not be incompetent -- they might just have different goals, for example, their personal career and making lots of money -- and building very secure software can be off topic?
Someone might seem incompetent, when the underlying problem is different goals? The principal–agent problem
It's like private access has been added as an afterthought and lags behind the "normal" (public) access.
A vulnerability on this scale.. and they pay out only 40K ? I don't understand, this is peanuts compared to the damages this could cause no? Why do they not pay out respectable amounts?
How have you confirmed that this amount is accurate? Have there been bo zero-days on the black market, instead all hackers have gone to microsoft?
Would the number of hack on black market change is amount paid out increased 10x? It seems you have no basis for claiming that the amount paid is optimal
> Some of the best hackers within the NSA turned into independent contractors so they could work faster and make more money, but were on the outside? This is one of those things that someone like Microsoft is afraid of, too. If they pay too much for bugs, then some of their internal bug hunters might decide to quit but keep doing the same thing; just make more money on the outside.
He even used Microsoft as an example. I wonder if they paid less because the researcher was a turncoat in their mind.
Or they could release information publically, without informing M$
Should a surgeon ask you for millions? After all he saved your life.
What about the mechanic that finds an issue with your breaks?
If you paid someone not for the work done, but for the resulting loss if criminals exploited something, a lot of professions would ask for crazy prices.
Anyway, seems like an NFT with the instructions would've sold for far more than 40k...
If there are 50 others who can fix my brakes, one of them will likely try to undercut your price so as to get my business.
In this case, M$ could have paid ten times that amount and it would still be pocket money both in terms of their ledger AND vs the damages the PR could cause alone.
M$ does have black market competitors that would have paid far more.
Doing the right thing morally is good. But when people see time and again that simply cashing out once could set you up for life.(Maybe not in this case but maybe in others.) That is an awfully tempting risk to take and I wouldn't blame them.
We're literally saying it doesn't pay to be moral.
At the same time, these discussions about then paying to little aren't good PR either
There was a story on HN a while some guy just got 100K for an Apple account takeover exploit. That is a huge exploit, paying 100K is peanuts for 2 Trillion dollar company.
I'd be in favor of something like this. Why should the criminal justice system and every tax payer be on the hook for protecting these big companies from the consequences of their bug-ridden software?
There's many examples where the law doesn't stop companies from ripping off individuals, obfuscating terms and conditions to their advantage, hiding prices, etc. Recent example that bothers me is the paperwork at a doctors office, expecting the patient to be responsible for all bills insurance doesn't pay, without letting the patient know the costs involved, basically signing a blank check, even on a signature pad where the contract can't be seen, being told "this is just consent for the exam". The justification that these are standard forms is not a justification.
Too often big companies get subsidized at public expense. That money goes right from taxpayers to shareholders. Microsoft should deal with the fallout from it's less secure software, not taxpayers. Instead these big companies race ahead to make features and sales, playing games with juristictions to avoid paying taxes, etc.
Doesn't seem fair, and just saying "because rule of law" doesn't seem like a good explanation of why we should defend this kind of thing.
Maybe a bug exploit should be considered protected free speech. How about that law?
But if no one can succesfully sue for those damages, then what is it really worth.
The idea that Microsoft with its gargantuan resources cannot discover these mistakes and has to wait for "security researchers" to discover them and then pay them a paltry sum for their "valuable work" makes little sense; a more sensible interpretation is that the company has little incentive to find such mistakes because it has little exposure to potential liability arising from them. It needs to stage "security theater" to avoid reputational damage but does not need to achieve real results. Mistake after mistake, the stock price is not affected. Regarding an ongoing lapse in quality control that spans four decades, it has no skin in the game.
Its been a long time since someone forced me to break out these articles. Very comforting to see the justification continue with such frequency.
Don't get me wrong - I absolutely love parts of what Microsoft does. Just not this whole cloud thing.
I get, on average, 3 Microsoft major change notifications every day. Can Microsoft even keep up with their own mess internally? I totally stopped trying from the outside. I just keep an eye on .NET and visual studio stack these days. Everything else winds up in my spam folder.
if this is amazing I'd hate to see bad
I’m not claiming to be better at my job than their response team. I’m almost certainly not. The difference is I didn’t have to have meetings with 20 different internal stakeholders before acting.
It took a few hours for Microsoft to fix the problem, they responded within a half hour or so of the first complaint.
Two days seems like a long time for Microsoft to respond when commercial customer's businesses are at stake. Say what you want about Microsoft, i don't really like them, but their enterprise/commercial support is honestly pretty top notch, I'm really surprised it took them that long to get on top of that considering the calibre of customers involved.
Now, I bet you anything I could leave my current employer, get a job with a security firm and then find a major hole in my old company's service. Perhaps one that my team warned the CosmoDB team about internally. Or am I just being cynical?
If they still have that hole 18 months later, they kind of deserve it...
About 20 years ago I was working with a large company that had dedicated servers in a huge east coast data center. The servers were firewalled, but they also were attached to a secondary network for scheduled backups.
I decided I should explore that further, and sure enough I was able to connect via SSH to other customers’ servers.
I alerted the hosting company, but they didn’t seem to take it very seriously.
Then we go back to his office, and load up the 1,000lb of computer gear he's donating and I'm transporting. We wheel 3 cartloads of stuff down the back elevator, through the secured DC floor we saw through glass earlier that i couldn't take a picture of, and out an open back door right near where i parked my truck. no further interaction with security, no badge waving, no questions asked.
> A clipboard and a confident wave will get you into any building in the world!
Then I read the article...
There's a vulnerability in some jupyter notebook implementation allowing them access to unrelated notebooks which hold read/write keys to database they're connected to.
...but due to a poor deployment strategy, the instances were not sandboxed from each other.
(and yes, you would be correct in thinking that was a catastrophic failure; it's like having a folder with a directory per customer and a master access key in each folder, and WOOOPS, didn't realize people might look at ../Amazon/master_key.txt
It's pretty hard to brush this one off; it's a colossal screw up)
I’ve seen some azure documentation that stresses you should never ever use your primary keys in a deployed service. Yet they do it willy nilly.
You see the former with a lot of “online Python interpreter” sites.
1. My service's users would have to grant my service permission to read all access keys for the queue
2. My service would look through the access keys and try to find one matching some conditions (specific provided name and read-only)
3. If it couldn't find a limited one (which it would not in pretty much every real case due to the defaults) it would grab any read/write key and use that
(Later, we used Managed Identity when that became available, which avoids the song-and-dance but also has performance issues with very large scale sets. Out of the frying pan...)
I'm wondering if Jupyter had similar ability to read the primary r/w key for every database that had it enabled.
> The flaw was in a visualization tool called Jupyter Notebook, which has been available for years but was enabled by default in Cosmos beginning in February.
The way to avoid this, wouldn't have been firewalls, but using another database
Azure actually, in my opinion, made the original Cosmos worse via "synergy"; but apparently they ALSO used the name for something unrelated :) WTF? Now I cannot even tell people anymore because the name is forever associated with this debacle.
To me it seemed vaguely interesting but a little bit overpriced for real-world usage outside of fortune 500s or big government, where budgets aren't always a concern.
It has a MongoDB API Layer and a Cassandra API Layer (as well as SQL which is preferred).
Works just fine, apparently cost can blow out though with high usage, I haven't got there yet.
Does have a free tier.
The former, lift and shift scenario I can understand as you need to structure your data to work best with CosmosDB/DynamoDB etc. If it's the latter that is useful to know as I'm currently building a POC on Azure so I should watch out for it.
This was all part of a benchmarking exercise to figure out how to scale an app with minimal changes so we expected some performance penalty, just not as steep as what we observed.
However, CosmosDB is a managed service, easier than setting up your own database. Did you exclude Atlas from your conclusion for some reason? Ie did AKS do things that Atlas could not?
You must be diligent to develop your application to use it efficiently from both a cost and performance/latency perspective.
Those that put in the work may not love it, but they are very successful at operating at scale. Massive, insane scale.
I've recently tested it as needed geospartial query support for a hobby project and the cost is cents per month with the serverless pricing model. I'm also accessing CosmosDB outside of Azure to use low cost VPS hosting and latency is pretty acceptable for web type stuff.
If you are DB read/write heavy obviously pricing is going to be higher due to pay for what you use, pre-provisioned pricing will become the better value option at that point.
I couldn't justify it prior to the serverless introduction; but I'm doing 1m+ ops/mo now for <1usd for some high-concurrency state tracking ops.
For comparison, I had the same eventhub feed Kafka Connect to dump it into Mongo 5.0's new timestream collections on a $200/month VM, and after running both outputs at full volume for a couple days, Azure offered me a helpful suggestion to downsize the under-utilized VM and save money.
I haven't quite figured out if the RU rate is just too high to be economical, and counts on large orgs with large data to expect a large bill; or if the RU model simple works badly for high velocity data. I've verified that the pricing model now scales down properly for small/medium loads, at which point the various high availability/geographcial distribution features really add value. I'm trying to conceptualize how the RU cost, amortized across longer term storage and less demanding ingress/egress schedules, can work out. For our scenario, Cosmos functionally offers exactly the horizontal scalability we need.
As a different sanity check, I set up a SQL Server account as the ingestion database and saw comparable costs, so it wasn't that Cosmos itself has an unworkable pricing model.
We use it with a private endpoint and turn off public access. I've been generally happy with it.
As long you have a good partitioning strategy seems work without much effort put into it.
the blessing/curse of Jupyter being a university oss project and private companies largely not contributing back their multitenant notebook stuff (as hosting is the $ maker) is this kind of scenario. the project provides some multitenant stuff, but guessing not what MS uses.
Data science envs generally need wide read access across many data sources, yet mostly only used by a few trusted-yet-security-agnostic power users in an org, which leads to relatively low sec eng infra investments. so my guess is Jupyter security flaws (ex: extension vulnerabilities) are increasingly ripe targets for big escalations.
Then mailing customers on the 26th.
Depending on whether or not this can be classified as a 'personal data breach' (IMO you should treat it as such, since it has such widespread implications) that may be a GDPR violation. The GDPR requires 72h notice to the data protection authorities and notice without 'undue delay' to the controller.
Will be interesting to learn more info down the road
Notably, some of the customers mentioned in the security researcher's blog are the type that have exploding wallets.
As in: "My wallet is bursting open because there's too much money in it! Mr Cloud Sales Guy, can you help me with this problem?"
On one hand, my understanding is that they used TLA+ to validate the model[0]. One would assume that if they went to that trouble there is at least a specification for how it should work.
OTOH, Specifications can be flawed. This security issue appears unrelated to the design of Cosmos DB itself tho.
This bug is a security issues in an entirely separate component. No formal logic is going to warn you about exposed read/write keys.
I'm starting to understand why you found using Azure such a challenge.
To be honest, I don't think you understand anything.
In reality, all cloud providers have hiccups now and then and it seems like you may have caught Azure on a bad day.
Imagine, if the first day you tried GCP you stumbled into one of its many incidents[1] and gave up on it entirely. You'd have been deprived of a lifetime of joy with its mentally balanced API.