Microsoft employees exposed internal passwords in security lapse
techcrunch.com
techcrunch.com
Someone inside a company sets up an Azure/AWS instance, to do sandboxing, and doesn't secure it.
Many of these giant data breaches, are because some Marketing person, set up an AWS instance, dumped their entire user DB into it, and did a bunch of data mining.
I think that most cloud providers are now defaulting to locked-down, but the issue seems to be, that folks are still going outside their org, to do this.
I suspect that having some kind of secure internal cloud setup would be helpful.
However, and this is a biggie: It needs to be easy to use. So many times, security has so many roadblocks, that folks work around it, simply to be productive.
In my opinion, security needs to be the easiest path. If it is not, then that is the fault of the Security folks; not the end user. I worked for a company that had such terrible security policy, that it basically completely stopped all productivity. You could tell who was violating policy, because they accomplished their goals.
Having a policy is worthless, if it is too difficult to follow.
There are plenty of people and teams who just want a button to push to run their build. That's not so hard. Give them a CI/CD system and they'll use it.
The problem becomes the people who want to use weird, wildly divergent build processes made of fifteen shell scripts strung together and downloading arbitrary content from random remote servers because they enjoyed engineering it. They'll insist on having a blank slate of a cloud tenancy because no build system can meet their needs. The CI/CD team does not take this case seriously and will never meaningfully support it. Security is in no way staffed to build out a major extension of the CI/CD service.
Perhaps it's the data team, who has decided they would like to datamine large quantities of private information at their leisure, in contravention of privacy policies and contractual language. So they'll jump through hoops and contortions and share passwords every which way in order to do the thing they want. They will deliberately set out to disable monitoring systems because they resent the implications. At no point will they pause to consider if any of this is a good idea.
It's not just about making it easy to be secure. It's also about being able to find and stop people being insecure. One of the important things a security policy - and security organization - does is set boundaries for what activity is and isn't permitted. Crossing those boundaries needs to be watched closely... and yes, punished.
What the actual **. That's a very hefty GDPR violation and you should be fired for doing that.
> simply to be productive
If you are using private and sensitive data, that should be pretty much out of the question. You develop with a dummy database only, which doesn't matter if leaked. And after having finished testing, move to the actual data.
But those damned deadlines ...
It doesn't matter if any of this is true. What matters is they believe it and it leaves them feeling empowered enough to invest.
Today, give that person an AWS account and they can absolutely build cloud-based data solutions by clicking around and following SO posts. Add ChatGPT to guide them and the sky is the limit…on how much data they can expose.
They inherently desire to work with the company’s Crown Jewels of data. They want unfettered access to, and want to give other people access to, an entire data warehouse of some of our company’s most sensitive customer information. And when it comes to securing it properly, they often do not have the same engineering capabilities as software teams.
Anything that tries to get them to understand the risks they are taking or the sensitivity of the data, much less de-risk their workflows, is treated as an obstacle to be routed around. Often, the best I can hope for is a token effort at negotiation where their goal will be to avoid any and all changes on their part. After which I will have to monitor them carefully, because from experience the odds of them backsliding within a week are uncomfortably high.
Nothing about this is conducive to producing a healthy environment. When people's idea of "easy" is they can download the company's most sensitive data to their laptop to load into Jupyter, any amount of security controls will come as an imposition.
Given that many data engineers have a data science, data analytics, BI, or software engineering background, I'm curious if you've noticed any trends in their approach to data security?
Snark aside, there's usually a reflexive assumption that more data is always better and that anything that gets in the way of more data is bad. Anything that limits how data is analyzed is bad. Anything that limits or restricts their choice of tooling or where they use it is rejected.
Data scientists and engineers are people who are, often, working with a company's crown jewels. They are trusted with data representing the private lives of hundreds of millions of people assembled in a data warehouse. I want them to have a care. To treat this with due gravitas. All too often, all they seem to see is a neat data set to feed into R on their Macbook.
Yes, such workflows are something to be routed around.
The rest - a VPN to a virtual desktop - is entirely reasonable.
Security measures in operational workflows are only a little bit about what the end user - you - sees. Generally they're really about delivering something less obviously visible to the user but very valuable to the business, investors, and regulators.
In practice, the answer is to hire a consultancy. There are fractional CISO services out there. It's very likely that a startup without significant security experience in-house does not have the expertise to make good use of any guidance they can find. Every scenario is too specific, every personality dynamic too unique, and every new technology question too novel for there to be easy pre-fab answers that a non-security-practioner can find, understand, and apply in a correct and consistent manner.
Data engineering teams need at least part of their team to be designated software or systems engineers that are experts in stuff like how to create a robust cloud environment. This is not only beneficial for security, but also for cost optimization, performance, operations, etc! It drives me crazy when I see a data engineer or marketing team told they have to fend for themselves when trying to figure out how to build a cloud environment.
https://www.nist.gov/itl/smallbusinesscyber
https://www.nist.gov/itl/smallbusinesscyber/nist-cybersecuri...
In my organization, the problem is the security people, not the marketing people. They're simply not responsive.
Someone on a non-IT team will send a message to someone in IT Security asking a question, checking to see if a procedure is correct, or if it's OK to so something, and the IT person simply won't get back to them in an acceptable timeframe.
I'm not talking about minutes. I'm talking days. I've seen my coworkers' IT Security tickets go unanswered for more than a week.
Yes, securing your big fancy data warehouse is very important. But guess what — hackers don't knock on the front door anymore. They're going through the small cracks that you allow to form when you don't realize that your job is securing the entire organization, not just your silo.
This is why the rest of the company is going around security to get things done. Because if the money doesn't keep flowing, everyone loses their jobs.
Honestly, 1 week sounds totally reasonable to me. If you’re at the point where you need an answer immediately or it’s going to delay your timelines, then you didn’t plan your timelines very well, or you waited too long to engage your security team. You have to keep in mind that while you might be laser-focused on your app and think it’s the highest priority, your security team is likely dealing with hundreds of apps, each of which all claim to be the highest priority. Triaging has to happen, and it doesn’t always go in your favor. It really sucks for everyone involved, but I don’t see a good fix for it.
The sweet spot is empowering teams to make most security decisions on their own so the capacity of security people can be spent more wisely and development teams don’t have to wait for an answer except for in very rare cases, but the pessimist in me thinks we’re already too far behind on this and technology (and the opportunity for new security dangers to arise) moves too fast for us to ever catch up.
Basically, there is always an assigned person on call to oversee the use of any "crown jewel" data?
This way the internal users can continue to move fast, but maybe break way fewer things?
You can’t rely on that as a fix though, because it’s inherently reactive, when what you really want is proactive protections. By the time you get paged about a breach, it’s already too late.
1) Marketing person wants to play with sensitive data
2) Prior to anything happening, security-aware person is notified
3) Same day, security-aware person deploys the data where the marketing person wanted it, with proper safeguards, hands-off access
edit: I believe you addressed what I am getting at in your recent comment here... https://news.ycombinator.com/item?id=39995747
However in my experience in a big org, #1 happens frequently enough where it basically would be needed at all times, and would need a security person (or entire team) dedicated solely to that purpose. And sadly in my experience, companies are not willing to spend the money, or able to hire enough security people, to have that level of a dedicated security concierge for every team that needs it.
I may be grasping at straws here. It just feels ridiculous that my national security is being risked by these obviously super-dumb moves made by trillion dollar corporations.
The problem isn't just knowledge. The problem is getting people to use that knowledge to push back on bad ideas.
From my external POV, it feels like for all its faults, Google/Alphabet took the time to create BeyondCorp, and does not have the same record of infosec errors that Microsoft seems to regularly display in recent times.
Is that correct? If so, in your opinion, what is the difference? They both print money...
Is this just a difference in "corporate culture?"
Disclaimer: I grew up in Kirkland/Redmond, have a bias that is favorable towards MS, and would love to understand what the heck is happening.
This has been my experience. We had someone set a secure instance following all the policies and such. Then they moved to another org in the company and I started getting email violations going to the VP every few weeks because they didn't do some sort of attestation. The problem was only the user could do it and the automation had no guidance on how resolve the issue or stop the alerts. This went on for months before it was resolved.
A mistake is one thing, but negligence is another.
Beyond that, whoever allows customer data to be freely accessed and used in cases like this should also be fired.
Start culling those who are doing the job poorly to start making space for others who may be able to do it properly.
Things that limit civil liability like mandatory arbitration clauses have made this worse.
How about starting to cull top level managers who constantly put pressure on people, don't hire competent security people but reward quick results that create future problems?
Sure, the top managers is at fault, but individuals are also responsible for their actions.
Why would there be an environment or incentives where this could be a valid strategy?
Lmao I love how suggesting that developers make take responsibility for their work is met with such hostility here. Nice job removing my comments, you jokes of "developers".
There's truth to what you say, but you're blaming the wrong crowd. Security people are rarely (if ever) the same people that are creating the tools and the security features of the tools you're using. In most cases, I'm powerless as a security engineer to "make security easy to use", the only thing I could do in that regard would be to loosen our security requirements and make our systems less secure, which isn't what you want either.
IMO the problem is more that when developing products, security (and ease of security) is still not seen as that important of a feature (if its even seen as a feature at all and not an annoying cost that the product managers have to deal with). In our application security group, we actually do have requirements that any new product being built must have certain security features that make it easier to secure things, but those requirements are often some of the very first things that the product development teams try to justify delaying or ignoring.
In the app I just released, we take security very seriously. I have spent days on just the login screen, tuning it, so that it lubricates the sign on experience.
In fact, that's what I'm doing, right now. Some of our users have been getting confused with Sign In With Apple, so I'm looking at ways to make it even smoother.
To engineers, especially ones that use CLIs a lot, "UI" is simply "silly chrome for lusers."
However, the other 90% of users think that it is very important.
I have been looking at the signups for a while, and wondering why the SiA signups have been going unused (or the person tries again, with a non-SiA signup).
It was because of a "UX thing."
It's really, really important to take the "UX thing" seriously. It can make or break products. The OXO line of kitchen gadgets got huge, because the designer's wife was disabled, and he designed tools to make her life easier, which also made everyone else's life easier.
It's just that it only makes sense to me as a security thing if Sign in with Apple is specifically more secure than other methods of sign in.
I usually like 3rd party logins, don't need passwords, but there's potential for them to be multi-edged sword. Depending on the implementation.
The workflow is a bit different from the standard login ID/password entry. You use a "Sign In With Apple" button that the OS provides, and that has its own on-device (and cloud) credential generation.
I was not optimizing for the button. There's no need for a user ID, if you are using SiA, so I should not have presented that field to users. Also, some credentials are only available at generation time, and have to be maintained by the app (securely, in the keychain).
That behavior can get reflected in the UI, and may be confusing to folks.
SiA is the only SSO solution we use. I won't use anyone else's code for that stuff, and we keep all user data inside the app. We also don't collect very much.
Because in 99% of cases the code will never be able to cause harm to a human being and there is zero risk for loss of life.
The comparison you are making is kind of ridiculous. To work on the software that works on an airplane then you’ll be vetted appropriately but to work on random CRUD app you don’t really all that much.
It has to be secure and functional otherwise people will use office365 or google docs or dropbox.
And now I wonder about AI tools. There are no self hosted solutions yet so what will the leaks look like ?
Depends on your definition of “easiest.” It will become the easiest path to conduct business activities by throwing roadblocks into the insecure path. It will not be the “easiest” to use.
It’s the equivalent of lowering all the speed limits on every road to 10mph “out of an abundance of caution” instead of doing the hard work of determining the appropriate limits.
I see coworkers storing their entire password (1 week expiration) in notepad in the meantime.
I was going to store the password in a portable Keepass, but I got blocked from accessing a software download website and was scorned by the proxy block page.
Frankly, I think the cyberArk method is effing foolish and having regular 2fa integrated into a product's login makes a lot more sense.
There’s nothing inherently wrong with this tbh, from a practical (not policy) security standpoint.
If I have code execution on your box and can access your notepad, I can also simply monitor the clipboard for when you copy the creds from cyberark.
Secure, accessible password storage shouldn't be a burden.
Secure data handling can't ever be the easiest path. The easiest path is to have all data available all the time to everyone with zero access controls. Adding any kind of security to that is going to reduce the ease, even by a tiny amount.
Security needs to be pragmatic, people have a job to do and a business to run. Security teams should never forget they are there to enable the business.
But at the same time, the company needs to have a culture that recognizes data security is important. If that is not present, there's nothing the security team can do.
The article basically says "these dudes told us they found some passwords for databases on an open machine". Ok, well a) that may not be true and b) they may not be passwords with high value. Just because something is a "password" doesn't mean it is useful for anything.
There seems to be nothing on the web about this incident besides more articles repeating the TC mother article.
For reference: https://www.wsj.com/articles/microsoft-employees-are-hooked-...
In some of the orgs and products there are zero career incentives for addressing legacy setups and there are big incentives for pushing out new features and initiatives. I've never heard of anyone being promoted for driving an initiative to fix obscure technical debt like this. It's a systemic issue that won't be resolved with security trainings.
Awareness is one weapon, but really security is so technically complex to do well is that security needs to offer good solutions.
Think of it like a GPT prompt: "give me a test harness for user profiles" "give me a secure login process" "give me a SSO service"
Instead security groups want to "review"/"approve" which is just shitty compliance overhead. Because of course they don't want to be the ones to blame for bad security code. They want their compliance review checkbox, and everything else was the "evil rogue programmer/hacker" to blame.
And alas, the security people I've seen in large corporations I've worked at have been almost morons. Internal support portals for enterprise passwords? Dumb password rotation rules, 6-8 character limits, and chrome complaining about obsolete ssl suite usages. Degrees from specious Florida universities that mostly seem to be about tuition collection and beach access than actual academics. Asking them about major breaches and the technical basis of them and getting blank stares.
No idea what the vaunted Amazon or Microsoft do, hopefully it is better. Amazon abuses its employees, so that opens them to a rich array of social attacks and leaking, in addition to the high probability that Amazon farms its customers for business strategy. Microsoft has always been about monopolistic sociopathy trumping technical concern.
Cloud providers are a high customer support business and it is resoundingly obvious that Amazon and Microsoft view those in as high regard as most SV organizations and really all corps in general: low paid and overworked, if any.
Corporate management will only invest in things they see are worth the reduction in managerial bonus payout, kinda shareholder value, kinda risk reduction (at least within the period of time their stock grants are active). There's really no good aligned incentive, just like environmentalism because it is an unquantifiable risk to the great religion of economics and finance, and therefore doesn't exist.
> Internal support portals for enterprise passwords? Dumb password rotation rules, 6-8 character limits, and chrome complaining about obsolete ssl suite usages.
I'm curious as to why these would require some outside security group to deal with? These all seem like issues that should be solved either by someone in the devops category.
Update your dependencies, disable unused protocols, have a QA guy find your improper memory glitches, and now 99% of the security establishment is irrelevant to you. These are operational issues. Most security issues are just operational issues.
Generally I find that I mirror the effort I put into training to the effort the creators put in. If a training is just 20 slides of text to speech, I will put that on 2x on a background thread.
If the training is a well-produced video series I'll watch the full thing at normal speed.
I'd figure out what the least-technical person (with data access) in the company needs to do their job, and find the friction points that prompt this person to find the path of least resistance.
2FA in general makes a bad solution worse all for the sake of security; it's not long for this world.
Can you expand on this?
With reasonable setups (trusted locations and trusted devices) the friction is absolutely minimal. And it isn't even a question, the security benefits are astronomical.
I am not understanding how you can say 2FA is "not long for this world".
2. 2FA way - type email - type password - click submit - switch to secondary email/text app - wait for email/text to arrive - copy text string (not consistent, depends on input method) - switch back to main app - paste or manually type text string - click submit
3. Hybrid (passkey?) setups are a great step in the right direction. Even more convenient than passwords, and more secure, without app switching, which is the huge friction point.
#3 is still MFA.
Enter credentials -> receive push notification and press "yes" -> login.
They wrote #2 to be purposefully long and convoluted. "Copy/paste code" is somehow 5 steps, with a waiting period? We really needed to detail out "switch app" as 2 steps? Come on.
As another example:
If you were to give directions to someone on how to get to your house, do you say: "Turn right at XYZ street, follow that up to ABC street and take a left, last house on the right"
Or do you say
"When you are 50ft from XYZ street, press on the brake pedal. When you get to the corner, turn the steering wheel to the right, hand over hand, then get the car straight again, press your accelerator, approach the speed limit, check mirrors every 20 seconds [...]".
Both are true. One is unnecessarily detailed to make it seem more complicated than it is.
Imagine needing two apps to login to one app
Do I need to go back and explain how the computer chip is made and what transistors are, too?
Or maybe we start at the part where you have to find a store to purchase a phone, and walk through that process?
I've been using it for a few months and it works very well.
I have not tried to set up MFA for someone with memory deficits, so I can't speak to that.
All of that is completely beside the point, though. I'm not sure why it matters. There is 0 chance that MFA is "not long for this world".
Yes, and given all the recent examples of accounts hacked because they didn't have 2fa for the sake of convenience, it's worth it.
Imagine futuretopia with me for a moment; when you are hired at a company they send a very secure cryptographic keygen to your subdermal ID implant. Maybe with 3 different fields for "employee" "division" "role"
When you enter the building, implant comms automatically using "employee" with the mantrap pads acting as a second factor as you type in your yearly revolving 9 digit employee passcode
Same with the workstation when you log in using "division", and maybe when you specifically access certain files "role". They all last a user session length, and can all use the employee 9 digit as primary login
This scenario gives you effortless 2 factor, 3 different times, and the employee only has to put in their simple 9 digit code thrice daily, maybe 6 times account for lunch break.
Does this scenario seem like two factor that's on the way out and causing more problems and inconvenience than it's worth?
Two factor sms where you have to type the code in yourself and sms is insecure? Sure that's dog shit. But we can make a better future instead of disregarding a pretty solid security concept
This is immediately the wrong design and is confusing authentication with authorisation.
Your "subdermal implant" needs only one feature, (two if we wanted greater privacy than exists for employees today, but let's not get ahead of ourselves)
1. "I am still me". The implant can produce a copy of its public key, and will cheerfully sign simple freshness challenges allowing the employer to confirm that this is the same implant that the person hired had.
That's all you need, that's the whole thing. The question of whether you're in Org Unit A9 or ZQ, whether you're a Deputy Senior Assistant or a Senior Deputy Assistant, whether you're part of Divison F or not, these aren't authentication they are authorisation questions and there is no need for them to belong to you, they can live near the decision since they can be changed by other people - you can be demoted or promoted, moved, reshuffled, fired - that's not a decision for you it's a decision for somebody else.
Most average people are just confused by it, but it's so important for security the focus shouldn't be getting rid of it, but figuring out how to make it buttery smooth to use.
That's pretty abysmal response time for what seems like it should be a simple fix for a serious problem.
I'm not defending it, because it is abysmal, just explaining probably why it took so long.
The system was already broken from the moment credentials were leaked (or if you're a purist, from the moment they were hard coded into a file). The impact just wasn't realized or felt yet.
I'm definitely not defending it, just saying the red tape in some orgs can be nearly impossible to defeat.
This problem should have escalated as high as needed though to bypass the red tape and fix the security issue immediately.
Probably a failure of weak middle management not having the guts to communicate the problem up.
2. In what hypothetical universe does a life support system connected to the internet contain a trove of customer PII? And where disconnecting this hypothetical system from the internet _cause the patient it's supporting to die_?
If you design a system that could cause significant personal harm or a meaningful degradation of quality of life for someone due to cloud provider downtime, you're just a bad engineer. There's no such thing as a data center that's failure-proof. If someone's well-being depends on a system, it's simply negligent to be intolerant of failure conditions.
Why in the world would they not use it?
> The researchers notified Microsoft of the security lapse on February 6, and Microsoft secured the spilling files on March 5.
What???
This doesn't leave me confident of the security of any data in Microsoft's possession.
> The panel said the intrusion, discovered in June by the State Department and dating to May “was preventable and should never have occurred,” blaming its success on “a cascade of avoidable errors.” What’s more, the board said, Microsoft still doesn’t know how the hackers got in.
...
> It said Microsoft’s CEO and board should institute “rapid cultural change” including publicly sharing “a plan with specific timelines to make fundamental, security-focused reforms across the company and its full suite of products.”
> This doesn't leave me confident of the security of any data in Microsoft's possession.
Why would you have ever had confidence in them? They have by far the worst cloud in terms of security, and it isn't even close.
A random selection of serious security incidents:
just from Wiz from the past 2-3 years, and of course they aren't the only ones:
https://www.wiz.io/blog/secret-agent-exposes-azure-customers...
https://www.wiz.io/blog/storm-0558-compromised-microsoft-key...
https://www.wiz.io/blog/azure-active-directory-bing-misconfi...
https://www.wiz.io/blog/omigod-critical-vulnerabilities-in-o...
https://www.wiz.io/blog/chaosdb-explained-azures-cosmos-db-v...
of course Microsoft AI researchers sucking at security: https://www.wiz.io/blog/38-terabytes-of-private-data-acciden...
Nice overview from Corey Quinn that predates some of those but things were already horrifically bad: https://www.lastweekinaws.com/blog/azures-terrible-security-...
Oh and there's also this, them selling your usage patterns to partners (hopefully they've stopped): https://twitter.com/QuinnyPig/status/1359769481539506180
Oh and another one where they bungled the response: https://twitter.com/QuinnyPig/status/1536868170815795200
I find it impossible to believe that Azure as a whole organisation takes security seriously. There might be individuals that do, but definitely nobody with decision making power. Half of the above described exploits are trivial and should have never passed any sort of competent review process.
Carelessness of an individual
I grit my teeth every time I have to interact with it. Not surprised some people use any other solution at hand.
The truth is that internally they probably don't like using their own solution for key management.
I usually opt for environment variables for secrets, which I know isn't awesome, but keeps them out of code at least.
Regardless, secrets should never ever be committed, even in a private repo (or one you think is private).