Microsoft lost its keys, and the government got hacked
techcrunch.com
techcrunch.com
However, I feel like it needs to be said: Microsoft makes bad products[1]! They're overpriced, insecure, slow (it's astounding to me how slow their web properties are), and hard to use. Easily some of the worst UI I've ever seen and, what's worse, they've been like that my entire career. They keep slapping lipstick on the same pigs and claiming to have reinvented themselves.
Their support is so ungodly bad that it literally took one of their reps over a day to figure out what timezone I'm in. Meanwhile, two of my employees' emails were being randomly CC'd to the CEO for no apparent reason. The cause? Nobody knows. It just stopped happening. Even Microsoft couldn't tell me why it happened.
With a $2.5 trillion market cap, it may seem like they're killing it but, I really think they're riding on momentum just like IBM did. Twenty years ago, IBM's market cap was double what is is now and, in my experience, their products are every bit as bad (admittedly, I have almost no exposure to their big iron products).
So, what are our alternatives? I know many people who don't like Apple for various reasons. Linux on the desktop has been promised for 20 years. It's closer than it has ever been to being a solid replacement but I don't know that it's quite there yet.
It saddens me to say, I don't know what the answer is. I just know that we need to get away from Microsoft products and fast.
1: In my experience, the only two exceptions are VS Code and SQL Server.
People are only not shitting on IBM so much anymore because they sold off Lotus Notes...
Having used Microsoft recently, after Google for work, I'm absolutely flabbergasted at how terrible Word, Teams, Outlook, etc are. I would describe the average Microsoft product as a zombie in its own feature set, at war with a different version of itself, and attempting to steal organs from other products to try to transition to every possible entirely different role in the suite.
Attempting to use a Microsoft product is like watching developers capitulate to some kind of bizarre internal political war between Product managers.
It's walled garden for lack of a better time and solving problems easily and quickly dilutes the value of MSCE and the army of paid consultants that exist.
It is a completely different universe from Linux
I found out later that they use their certs as filters for resellers to help Microsoft identify which ones are serious players. What a waste of time and money.
If I understand what you are saying correctly, this is not exactly true.
The IT industry is a box-shifting industry, and therefore Microsoft, like all other vendors, differentiate their resellers primarily on dollar-volume.
So the REAL reason a Microsoft reseller is Gold is primarily because they have shifted more product than other resellers on lower tiers.
Sure, to actually get the Gold badge, you need to have X sales-reps who've passed the course and Y technicians who've passed that course. But bodies-with-certs are by comparison a very minor secondary criteria.
The key criteria is always that your company has sold $big_number per year. Your entire organisation can be hiring exclusively Microsoft certified people, but you ain't getting the badge unless you're shifting the product no matter how many certified people you have on staff.
And that is why Gold certified is meaningless. Because all it tells me as a customer is you've got good sales-reps on team.
Problem is that there is no other provider I would rather recommend. They are all bad in one way or another.
App Insights, DevOps pipelines, and WebApp slots look like future tech when compared to codepipelines and x-ray.
DevOps in particular feels so close to being a really great tool. Not having to jump between 8 different UIs to deploy something is magical. If only they'd polish the rough bits and invest a bit more into it.
That's why I wish DevOps made it a really clear cut choice. There's some really low hanging fruit and silly decisions that spoil the initial impression:
YAML pipelines have been the "official" option for at least 4 years now but when I create a new project, every issue has a "deployments" section that tells me it only supports classic releases. Classic pipelines/releases can now be disabled at the project level at least.
They're introducing a new widget that shows open pull requests across repositories but for an absolutely insane reason it's limited to 10 repositories, just why...
Environments still have no support for resources other than VMs and AKS.
The mandatory "sprints" backlog is just stupid as well. It's a clear artificial limitation rather than one necessitated by the platform.
Another issue is that Microsoft have done a poor job at communicating the potential longevity of the platform. The Github acquisition is the first thing that gets mentioned when DevOps is brought up as an option, which is a shame because DevOps feels much more capable/cohesive/advanced than GitHub.
Despite these problems it's still been the tool with the absolute least friction and most potential when it comes to organised development work. The alternatives I've used come nowhere close.
running `az` without parameters takes 10 seconds on my machine. `az version` and `az --version` return differnt outputs and are also very very slow.
When I run `az account subscription enable` it tells me "Command group 'account subscription' is experimental and under development.".
Could you explain a little further what you don't like about Azure and why you would still recommend it over the competitors?
I recently collected some notes of bugs and errors for an MS rep. I was setting up just a .NET web app with a SQL back end, as simple and vanilla as it gets.
I found 60 bugs.
SIXTY!!!
Also,unrelated: Azure AD is now "Entra ID" (????) now lol.
Azure AD was initially a shim to enable O365. Now it’s evolved into a pretty awesome identity platform for Enterprise and B2C.
The problem is M365 adoption (ie O365 + the other crap) is high. So they are out of stuff to sell. Entra will add the governance and compliance bullshit and increase revenue.
End of the day, Microsoft chisels $38-55/head for every corporate worker. Services are all about cash flow, so they are looking to grow that up to $100 user/month. Every $1 of subscription spend is like $400 on the scorecard for the bonuses of SVP staff.
Because of anti-trust, they can’t just do what Adobe does and raise the price every few months. They just keep adding services and move stuff around in the bundles. M365 Plus+ E9 with Entra technology will give you $350 in value for only $103/month!
Same story when they moved to MSAL a few years ago.
I've worked with both Azure cloud and Azure DevOps (or whatever the feck they call it now) for a long time (maybe 10 years?), a lot in a corporate setting, but also for my startup.
Apart from the same price issues that exist across all the big 3 cloud providers, I actually really like Azure.
What I hate is the unfathomably sloppy implementation that their developers refuse to fix unless held at gun point.
You literally have to have an MS account manager about to miss out on millions of dollars to force a trivial fix through their pipeline.
They have zero “customer obsession.”
BIG businesses and governmenta are thier customers, demanding maintenance of terrible legacy issues because they literally use bugs as features in thier even more poorly made and maintained custom systems built by some fly by night dev shop 2+ decades ago....
MS doesn't care about individuals, individuals are just eyeballs to them.
Also, Azure is "all new", so legacy compatibility shouldn't be a big concern.
Except... it is, because a bunch of half-baked MVPs were thrown over the fence and now have to be supported while they work on the "proper" solution. We're up to the 3rd iteration of VM monitoring, the 3rd iteration of AAD client libraries, etc...
Not that I'm tethered to Azure... I've used a combination of dedicated server, DigitalOcean and Cloudflare for my personal stuff... though just started delving into the Cloudflare (edge compute) stuff.
There are definitely ups and downs to them all.
(Terraform does suck in that its import functionality can't spit out the HCL representation)
When most people hear the word "Microsoft," their initial thoughts typically revolve around Windows and, at most, Office. However, it is important to note that a significant portion of Microsoft's profits actually comes from their Dynamics 365 offerings.
Outlook became popular because it was tied to Windows 20 years ago, but today this just isn't the case.
And besides, it's far easier to migrate between two webapps than it is away from a native app full of infra.
I have zero confidence in that testimonial, given my own experience with open source desktop environments, and KDE. zero confidence whatsoever.
The experience is so much better I really don't get this mentality.
* You often need to do non-standard system tweaks to get the system in a working state. It's everything-doesn't-work-by-default.
* You often need to install/prevent instalation of bad quality drivers and associated bundleware to make devices work.
* Microsoft can't decide what UI framework they will support (or drop support) so built-in applications happens to be built with the current "thing", the old ones staying on the older unsupported frameworks.
* You need to constantly tweak a combination of Windows 3.11 control applets, Windows 95 control panel applets, Windows 7 unified control panel glass, and Metro Settings Universal Windows Platform app to keep things working.
* Microsoft can't decide what random widgets go on the taskbar, or the file manager, or your start menu, so it's always disrupting things every few updates.
It is arriving at a point where the only "consistent" UI thing in Windows is Steam, and Steam isn't built-in, and is a hot mess of inconsistent UI itself.
Just say “better.” Gnome and KDE are decidedly not perfect, but they are both quite usable even by casual windows-only non-techies.
FWIW I’ve been using Linux as my primary desktop OS for more than 20 years—it does no one any good to oversell it. Touchpad scrolling rate is still broken on Gnome/Firefox. Angels have to sing (or use xorg) to make s3 suspend work. Fedora auto-update can break if there’s “too many” packages to update. Touchpad scrolling can be misinterpreted as swipe-left or swipe-right by KDE on some hardware.
The point is, if you bang against any of these sorts of issues in Windows, you’re frequently stymied. On Linux, you can either fix it, or at least route around the damage with an alternative.
Most of these enterprise software (for non technical users) these days is just a web app so a vendor not releasing their software is not a problem here.
Like schools? ChromeOS is Linux and seems to be doing ok.
> Eschew flamebait. Avoid generic tangents. Omit internet tropes.
https://news.ycombinator.com/newsguidelines.html
I flagged it - I suggest you do, too.
I think the answer is a company willing to put in the work to force GPU manufacturers to provide high quality hardware drivers for a commercial Linux Distro. I mean all out partnership with the various vendors. Then sell prebuilts, maybe someone like System76 has come close, or has the capacity to do so. The Linux issues for me always seem to be hardware specific. Then its just making sure whatever the default DE is, has enough necessary polish.
Is there another aim of GPU that I'm ignoring?
Microsoft makes fine enterprise software, sadly. To know and believe that, though, you have to see what the alternatives are in the enterprise world. It is atrociously bad.
I do think Chromebooks are an interesting use case showing you don't need massive software. Schools have taken to those very well and are almost certainly competing with most enterprises for number of users and use cases.
There are various ways to get Windows games working on Linux or Mac, but they aren't out of the box, which is significant enough to prevent wide scale adoption.
And it is more than installed userbase, which is very real. For the major distribution targets, the devkits have a ton of resources poured into them that major studios can't ignore.
That said, still feels like a non-sequitur? I can't imagine "gaming as a target" is a concern for cloud customers. Certainly not the government?
Hum, no they don't. They make shit and use anti-competitive practices to push every other offering out of the market.
The fact that you only see experimental alternatives without the maturity to get any polish is caused by them destroying the competition, not the reason they won.
That said, fully fair that, even in the field, they probably rely more on bad practices to shoulder out competition than I'm crediting.
I'm amused that you seem to have missed what the "G" in GPU stands for. As someone who occasionally plays games, I'm a bit salty about what it did to the prices.
Even if you think about the corporate world, I'd imagine that there's a lot of CAD on Windows and maybe some 3D animation still? I'm not that familiar with those uses.
Is a bit like movie firms. The popular view is that Apple has that locked down, but that seems to only be the prosumer market. Most of the main studios have custom render farms that are probably linux based, nowadays. Studios had the motivation to not get locked into large contracts that grow based on the size of their render farm. (Incidentally, this is why Wacom has such outstanding linux support.)
That said, fair point on this not being limited to corporate. Not sure why I had that bend from the intro post. I was probably biasing it more to the topic of cloud than I was corporate, but those seem roughly related?
Some think tank should make an analysis how many billions of dollars are lost in productivity across the globe if you compare the current MS Office stuff to an alternate of "the Windows 2000-era stuff, but with security patches" or "the FOSS office stack, with more money behind it".
Anyone wanna estimate how many digits that number has?
(Teams, while being ungodly slow, at least has the value proposition of being an integrated product - persistent chat, file sharing and voice / video calls - that really didn't exist in 2000)
Since then, for a couple of years, my father got a notification saying that it was my birthday (and he even obliged to congratulate me, once; he was -- and luckily is -- old :) ). That setting was not available anymore anywhere, and my birthday was set to the correct date, but the problem still happened. I contacted Microsoft, but it was like they had no idea what I was talking about nor what could be the problem.
So yeah, I don't trust them nor like them either.
Microsoft’s operating budget was like $123B last year. There are bugs in office products that have been there for years
What exactly are they doing with all that money?
https://techcrunch.com/2019/07/22/microsoft-invests-1-billio... https://techcrunch.com/2023/01/23/microsoft-invests-billions... https://www.reuters.com/technology/microsoft-talks-invest-10...
The only customer Microsoft seems to care about or is really even cognizant of its responsibilities towards, is Microsoft itself. Every single one of its products and services exists solely because and only for as long as some division of Microsoft believes it requires those specific things and whatever they're meant to evolve into over time, in order to manage its increasingly complex and ever expanding internal operations.
It's like a snake eating its own tail.
Paying for ads and for developers that make ads I suppose.
could have been a backdoor, sure, could have been something else. could have been carelessness. could have been blackmail. could have been a rogue employee. could have been malware, spear fishing, or any one of a hundred other things.
did I miss something or are you premature in blaming this on a backdoor?
It's hard to secure encryption keys. If you had a backdoor based on a key that you had to keep secret, leaks like these can happen. If Microsoft (or anyone) can't keep their encryption keys secure, they'd be equally unable to keep a backdoor key secure for the same reason.
Either way, when the details about the back door leak, the system ceases to retain the properties of secure encryption.
That includes the comment you are replying to.
When your data is stolen because you were working in Cancun and the Mexican CIA was compromised, will you ever even know?
The point isn’t blaming a back door, it’s just pointing out that the more stakeholder that possess a key, the higher the probability of a compromise.
And it's not that the alternatives are perfect, but Windows is as secure as swiss cheese.
In IT security jargon, the "Swiss cheese" model is a good practice: https://en.wikipedia.org/wiki/Swiss_cheese_model
A self-evident realization: every single product/system is a liability.
There's nothing in any of Microsoft's revealed communications over the years where they ever seem to aspire to make better products. I mean, sure, obviously, there are pockets inside the company, but that has never seemed to be the purview of the board or the executives. If there's a thought leader in there somewhere who had passion to make great products like Jobs, I can't think of who it would be.
I almost feel bad for Microsoft, and I say this as a Microsoft-hater and Linux zealot who ran it on the desktop for 19 years before switching to Mac. It's the legacy backward-compatibility that makes Windows so attractive to a lot of organizations which makes it so vulnerable. OTOH, corporate IT divisions love it because Microsoft lets them do all sorts of stupid things to it, like prevent me from changing the desktop background, so it's kind of a deserved punishment.
Not that they are bad products per se, but as many organizations use Microsoft products they are prime targets for too many hackers and it’s hard going to bed thinking that your identity info might be hacked someday and/or not knowing if it will be.
The surface area that Azure/Microsoft have is just too large for it to assume to be protected by one company whose security as not been the most stellar. So we are actively moving to GCP as well.
https://techcrunch.com/2022/12/22/okta-breach-source-code-gi...
Okta looked very bad during the entire saga where they kept denying they were hacked until the proof was insurmountable. Pretty sure there was a large discussion on HN at the time too.
1. The IDP signer (like microsoft or google) 2. The Cosigner (like bastionzero.com)
...so that even if microsoft's signing key is stolen, the attacker also needs to compromise the cosigner's signing key as well. It's like multisig for authentication tokens.
I don't know if OpenPubkey would have helped in this particular case as the details are still coming out[2], but I think the future of authentication schemes must require that authentication tokens must be signed by multiple signers at different organizations; Authentication systems with single point of compromise signing keys is too fragile. Or put another way authentication via multiple independent roots of trust is just too powerful of a security tool not to use.
[0]: BastionZero, https://bastionzero.com
[1]: OpenPubkey: Augmenting OpenID Connect with User held Signing Keys, https://eprint.iacr.org/2023/296
[2]: It appears the key stolen was an MSA key, not an Azure AD signing key. The MSA architecture might not fit into the OpenPubkey model (or it might I don't know enough about MSA signing keys work to say). Had it been an Azure AD signing key then OpenPubkey would mitigate the theft of an Azure AD Signing key. https://www.microsoft.com/en-us/security/blog/2023/07/14/ana...
Same keys to everyone's kingdom!
By adding a small bit of human text, the human doesn't have to ponder the list and wonder what they will get if they click a link.
And if there were human text, someone arriving by way of the parent comment link would have an easy time understanding the context without having to scan a lot of characters and make assumptions.
Microsoft takes pains to obscure role in 0-days that caused email breach - https://news.ycombinator.com/item?id=36740133 - (64 comments)
Microsoft still unsure how hackers stole Azure AD signing key - https://news.ycombinator.com/item?id=36731731 - (9 comments)
WSJ: China allegedly hacked Microsoft Azure - https://news.ycombinator.com/item?id=36693469 - (6 comments)
Microsoft Warns That a Chinese Cyberattack Breached Government Email Accounts - https://news.ycombinator.com/item?id=36697272 - (5 comments)
Thousands of Microsoft Customers May Have Been Victims of Hack Tied to China - https://news.ycombinator.com/item?id=36692494 - (3 comments)
Microsoft government email compromised (and quietly fixed) - https://news.ycombinator.com/item?id=36696852 - (3 comments)
WSJ: China Hacking Undetectable from Less Expensive Microsoft Services - https://news.ycombinator.com/item?id=36713560 - (2 comments)
Chinese Hackers Breached Government Email Accounts, Microsoft Says - https://news.ycombinator.com/item?id=36690598 - (1 comments)
China-based hackers breached US gov email accounts, say Microsoft, White House - https://news.ycombinator.com/item?id=36694188 - (1 comments)
Microsoft mitigates 0-day, allowing forged AD tokens - https://news.ycombinator.com/item?id=36708294 - (1 comments)
A cloud flaw gave Chinese spies a key to Microsoft’s kingdom - https://news.ycombinator.com/item?id=36716028 - (1 comments)
Chinese hackers breach email of Commerce SEC Raimondo and State Dept officials - https://news.ycombinator.com/item?id=36719412 - (1 comments)
13 posts with no comments: https://gist.github.com/bcjordan/b579960792ee146f905c3e66f39...
Someone hacked Azure AD[1] and accessed the data of 25+ orgs and the reaction is *crickets*?
[1] I'm not exactly sure which Azure component was hacked and the MS communication seems intentionally unclear and obtuse on this. It sounds like the private keys of Azure-internal auth servers were stolen, but technically those private keys were not supposed to work for the accounts/orgs in question, except they did, and, in any case, MS doesn't disclose how the private keys (one? multiple?) of their cloud auth servers were stolen.
[1] https://arstechnica.com/security/2023/07/microsoft-takes-pai...
That's an amazingly big, world-wide security hole, and how much interest did it draw? Hey, look, there's a celebrity scandal over there.
See this mind-blowing blog post from Wiz where they live edited bing.com search results, injected XSS into O365, accessed exabytes of internal MS file storage and more: https://www.wiz.io/blog/azure-active-directory-bing-misconfi...
I have no idea how Azure maintains its reputation as a safe enterprise choice. Microsoft really just has an almost impenetrable reputation I guess?
Microsoft have two identity services being MSA and AAD. MSA is used for consumer Microsoft accounts for use with products like Xbox. AAD is Azure AD that businesses use. When a client wants to authenticate to a Microsoft service, it asks MSA to sign a token for services accepting MSA tokens, or asks AAD to sign a token for services accepting AAD tokens. Someone has gained access to an "inactive" (but not "invalidated"/revoked) MSA signing key. It's not stated in the analysis whether Microsoft services such as Xbox would have accepted a token signed with an "inactive" MSA signing key, but it sounds like this may have been the case. It's just consumer privacy and security at stake and the attackers apparently weren't interested in that, so the analysis is rather ¯\_(ツ)_/¯ on consumer impact.
The problem for Microsoft is that they had incorrectly configured OWA API to accept tokens signed by the MSA identity services even when accessing mailboxes that should require an AAD signed token. So now the attacker could access mailboxes for all business and government users around the world (excluding GCC High, DoD, etc which Microsoft state are not exposed to public networks).
Microsoft only found out after one of their diligent customers queried Microsoft on why they were seeing logs for user mailboxes being accessed in unexpected ways. Speculatively (not in the analysis) this may have been noticed as a random IP address outside of allowed subnets accessing mailboxes at odd hours, and the customer checked their AAD configuration to ensure IP subnet restrictions were configured correctly and verified the user was not overseas.
The attacker looks to have been quite careless by accessing OWA API from very obvious red flag locations for legitimate users. For example, the attacker could have used a residential ISP subnet in Washington DC to access a GCC hosted mailbox of a government agency with a presence in Washington DC, and only done so in hours that an employee may be working from home in Washington DC. Or the attacker could have gone after soft targets like small to medium businesses that contract to the government, hoping these soft targets wouldn't be monitoring nor have the ear of Microsoft for prompt incident response. Instead, the attacker has used random dedicated hosting providers in places as far as Europe and accessed mailboxes during hours coinciding with business hours half way between the US and Europe. It's possible the attackers thought this attack would be detected quickly, and therefore were always just going for a hit-and-run approach knowing they'd get their data prior to the attack being noticed and blocked. But it does make the attacker look brazen at best, and unsophisticated and careless at worst.
[1] https://www.microsoft.com/en-us/security/blog/2023/07/14/ana...
[2] https://learn.microsoft.com/en-us/office365/servicedescripti...
[1] https://www.cisa.gov/news-events/cybersecurity-advisories/aa...
[2] https://learn.microsoft.com/en-us/office365/servicedescripti...
Just stunningly bad and would make me reconsider how much MSFT I hold, except that no one seems to care.
(Maybe if you have an Microsoft consumer account that you don't use, forgot you had, and might be incompletely set up.)