Launching attacks during major news events surely also helped the attackers stay under the radar for longer.
It's possible that <Random F500 Co> has a great security team. But it's also possible that <Other F500 Co> doesn't.
It's quite funny in a way: regular mail worked for two hundred or so years without too much in terms of trouble, ok, we had some spam but that was about it. And now mail delivery has become so complicated that the mere act of accepting mail can lead to your corporate secrets being made public or lifted without your knowledge.
But you can still send it if you want that kind of security. There’s trade offs galore, but obviously the cheapness and convenience of email seems to have won out versus security concerns.
Email cam be copied and sent wherever without the operators knowledge, from anywhere with internet, if they break into the mail daemon.
In the digital world there is no such sentry.
Also, at some point the cloud provider may figure out that they can increase profitability by hiring more and more below-average people and just market them as world-class.
What I described is a situation of basically converting reputation into cash. Once you're known for having "armies of above-average developers" and then cut back on employee quality, it's going to take a long time for the market to figure it out (and you can probably extend that time significantly with slick marketing). In the mean time, you profit margins are increased.
Besides, it’s not really true today that clouds only employ “above average” developers. I mean hell, they employed me!
In part, this is the different service model: if I go to AWS and buy, say, S3 they have a very clear responsibility not to lose your data and to serve it quickly. If my CIO picks one of the bargain basement outsourcers and the centralized storage service fails badly, each different group will be saying that the failure wasn’t due to them but the company management, outsourced project management, the contractors who set it up/operate/monitor/secure, vendor products, vendor professional staff, Microsoft, etc. Since truckloads of cash will have been spent by then, many of those parties only care if it’ll reach the point of a lawsuit and everyone in the approval chain who didn’t say it was troubled before has an incentive to say the failure was unforeseeable and the solution is not to hold anyone accountable.
When you proceed to the logical end of enforcing simplicity to achieve security, you get OpenBSD. That's great for certain applications, but I think we can agree it doesn't check a lot of boxes for contemporary feature set demands.
My point being, achieving that is way harder than it sounds.
Speaking of OpenBSD, that might actually be a better OS for most stuff on the shop floor in companies that I have seen from the inside, where Windows is used almost exclusively. The plus being, nobody can really mess around with it. There is usually exactly one app that needs to run 24/7/365 with occasional opportunity to update e.g. during a maintenance window and that's it, anything that causes the app to close is lost time on the shop floor. OpenBSD being minimal is a large plus here.
Let’s say you are a big airline company, there is absolutely not a single reason you should manage your email system. Your job is to fly airplanes not to manage some goddam emails.
The really fun part in that is that most of the big airlines actually outsourced some key part of their core job (IT wise I mean), like how they manage seats and load, this kind of stuff, while keeping some absolute non-core IT services internal, like an internal Exchange system with dozens and dozens of people to manage it.
The state where big companies are make the second option impossible. That may be unfortunate, I don’t know, but that’s really where we’re at.
There is absolutely no way to cure big companies from all the shit they have accumulated. For them, the actual restart is to go to Cloud. Hopefully they will not go simply bare metal, because then they can recreate the exact same shit but in the Cloud.
One camp assumes if you don't expose it to the internet, and keep it on-prem, it's secure. Think exchange server on-prem (but let's overlook the gaping internet exposed parts - they don't see those, they see the fact it runs in their office).
On the other hand, it's public cloud, hosted service, rely on a big company with the resources (but accept loss of tenant isolation when something big goes badly wrong, and hope the cloud host has the skills to mitigate and detect issues).
We need more secure systems, but if they're publicly exposed then you'll require that team of experts around the clock simply to detect the potential of a compromise. Something I see a lot of confusion around is knowing when something is compromised. Responding is then "easy" in comparison for them, but they don't know what they should be looking for. With complex exposed services (mixed user and management plane over HTTPS, email interfaces for multiple protocols with different versions and authentication mechanisms), the likelihood of serious comprise tends towards 1.
Better hardening services would help to get some way towards the world you describe, but that has to filter through the whole supply chain and ecosystem - no, you shouldn't be able to manage the exchange server from outside, nor should any such interfaces be exposed. No, the exchange service shouldn't execute aspx code from folders on the local filesystem that can be modified other than through a privileged updater service.
But what we pay for is features.
https://offensi.com/2020/08/18/how-to-contact-google-sre-dro...
And the HN thread:
By consolidating targets when you can not even reach the level to protect a single one you are making the situation worse, not better by consolidating. For it to make any real amount of sense they would first need to demonstrate an ability to prevent attacks at least in the correct order of magnitude and then demonstrate that they can scale up without creating correlated risk. Only then does it make any sense to actually centralize on a single solution, let alone a single provider.
I'm not advocating for a single provider, and I'm not necessarily advocating for cloud hosting as a solution, I'm just pointing out that in this case the cloud fared better than practically all of the self-hosted systems
It's not the same OWA that one hosts on-premises. That one's still vulnerable even if it's hosted "in the cloud".
On a different note, if they could prevent this in "the cloud version", why couldn't they -- why didn't they -- prevent it in "the non-cloud version"?
Even if we assume that they did create two independent systems, there is no reason to assume that two products developed by the same company in tandem serving the same fundamental, lucrative use case should have material differences in quality/process. That there were multiple trivially exploitable, catastrophically effective vulnerabilities that were unknown for 8 years and that Microsoft never discovered themselves (they discovered it by realizing somebody else discovered it and was using it) should indicate that their cloud product is equally atrocious even if we assume that these were distinct products and thus would not be affected by the exact same bugs.
In conclusion, as you say virtually every computing system out there is a house of cards, so there is no reason to assume that consolidating on the cloud and letting one of those groups of people focus will result in anything other than more houses of cards, except in this case being used on an even juicier target.
The point is not if the Cloud can defend against a very sophisticated attack, the point is whether they can at least do a better job than what those big companies are doing.
And the answer is really easy: Fortune 500 are at the Stone Age of security (among a lot of other computer science topics) so of course the Cloud is doing better. It’s not even the same world or the same order of magnitude.
And the abyss will become bigger and bigger because it’s becoming more complex. There is no way a Fortune 500 company can keep up with the complexity of what AWS, Google or Azure is dealing with, and the new tech world we live in. And it’s also quite stupid, that’s not your job nor where you will be making money. Just concentrate on the app/code that is indeed your core job, on top of solid and proven Cloud services.
Also, you talk about centralisation and the issue of a single provider, well, here’s the actual joke: the level of centralization and concentration is way, way bigger internally than if it was on the Cloud. Most of those Fortune 500 companies have only a few datacenters. Although they are international, some even have datacenters only in their local region of origin, with zero region/local hub of some sort, as crazy as it may sound.
And most of those Fortune 500 companies have only one provider for each of their key component.
If they were on the Cloud (and they will be, eventually), reversibility and transferability is « built-in » almost, because it is an actual feature, or because everything is way more standardized, or just because moving into the Cloud, you will think from the start about how to move back or to a different provider. And in any case is much much better than the state there’s in.
Obviously, no engineer can have even a sufficient overview of the full Exchange Server implementation not speaking of full understanding. In such a situation security, quality and user (or admin for that matter) experience always take a big hit. It doesn't help Exchange Server is most likely developed using programming languages and approaches that more or less demand complecting the solution with OOP-related ceremony. Supporting two decades or more of legacy features and protocols doesn't help. Some companies even want to connect AD and Exchange to SharePoint... which is at least as complex as Exchange.
The problem companies don't understand is that you have to work on simplifying, which is very hard - much harder than adding features. If you don't, the interactions between components will overwhelm even the largest and best skilled team on the planet. The result is, we see breaches and security issues like this every day and realistically, nobody who can decide anything in the corporate environment gives a f** anymore because nobody pays the more or less laughable fines with their own money and nobody really goes to jail but the user data is lost, peoples lives are shattered.
Certainly, "in the upcoming version" is a bit late for those affected and most of those other Exchange-related hacks in the past. The thinking around Exchange is still more or less left in the 20th century and it shows.
This statement certainly doesn't help the credibility of your comment.
There's a reason why everyone uses microsoft exchange, despite all its myriad of flaws, and the flaws of its major client Outlook.
And it's because it offers so much functionality, precisely because it so much more complicated.
It's like saying you can secure your house if you build a 20ft wall round it with no gate.
Sure you can, but it becomes pretty useless.
Like the majority of awful “enterprise” products on the market, the primary reason that it’s popular is because it’s from a megacorp who speaks the language of the buyers, who are all aspiring megacorps. I was horrified the first time I used exchange and couldn’t wait to change providers the moment I had the chance.
So it’s more like saying you can secure your house if you use a security service who sets security targets instead of sales targets.
Sometimes being the least worst option is all it takes.
I call maximum shenanigans on this. Exchange is a fully-integrated groupware suite with a single-pane-of-glass on both the management and the user side. I am aware of precisely zero feature-complete alternatives, let alone anything "better".
So I’m sure that for some huge enterprises, the complete feature set from Exchange is actually necessary, or at least desirable. But for everyone else - including many companies I’ve worked in at a senior level, and almost certainly many of the victims of this vulnerability - I’d call shenanigans right back at you.
And you are right, loose coupling does rule out a very small set of functionality. For example an email sent to a user might have an smb: link, and then Outlook used to do a preview of the email, automatically loading all the links, which would cause your credentials to be sent to the smb:// server just by previewing the email, thereby allowing malicious attacker to steal password hashes by sending emails to victims (no click was needed).
So that would be an example of excessively tight integration and a design philosophy that was fast and loose with shipping both credentials and executables across the network. I think we have learned from those lessons.
In terms of why it is dominant today, it is because of fairly rational C level decisions, not users clamoring for it as opposed to some generic email/calendaring solution. Microsoft still knows how to do support, there is a large pool of cheap IT admins certified to work on it, and it allows you to run your own server instead of buying a service from gsuite. Really if Google could shed their disdain for human beings and learn to think of them as customers, they could take a lot of market share away from Exchange, because right now it is a trade off of security versus support - the functionality is basically the same.
Gsuite email doesn't even have good support for things like delegated access to shared mailboxes, treating them more like a distribution group. On Outlook they appear by magic on your sidebar.
Source: I am currently migrating some acquired users from gsuite to 365
You conflate functionality and complexity. If you think about it for a minute, complexity actually hinders functionality. There is some intrinsic minimal complexity to useful features of a software system for it to be functional. Exchange could be way more useful, if it wasn't so complicated and it could be a lot easier to keep somewhat secure.
Exchange in many circumstances feels more like a banks vault but instead of steel door with a wooden one with the cheapest padlock you can buy and a sign "we go here once a year to check everything is in order" where real banks usually work a bit differently... There are many cases, where an attacker gained access to the complete Active Directory through Exchange. At least so I was told by a company that did the consulting afterwards to clean up the mess.
Of course a server process which is designed to modify (among other things) group memberships needs different permissions than a user, why would that not be the case...?
If you don't like it being highly privileged, don't grant it the permissions. Or hire someone who can.
What on Earth OOP has to do with the quality / security of the Exchange? This reads like someone is on crusade.
You should really watch "Simple Made Easy" by Rich Hickey and think really hard about it. If you don't come to the conclusion that most software development could be way more sustainable in the long run would we use simpler tools and approaches instead of complecting everything especially with questionable OOP balast then maybe we have very different experiences.
I see nothing wrong with OOP. It is convenient for many things. It is not a silver bullet though. Nothing is. Personally I do not adhere to any concept / programming paradigm. They're just tools. I use many. Depends on what I am doing.
Generally one can take a tool and put it to good use while the other will fuck things up regardless.
It's a little easier to have foobar.update(), rather than update(foobar, state).
I started off mostly programming in R, so using mostly bare functions, but I have to admit that objects are really, really useful when you need to maintain state. Yes, you can do it with closures, but it's a little harder and a little uglier.
That being said, the mutability that makes objects useful is also problematic in that you can end up with magically updating references without defensive copying.
I don't know R and I don't really want to know it. For me, it doesn't seem to bring anything extra to the table that I couldn't do in Clojure or ClojureScript much more consistently and simply. If in my project, I have a number of transformation functions for my state, passing it around isn't a huge deal as it is just a nested map usually. It forces you to be very consistent and helps you as the project grows. Also, most of the functions are easily transferable between projects even when the state would have a very different structure.
Of course the whole thing is a complicated topic and in some cases you want mutability and local state e.g. because the performance is a bit better. Usually, that involves a few simple transformations.
And if you figure out how to fit a generalised additive model in clojure with one line of code, please let me know :)
So, in DS/stats you end up needing mutability because the datasets are really large, and the models take a long time to run, so copying is generally bad.
Convincing a developper to add features rather than remove requirement when the feature has no simple implementation in view :D
Actually, you want to work in a setting where you understand the need and value of a feature and how it fits into the overall design and feel of the (software/ hardware) solution to a problem. Is such a case, you understand that there is no requirement but a need or pain point that needs to be addressed, if you want to deliver more value to the user, some of which may turn into financial or other benefit for you.
Until you've personally experienced the full horror of attempting to keep on-premises Exchange patched, especially in the SME space where you may have few servers, it's hard to imagine how awful this is.
Cumulative Updates are essentially "completely uninstall Exchange" and then "reinstall Exchange again". This is not what one might call a "patch". Then you get into dependencies on .Net and suddenly you need to upgrade the OS as well while you're in the middle of completely-uninstalling-and-reinstalling-Exchange.
Last time I got sucked into this, I told my client it was nuts to run on-premises Exchange, to bin it completely and move to a cloud-hosted [Linux] IMAP mailbox system.
I wouldn't put out any new on-prem Exchange today, but the ones I support have reasons to be on-prem or planned migration off-prem.
Aside: I've been administering Exchange since version 4.0. I've never experienced "horrors" like so many people talk about. Failing to follow best practices, using dodgy hardware, and cutting corners are the reasons for problems that I've been privy to by way of friends, emergency engagements with non-Customers, etc.
I'm sure there are some SMEs who are happy to throw serious budget at doing on-prem Exchange "right".
For everyone else, I'm not sure what they're supposed to do.
I don't buy the "Exchange is expensive to support" argument. It's cheaper on-prem than paying for the subscription. We always saw break-even at around 16 - 20 months.
I have billing records for a small business Customer w/ a single Exchange 2016 server for last year that amount to 6.5 hours for the entire year, including installing CU's 16 thru 18 (CU 19 fell in this year). Yes-- a piece of their overall Windows Update application budget applies to Exchange, as does the amortized cost of backup software, and server computer and support hardware. Even w/ the OS license, Exchange license, and CALs at 120x an Office 365 E3 monthly subscription they're still money ahead over the 4+ years they've been running Exchange.
Moving to subscriptions results in a net increase in spend for organizations that were executing on-prem IT well and frugally. That's the only game now. I just think it's disingenuous to say that it's a cost savings. I reject the massive availability increase argument too, at least in the US, because of the lack of competition in the ISP space and the tier of service that is available to SMEs in their budget.
You spend more for the same stuff, are forced to "upgrade" (read: lose features, see changes in UI) at the whim of a third party, and may experiece decreased availability if you're unwilling to spend more on Internet connectivity. There "upsides" for sure, but too many people peddling hosted solutions fail to recognize downsides.
365 is a really good value, even comparing it to running an large scale standalone environment. Ditto for Google Workplace. For almost any other product, I subscriptions always drive more cost than value.
The
Some people disabled /ECP facing the Internet. It was "unsupported" by MSFT so I never did that. In retrospect it would have been worth the gamble. If I had it to do over again I would have taken that bet.
None of the compromised boxes I saw this week showed signs of post-exploit activity. They dropped their payload and left. Every compromised box was restored from backup, temporarily isolated from the Internet, and patched.
It was more beastly back to run back then though. We did reduce our risk profile at the time by putting OWA behind a sslvpn and only allowing BlackBerry.
But most Exchange management I do is mailbox management, and you have to do that if it's in the cloud too.
What did they reply?
Why? I don't see moving to a cloud solution being much better. The cloud service itself would be the single point of failure and would be just as vulnerable to a zero day. The organization would have even fewer risk mitigation options like NAT, firewalls, etc.