Oakland declares state of emergency due to ransomware attack | Hacker News Reader

Oakland declares state of emergency due to ransomware attack

nbcbayarea.com

103 pointsparrot98781 comments

It's been quite a few years since I did this kind of stuff for a living, so this may be an antiquated notion...

"In my day," desktop computers saved their files to a server. That server would get backed up daily. The backup tapes/drives would be stored offline and rotated to an offsite location. (Back then you were more concerned about the building burning down than a ransomware attack.) The same would be true for any apps running on servers; their data/databases would be backed up daily and the tapes/drives used for backup would be stored elsewhere.

What is this old guy missing? If a process like this were in place, nearly all of their data would be intact. Yes, it will take some time to do a full restore and you will be missing some amount of data that was created since the last backup. But it's survivable in many cases. And you're not negotiating with criminals.

The big change is that many places now send data to an offsite location (or cloud!) through a network instead of physically moving tapes, and the attacker can often use the same network connection to destroy backups.

I think most backup providers (e.g. rsync.net) allow and encourage read-only backups.

The bigger issue is that nowadays organizations have lots of interdependent systems, and if you seize the data of one, you basically cripple the entire organization. So for each system you need to institutionally require both backups and backup testing procedures, which is easier said than done.

The article does not say anything about Oakland negotiating. They may just be in the "it takes some time" phase at the moment. Tapes are not exactly the fastest medium.

Plus, you may want to determine the exact time at which you were compromised, or else you'll be restoring potentially tainted backups. Depending on how well you're organized that alone will take quite some time, especially considering that your logs may be encrypted as well. Sometimes you don't even know how to contact everyone, because your comms are down, too.

Sure, if you do everything right and adhere to all the best practices, it won't be that big of an issue. Just don't forget about the amount of legacy crap and budget constraints many orgs have to deal with. That comes with many pitfalls and a lot of opportunities to make a mistake.

This right here, the company I work for did some back of the napkin calculations for restoration times for archival storage (tape) and determined that it’d take us 2 days to restore minimum and up to 7 days to restore if the queues were full. This was with Azure and a relatively small amount of data (around 6TB).

We’re using their immutable storage option, with a 60 day window with multiple rotation intervals, and just biting the bullet on the cost of cold storage vs archival because of how slow tape is.

I could definitely see a larger entity having significantly more data and the restoration process can’t even start until they finish triage. No point in restoring until you know the source of the intrusion or at least have a plan to prevent it from recurring.

For regular backups of 6TB why would you even consider tape? Seriously asking because I have priced tapes and the break even point vs hard drives is high.

Because we’re not paying for the tapes, Microsoft is.

Their archival storage tier is extremely cheap, but has a high retrieval cost and a very slow retrieval time. If you wanted to keep 1+ year full and incremental backups of a large amount of data it’d be a no brained. If you’re only keeping 60 day full and incremental than the retrieval cost and time doesn’t pan out cost wise IMO.

You also have organizations that have certain retention periods - say for example, keep all data for 6 months.

If your ransomware stays resident in your systems for 6 months, any backup you recover from ends up being infected and can potentially be considered useless to restore from unless you're very careful in how and what you restore from.

A lot of organizations also don't have the money or processes in place to manage backups. It's a huge cost outlay and in cash strapped SLGs, it simply ain't happening - especially when any half decent talent can make way more money working remotely for companies that respect Engineering.

That sounds completely self-inflicted. What are they spending their money on? Not Oakland, but across the bridge, last I heard, 16 millions for a few tents [0].

[0] https://www.nbcbayarea.com/news/local/san-francisco-paying-1...

Debt. City of Oakland has has a budget deficit of around $100-200m a year for the past 10 years at least.

There is no longer any discipline around filesize and data. People regularly attach huge files and have terabytes of useless data in databases. This makes daily backups prohibitively expensive for most orgs.

This sort of stuff doesn’t surprise me any more. I’ve been on a number of “desktop support” sessions over the last few years and seen some shit. The common denominator seems to be entirely unpatched obsolete stuff (stock RTM windows 7 with stock IE in 2021 was my favourite) where either someone turned the updates off because they knew better or stopped paying their MSP for service immediately after they had been set up and assumed it’d just work forever.

People like that and the associated competence level are rolling out the red carpet.

If its really important. Airgap. Or VM-Wrapped with restore points.

I completely understand that somebody does not want to upgrade into the warp-abyss-abomination of modern windows, especially if huge expenses software was written once, that needs backwards compatability or contains sensitive data. You can not use windows if you work for anything with sensitive data.

In todays world the legacy is the good stuff. Just needs protection.

It doesn't look like Oakland would have the IT people, time and skills to deploy and maintain a VM-wrapped infrastructure - which has all the same issues with needing to keep it up to date; e.g. I know people for whom this VMWare ESXi attack https://www.crn.com/news/security/vmware-esxi-ransomware-att... managed to ransom-encrypt both their main virtualization environment and also the backup one.

An airgapped system is one that's basically unusable because you can't communicate with other systems.

Can't help but think back to my youth where nearly every system was airgapped, but were plenty usable regardless.

A system can be more than one computer, i.e. mainframe. Airgapped systems can include multiple computers that are disconnected from external networks. They can be very useful for specialized applications.

I witnessed a ransomware attack where somebody in operations had a SMB share on their desktop to the backend storage for the VMWare ESXi cluster. So the ransomware was able to encrypt many of the vdisks.

I love people that believe there exists a version of windows that could be deemed secure. I was there once. Install the latest update to fix the security problems. Don't worry, our software becomes 300mb larger due to 500 other security problems we are rolling out today, but we managed to close off this one tiny hole over here.

Why does it matter anyways. With both Intel and AMD running processors independent of your machine, there's really no way to keep anything secure unless you use a machine that's over 20 years old.

But, isn't that backwards? 20 year old systems have been thoroughly exploited and usually do not benefit from more recent updates. It's true you can't patch every single vulnerability, but probability is a huge factor in risk. If many of the common exploits have been patched, it's simply harder for your average hacker, the difficulty and opportunity cost just go up.

I love people that believe there exists a version of any operating system with C code on it, that can be deemed secure.

https://en.wikipedia.org/wiki/Morris_worm

It is true that C does not protect against a class of errors related to memory safety, but it disingenuous to imply writing an OS in any other language will make it secure. At best, it will only reduce the porosity of the attack surface.

One of the reasons why Multics had a better security assessment than UNIX from DoD, was precisely how PL/I does strings and arrays.

Not wearing seatbelts and helmets doesn't save everyone, so it is worthless to use them as a vain attempt to save human lives.

Microsoft have 122k employees. Assuming that every one of them takes the upgrade, it uses an extra 61TB of storage. I can buy 61TB of NVMe storage from a high street retailer for under $5000. It's less than half that for a normal SSD. It costs more than that for the electricity to install the updates to 120k people I would bet.

> there's really no way to keep anything secure unless you use a machine that's over 20 years old.

This is nonsense. Security isnt a binary thing, and even if it was, you're still vulnerable to wrench-ops. If your threat model is that you suspect your procedure manufacture have backdoored your CPU, you better be running your own fab, air gapping your machines, and desoldering input ports.

Meanwhile for probably 95% of people and businesses out there, keeping windows up to date, 2FA required, encryption in transit and at rest, and regular tested backups is enough.

At this point I don't think the goal is to reach a state of 'secure'; it's shifting vulnerabilities around to be less predictable. Intentionally or not.

Windows 1.0 was pretty secure by todays standards. /s

No networking, right? I guess it was pretty good.

Also no USB support, so even stuxnet won't work.

So I work in this space and I am honestly quite surprised by the users here who think a Linux deployment would do any better. They won't.

This isn't a Windows vs Linux vs Solaris vs BSD issue, this is a "did I manage and configure ACLs, RBAC, GPO, and other security features correctly" issue.

For example, I've had customers have had RHEL 6.x enviromments that still got hit because they wrote a security group that allows all traffic from all ports from 0.0.0.0/0 (aka everywhere).

Security issues always come down to misconfigurations and the lack of best practices in my experience. In that regard, the MS suite is actually superior to Linux because if you need a Security Solution Partner, Microsoft Professional Services is infinitely more competent than the largest Linux solution partner righ now (IBM).

I'm with you right up to the "infinitely more competent" line.

The big thing that Microsoft and Windows have against them, is the crapshow that is all that they include on a standard installation. That said, from what I'm seeing, this is not really unique to Windows anymore. Seems everyone wants everything on the machine.

So, yes, it is theoretically possible to setup all access rules correctly. But it is essentially a lines of code problem, at this point. Given a mountain of things to setup, you will make a mistake somewhere.

It's generally professional services that set up these deployments at scale. MS's PS team is extremely competent and does push best practices in my experience. The issue is organizations that cheap out and decide to have an IT Service Desk guy manage everything from deployment to network architecture to security - these are extremely hard problems that require a large team of SME, not a single guy doing the best he can. City of Oakland is one of those kinds of organizations IME.

The lesson here is that the City of Oakland and similar organizations shouldn't be deploying such systems at all. They should lay off most of their IT staff and outsource their entire IT infrastructure to one of the large vendors who has the resources and technical competence to deal with advanced persistent security threats. Blaming the OS vendor won't accomplish anything.

Microsoft is an MSSP as well. And they are one of the multiple vendors the City of Oakland uses. But vendors can only do so much for organizations like Oakland as the final decision and implementation ends up getting stuck in red tape and bureaucratic hell between multiple disjointed teams.

What I'm proposing is that cities should outsource their entire IT infrastructure to a single vendor who runs the whole environment, including security. City employees shouldn't have any authority in these issues beyond vendor selection. I understand that might be politically difficult but what is the alternative? It isn't reasonable to expect city employees to have the skills and resources to defend against advanced persistent threats.

There is still room for city employees and other vendors to exert some control over higher level IT services and applications. But the core infrastructure needs to be under the control of a single competent vendor.

IMHO the core issue is a lack of resources - they can't afford an outsourced vendor that will do stuff properly just as much as they can't afford to do the same thing in-house. There's barely a budget to get hardware, and definitely not to deploy it properly; there's barely a budget to replace what dies of old age, and definitely not to do proper maintenance and updates.

Then the Oakland city council should make hard choices and reduce discretionary expenditures in other areas. That's a shame and it will hurt underprivileged city residents who are already struggling, but they need to face reality. I hate that we essentially all have to pay a "tax" to protect against IT security threats but what is the alternative? There doesn't seem to be a cheaper option that actually works.

I expect that pricing has made it so most all smaller places are these kinds of organizations.

And the incentives are to keep it that way. As long as MS's PS team can make more money from one whale of a customer than they can supporting local districts, expect that this will remain.

Such that I don't think it is excusable to say "if only they had paid the professional services."

You don't need professional services all the time, assuming you make sure to hire a large enough team of SMEs of your own, but a lot organization simply view tech as a cost center and try to spend almost nothing on it. That is the kind of issue City of Oakland faces (for everything btw - for example, starting salary for teachers in OUSD is $50k compared to $70-80k in the rest of the Bay Area). Also City of Oakland does 100% engage with Microsoft PS, but they don't have the requisite staff and budget to finish building out best practices.

I clearly don't know all the specifics of their case. Such that, if you have closer information, I am not trying to gaslight you on what you know.

I am concerned with "best practices" in our industry, though. Too many of them are not geared to wide adherence and have fantastically bad failure cases.

All good mate! There are some specifics I don't (and don't think I can) get into, but big picture, a lot of the core best practices in configuration management and security do work, the issue is whether you are able to hire people who can actually implement and understand WHY those practices are in place and how to iterate if said practices don't work. Most organizations across the globe have barely gotten a handle of ACLs and Security Groups, but evangelizing best practices for Endpoint Security, Cloud Security, OT Security, etc will take another 10-20 years simply because of inertia and the common sentiment that IT is a cost center.

Random think tanks doing thought leadership on CNN or at Brookings will jack themselves off to the notion of "online warfare" and whatnot, but those guys can barely type, let alone write cohesive policy.

I've been on both sides of this - both in the policy making world and in the private sector tech world - and cases like Oakland keep happening in a daily basis everywhere and will keep happening forever.

> Such that I don't think it is excusable to say "if only they had paid the professional services."

Would you apply the same logic to road infrastructure? Why hire those licensed engineers...

The target of the question is to the vendor of the road, though? If I pay to get roads installed and they counter that I should have paid for a more expensive process? Yeah, I'd say that is inexcusable. If they have strong reason to think that who they are selling to cannot maintain things correctly, they should consider not selling.

What specifically does a modern Windows installation include that is inappropriate or insecure in terms of default services or access rules?

Just exposing SMB on port 445 by default is a huge issue. The vast majority of systems does not need to provide this service, yet there have been two catastrophic RCE vulnerabilities (MS08-067 and MS17-010) in this service. Also, it's basically like an SSH service for attackers to move laterally within the network. If it's not a file server, domain controller or print server, it should probably be turned off or at least severely restricted for a whitelist of hosts.

Next, the NTLMv2 authentication protocol is on by default and vulnerable to relay attacks and offline password guessing attacks. Plus: pass-the-hash vulnerable. Huge problem in corporate networks.

I'd argue the broadcast domain name resolution protocols like NBNS or mDNS are unsafe as well.

Disclaimer: if you were just talking about Windows on your home desktop PC, then yeah nevermind.

Certainly a fair question, but a big part of the problem is I don't know the specifics of what is on a new installation anymore. Worse, I'm not sure where to find such a list.

Just scanning on the things they are proud to list at https://learn.microsoft.com/en-us/windows/whats-new/windows-..., I'd be worried about Teams, Windows 365, and Widgets. I'd also be worried about all trial software that is on the machine. I could not find a list of that, though.

And again, this is not unique to Windows. It used to be OEM bloat that was added to all things. In linux land, it would have been all of the "power tools" included by default.

Do the popular Linux distributions for both desktop and server environments provide such a list? Have Teams, Windows 365, or Widgets been used as attack vectors against real systems?

Generally users in most enterprises are going to need instant messaging and online meeting tools, so if it's not Teams then it will be something else with an equivalent attack surface area. Windows 365 appears to be highly secure.

I'm not trying to defend Microsoft here. They have had many security flaws and there will be more to come. It's just not clear whether the alternatives are significantly better.

> Do the popular Linux distributions for both desktop and server environments provide such a list?

Yes, this is common. You are generally given the option at install how "minimal" you wish to go (do you even want a GUI installed, etc). These are often listed on the distributions website.

For example here is a few from Arch:

- Base (bare minimum) install: https://archlinux.org/packages/core/any/base/

- Base-devel (what you need to run makepkg): https://archlinux.org/packages/core/any/base-devel/

I already said this is not unique to Windows. Such that I don't see the point of the first question. On whether those have been source of attacks, I don't have numbers. I do remember getting hacked through Office back in the day. The whole "open this document" crap college students did to each other. To that end, I fully cede this could just be a false view from my side.

I'd expect most attacks are still of the "what is your password" variety. That along with a giant shared drive that everyone just dumps everything into.

And I don't mean this as an offensive against just Microsoft. They are/were somewhat unique in the success they had with embrace/extend. That said, the blame almost as surely rests on typical "growth at all costs" mental model that is modern business.

The emergency declaration will assist with equipment and materials and the activation of emergency workers as the city seeks to safely restore its systems.

It's important to remember that 'state of emergency' is less of a 'everybody stop and listen to this' than a legal circuit breaker that allows the signing of checks and assignment of tasks without being bound by the normal web of procedure and contractual obligation. We tend to imagine (in popular culture) the executive aspects of government as being somewhat by fiat, but much of the time it's more like incremental product development, with most of the job being workarounds, excuse-making, bullshitting, and tedious social obligations.

I don’t get why any user has the ability to cause so much damage. Sure they can lock their own files out and need to restore from backup, but how can that knock out other departments, let alone things like email.

When ransomware attacks began, it was more typical to see the blast radius centered around a single user who did something stupid, like run an exe or enable macros.

But that’s not how it’s done on these large enterprise networks. Ransomware gangs will still use single user entry points, but the hackers will work quietly inside the network to escalate privileges and determine key servers that should be targeted first.

I'm a penetrations tester. When the client gives me a windows laptop with low privilege credentials, I'm typically domain admin by lunch time. Sometimes even before I finish my first cup of coffee. As a domain admin I could encrypt almost any computer, often including the backups.

Privilege escalation in Windows Active directory domains is really easy. Securing a large corporate network is really hard. Especially on a tight budget.

It's not any user, it's a ransomware attack. So it was intentionally done to limit their ability to work. Also, don't assume they had backups, or that these backups weren't also targeted.

Security is expensive.

surely less expensive than the fallout from this

Prevention is orders of magnitude less expensive than dealing with the fallout from an eventually inevitable atack.

The tragedy is that in the absence of attacks, local governments don't always allocate the necessary funds to employing competent admins who take a proactive approach to security.

Even more importantly, these admins need to be given authority to block attempts at lowering defenses in the name of convenience or "money-saving".

The problem is that lowering security expenditure is a good gamble for managers/executives: Chances are it will take a while before things blow-up.

In the meantime, you get the credit for "saving money", you will get promoted, perhaps move to another company, and the bomb will explode in the hands of your successor.

Here the solution is personal liability, including CEO and board of directors.

True, but not always. Also, until something happens nobody would approve budget anyway. Exceptions from this rule are rare.

Depends, but usually the problem is that it is difficult to properly assess the probability of a successful attack and to get decision makers to believe that number.

Are there no agencies that can help out? CISA is, I guess, more of an advisory agency than operative? Or maybe there are but on federal level?

At this point it's too late, and before that they didn't really need advice or some fancy technology, they needed to dedicate enough resources/people/effort to simply do proper maintenance of their IT infrastructure. It's also plausible they simply couldn't afford the required resources, but that's not something fixable by CISA or other federal agencies.

It’s really easy to cut back on your IT infrastructure until stuff like this happens, and suddenly everyone is up in arms about why something isn’t working.

But it makes great budget headlines, “I slashed the IT budget in half!”

Hardly anyone is interested in defensive security because if you do it well your job looks unnecessary. This goes both at the national security level and the individual organisation.

When an extremely high profile attack like this happens, CISA ends up taking over the organization and revamping the entire organization's IT team. This happened to Atlanta back in 2018-19. It doesn't mitigate the current incident, but helps prevent the next one.

I don’t think there’s much to be done retroactively. I’m sure there’s an option for proactive help (trainings, advice) but it is a big country, some attacks will slip through.

What crypto are Ransomware asking for these days? After all the Bitcoin mixers seem to be taken offline (have they?). Sorry, I'm kinda out of the loop and was wondering how these thugs were cashing their attacks.

Monero https://cointelegraph.com/news/monero-crypto-of-choice-as-ra...

Thanks! That is from 2021; is that still the case?

new mixers come around. there where some news about one called sinbad for btc recently which is being used to launder money by the NK hackers

You don't need a mixer. There are plenty of ways to sell large amounts of BTC 'over the counter' in other countries.

SEC says that Do Kwon sold 10000 BTC to a Swiss Bank.

https://news.bloomberglaw.com/securities-law/do-kwon-tapped-...

imho we have to look at what limited set of tools and functionality we really use. The days where we didn't know what computers were used for are long gone and the justification for doing everything in software along with it. You want to exchange strings of text with video and images. Not much more than morse code offered. Direction of dataflow can be easily enforced in hardware. The backup drive takes input that you can't read, you break off part of the print and it becomes read only permanently. It can easily be made an insane amount of work to regain write ability.

A completely finished os can be stored on a read only device.

We just have to start from scratch :) that is all it takes :)

> A completely finished os can be stored on a read only device.

ChromeOS has entered the chat

Seriously, if it's good enough for school children, it surely is good enough for government. I love my Chromebook, and while I cannot yet do my day-job on it, I did interview at a crypto company that did do their day jobs on it, so I believe it's possible

In the modern threat environment it's no longer viable for small and medium enterprises to maintain their own IT infrastructure. This includes city governments. They should outsource infrastructure to one of the major cloud vendors with the scale and technical competence necessary to counter advanced persistent threats. It's a shame that we all have to pay this "tax" and give more control to a few big tech companies, but that is our reality.

This doesn't help. You still need people to configure the group policies and firewalls. You will also need a local installation on various PCs running on-prem to connect a lot of hardware.

You might get away with Azure AD instead of a local domain controller and exchange but you won't get much farther than that. And if there isn't a backup strategy in place already, this won't change with cloud.

They already have for at least 15 years.

https://www.oaklandca.gov/departments/information-technology...

Are they ever going to hold the leadership accountable for sleeping on the job ?

how many of these systems will be safe if they had linux running? just saying because the linux is a smaller target and it would be a long time till it reaches the "year of linux desktop"

The same amount as would be if they were using MacOS or Windows 11. This isn't an OS issue, this is a "I didn't manage and configure my ACLs and RBAC correctly to minimize lateral movement in my environment" problem. Linux isn't anymore secure than Windows in that regard, as can be seen with ransomware such as Elbie. I can also say with extremely high confidence that in a number of orgs that are ransomware victim are running Linux seployments for their servers (usually Centos 6.x-7.x or RHEL6-7)

Reply on news.ycombinator.com