Patching is hard; so what? | Hacker News Reader

Patching is hard; so what?

blog.cryptographyengineering.com

120 pointsrunesoerensen61 comments

> I don’t operate production systems, but I have helped to design a couple of them. So I understand something about the assumptions you make when building them.

Start by operating production systems. You will rapidly discover that patching is not a technical issue (as I already said last time there was a patching discussion on HN). Patching is technically "easy".

But if the business does not prioritize (and allocate time towards) patching (and testing that systems work with after) patching won't get done. There are features to develop and deadlines to meet and new systems to turn on and old systems to retire and oh-my-god don't touch those servers otherwise the vendor will not support the prod environment anymore (no longer certified, yay!) and it costs us 30000$ to get certified again.

Too many times I see HN assuming that everyone is running on a public cloud with B/G deployments and immutable infrastructure you can rebuild and redeploy easily. Unfortunately that's not the case.

Those "don't touch" systems are death. It's usually in contracts with the big players like Honeywell and others. You literally can't touch them without breaking the contract. The second you call for support and they find a patch - you have to uninstall before they continue - even if it's unrelated.

BTW hands up if that includes WannnaCrypt. Yep.

This is the same situation as those MRI machines with XP on them. The answer is don't buy those machines or don't connect machines you can't patch to the network. If hospitals can't buy them because those are the criteria, then industry will change.

Don't sign contracts that leaves you in the situation that you can't patch machines.

If a hospital doesn't have an MRI machine, it won't be able to compete. It's really not an option to say "I won't buy it unless X." You buy the one that meets most of your needs for the best price. Not buying is not really a choice.

I am responsible for about 200,000 production systems, about 90/10 client to server.

Delivery is not difficult. My teams can deliver anything to 95% of those systems in <24 hours. The hard part is testing, validation and coordinating rollouts.

It's also more effective to understand and have mitigating controls. Many java related exploits cannot be patched with off the shelf software -- you need to wait for the vendor. So you to segment or monitor to live at risk.

Companies who are good at this have good asset management, good process and good vulnerability monitoring.

Good fonts?

Thanks for catching that quickly! I don't remember precisely what word i intended to write... but autocorrect butchered it!

I suppose adding "quality control" to my original post would be appropriate as well! :)

> with B/G deployments

Why do we call it that now, instead of A/B deployment like we used to?

To me, blue/green is for infrastructure, and A/B is for the user side of it. That is, you should be able to do blue/green with the content not changing, but when you're A/B testing, you're collecting metrics and other usage data about the users to make future product decisions. You can also do A/B testing without having a B/G setup, using things like user gates. This distinction may be incorrect, but that's how I've used them.

A/B implies certain symmetry, while B/G is highly asymmetrical: G is known good, battle-tested, and B is new and suspicious.

Patching at an OS level (not what the author is talking about, probably) is extremely extremely hard - from an enterprise Microsoft platform production point of view

1) Because there's no budget for tools. It's mostly WSUS and SCCM if you're lucky.

2) Because staff don't even know how to use those tools properly. Can't blame them because there's very poor documentation from MS and very little accessible training on design patterns that integrate them into large corps.

3) It's outsourced. So it's even worse than above.

4) Because management fights it every step of the way. They hate patching because they're stuck in the 80s where they think every patch is deeply documented and you should only apply what's necessary. And you should only apply security - ignoring updates which give critical stability fixes.

5) Because the enterprise as a whole is so poorly documented it's unclear who owns what or if a rollback is possible and what it would affect. So everything is in CYA overdrive.

6) And then nobody working there wants to own it because it's not sexy to patch. (As a geek I think patching is sexy and very fulfilling work, but I'm in the extreme minority).

None of this excuses anything. But most largish companies are still in the 80s, with IT managers in the 80s. I can't wait for them all to die.

For teams that use a DevOps model, fast, predictable deploys that can be safely rolled back are important for security, for this reason.

If deploys are like playing Jenga on a sailboat, you're not going to be able to patch fast or safely.

That said, even becoming aware a CVE exists in the first place is still a problem for many teams. There are plenty of good options, it's just underinvested in early on.

Even if they learn of a CVE there's no guarantee a patch exists for it. You might even be stuck with no alternative but to disable whatever the attack vector is... which could be key to your business.

Exactly. I'd say the check of how fast can a security patch be deployed from the moment upstream OS or library releases is a good benchmark for a team/system and should be evaluated at the same level as log audits, what ports are open and what openssl versions are running.

> it's just underinvested in early on.

And for cases where there is no personal or financial data involved it is shameful and just a bad practice to not have the system which allow fast security patch deployed. For a system that holds sensitive personal information for hundreds of millions of people it should a very serious issue not to patch the system for months after vulnerability was known.

Deploying patches is easy, validating that they aren't going to break anything (sometimes in hard to detect ways) is hard.

It isn't necessary DevOps or system admin not capable. You've vendor software not up-to-date that is beyond your control, and often users do not know until vendor sends a notification or a hack headline comes online then "oh fuck" moment.

Just to be fair too: I have seen a number of enterprises run their "legacy" infrastructure (mostly refereing to on-premise servers and old applications) doing a good job keeping up with updates such as Microsoft and whatever Linux flavor they use. But the new system/DevOps havr trouble getting through the gate, mostly IMO because we adding too many layers to the stack.

Your first paragraph, do you find any of that a valid excuse for a company like Equifax, who handle so much sensitive data? _I_ consider them responsible for the entire chain. I understand it's hard, makes no difference with those stakes.

No, not at all. My point was DevOps is not a magical group of wizards, and often DevOps team can be unproductive compared to whatever enterprise IT operation team/client service team. If DevOps is unable to carry out its core values, yes, entite chain is to be critized.

Companies like Equifax are fundamentally timebombs. The problem there isn't an IT problem.

Back in the 90s people used to have their pager connected to CERT to make sure they knew if anything happened. Of course no one uses pagers anymore, but there are places you should definitely pay attention to if you're looking at the software.

I get SMS for CERT.

Patching isn't that hard. Once Equifax finally noticed the intrusion (intrusion detection system?), they took the system down and patched it within a day.

I'm working with a fifteen year old app that's slowly modernizing. There's two types of updates, the normal kind that's slow, and "everything is on fire, get the fix into production ASAP, then pray". The latter kind is super quick, and it's what Equifax did here. But that doesn't mean patching is easy for them. It means patching without doing whatever laborious QA/manual testing they normally do is easy.

(Btw: it might sound like I'm defending Equifax. I'm not. I fully agree with the article. They have a responsibility to figure out how to secure their systems. "We can't patch quickly" may be true for them, but they needed to figure out how to change that).

Yes. You've hit the nail on the head. They'll have such a tight bureaucracy on patching that it's impossible (which is an IT problem in their eyes) but the second a senior manager says just do it - its quick and painless.

I see this over and over again everywhere.

It's quick, but it's risky, because testing is manual, you don't have blue-green deploys, etc. So it's not that it becomes painless, it's just that their are circumstances where you accept the risk.

> It means patching without doing whatever laborious QA/manual testing they normally do is easy.

Everyone learns how to do Continuous Deployment when prod is on fire.

If you're not managing the risk of upgrades, you're arguably not really doing Continuous Deployment.

I know equifax is probably dealing with a hairy legacy system, but really you should design all your systems to be able to push out a fully tested blue/green deploy in hours not months. There's no excuse not to with Jenkins/aws/docker/etc.

You should be testing and pushing updates regularly as a matter of course, not as a red alert only when security vulnerabilities are published.

That's not the mindset of enterprise IT though. The mindset of enterprise IT is "if it works, don't touch it." This mindset developed because of decades of dealing with arcane, complicated software that was at the same time critical to business operations. I'm aware of a major university that still has a three-week "change freeze" at the start of every semester because of this attiude and I'm sure they are not the only one.

I've worked in enterprise security teams for the last decade. This is spot on. It's retarded and will never change because there is no accountability. Corporate IT don't give no fucks.

Then let's hope Equifax shakes the foundation of their egos and they start to give fucks about this sort of thing.

Corporate IT is doing what software developers have taught them through experience and pavlovian condition:

Don't touch anything, because it will break if you touch it.

IFF people who release software learn how to release security patches and bugfixes that don't include new features or break existing parts (except when breaking changes are actually needed), then, after several years of good results from pushing changes, Corporate IT will be conditioned to just push security fixes, it's not a big deal, plus it's important.

Right. Which is why ci/cd and devops, etc were invented. If you're not patching and updating frequently, you're committing business malpractice.

Three weeks freeze twice a year during the most critical time does not sound that horrible to me.

Software developers tend to break system with changes. That is not some kind of mythical rare issue - that happens often (with some vendors very often).

Agreed. A company that doesn't have change freezes at the most critical times has very poor practices.

The "change takes time" issue does not happen because of technical problems. It happens because of organizational problems. Technical problems with patching are easiest to solve.

Side note: fully tested within hours assumes small stable software. Large software with frequent changes into it needs to be manually tested. If you have microservices, you still have to test whether something important did not broke.

> Specifically, these folks point out that patching is hard. The gist of these points is that you can’t expect a major corporation to rapidly deploy something as complex

Then they should be taken to court and be forced to pay for it. It's like saying about a company which runs trains and didn't do maintenance which caused an accident to happen "Well, you see maintaining the brakes on the trains is too hard so don't blame them too much...".

> The gist of these points is that you can’t expect a major corporation to rapidly deploy something as complex as a major framework patch across their production systems.

So those corporations will be hacked, and then they should be dragged to court, made to pay, some might even go bankrupt and disappear altogether. Eventually only the companies that manage to build better infrastructure will survive.

Oh well, one can dream...

The fact that Equifax reported the breach at all represents a good faith understanding: they tell the world, and in return the world does not make them cease to exist.

If you kill companies over security incidents, then magically you will no longer hear about security incidents.

They aren't doing us a favor that we have to reciprocate by reporting a breach.. its more of a responsibility. If its not required by law already, it should be.

It's not a good faith gesture, it's required by law almost everywhere they operate that they notify of a security breach of PII.

http://www.ncsl.org/research/telecommunications-and-informat...

And yet, as if by magic, you will no longer hear about security incidents if you kill companies.

If there is no penalty that hurts shareholders and executives breaches aren't going to stop.

There exists some space between no penalty and a penalty that effectively kills the company.

Besides, there exists the level of mismanagement which deserves to tank the company.

Investment in security costs money. Ignoring it saves money and gives company advantage against competitors who invested into it - untill breach happens. If we then effectively ok breach, we practically ensured that only bad security companies will remain living in the long term.

Taleb called this "picking up pennies in front of a steamroller". You make a little money every day, until you get killed. Lots of management consultants specialize in recommending this stupid practice, because corporate executives can't get enough of it. It doesn't seem likely we'll see a law that effectively punishes firms for this behavior in the PII security context, because the same behavior in other contexts led not to failing firms but rather to "too big to fail" and bailouts.

Penalizing shareholders and executives to the extent that reporting breaches becomes enterprise suicide is only going to make breaches unannounced and prevent the collective from learning from individual mistakes, so therefore more common.

The way to reduce breaches is to encourage open admission of security failures, so that preventative knowledge spreads quickly. It is worth some positive amount of respect to follow through in one's obligation to warn one's dependents, and moreso if one tries to warn society as a whole.

> so that preventative knowledge spreads quickly

It doesn't need to spread. It's not original research. People do this every day in other places. This shouldn't be talked about as some black magic skills.

It may be tempting to believe this, but evidence would contradict you.

If security best practices weren't made irrelevant by ignorance in the first place then people wouldn't cause these preventable mishaps in the first place.

It's not rigorous enough to say "I am certain some people can RTFM and understand without resorting to glorified mimicry, therefore the benefit to the group of encouraging people to share mistakes is marginal".

There is just too much at stake in the outcomes of security breaches. It is far far better to be have a high variance of variance of protective + relayed benefit gained from analyzing someone else's failure, than it is to overestimate your certainty of understanding while at the same time underestimating the lessons of someone else's failures.

Security is too important to put the focus on people learning in rigorous and logical ways over the protection of users. That said, due to the informational nature of security, to protect users necessarily contains as a sub-goal wholly understanding what your security is. But it's disingenuous at best to say that incident disclosure isn't actually empirically sound but instead gives security the appearance of black magic.

That is the air crash investigation model, which has as a primary goal not assigning blame but understanding causes and what can be done differently in the future.

But in many cases it boils down to "pilot error" and in those cases the pilots (if they survived) see their careers severely limited if they even fly at all anymore.

How about this: You have a vulnerability and you disclose, your company ceases to exist.

You have a vulnerability and you fail to disclose, your company ceases to exist and all your C-level execs go to jail.

I know people who work with trains. They have truly awesome mechanical systems but cobbled together to IT systems they weren't qualified to program and as awful as you'd imagine and more - undocumented and unpatched.

On the other hand management takes a complete 180 view to their stuff because a train derailment is extremely serious from a profit point of view, costing into the millions, and not so rare that they've never seen one.

It ends up in a real weird jumble of still not acknowledging or patching anything but also being super serious about everything else like the hardware and train maintenance.

I guess it's easy to point at a failed break and say that's the cause but WannaCrypt didn't cause any trains to derail.

TL;DR: patching is hard, so use defense in depth to minimize/mitigate the impact of individual components becoming insecure from time to time.

The whole problem boils down to a choice :security or availability.

If you prefer to have systems 100% up (no downtime for patches and fixing potential issues after the patch) then it is fine. The actual risk (described beyond "having security") has to be accepted by the owner of the data / service and the problem is over. I have yet to see a CFO who will accept in written that his systems are prone to hacking, theft, manipulation.

Until the CISO does not force the company to sign off such risks, he or she will be the one who fucked up.

Is it so hard though? Do Facebook, Google etc. have this problem too? I doubt it.

Systems need to be designed to be patched regularly from the outset. Yes, defense in depth can help, but it's not defense in depth if you one of the layers is pretty much always broken.

If they are relying on legacy systems that can't be proactively and regularly be patched, they shouldn't be holding that data.

I think part of the problem here, is that we can't see the risks companies are running with our data and goverment doesn't regulate it.

P.S. I do manage systems, though not on this scale.

> Is it so hard though? Do Facebook, Google etc. have this problem too? I doubt it.

Companies with monopolistic pricing power like Facebook and Google are a poor comparison for almost any other company. Many problems become very easy when you have a near infinite supply of money. Other businesses need to work with much more difficult constraints.

Average IT spend runs between 2 and 5% of revenue for fortune 500s. For reference... GM makes 166 billion/year in revenue, and on the low end their IT spend is probably more than 2 billion/year. In 2004 they were making the news because they went from 4B/year in 1998 to 3B/year. In 2006 they were making 190B/year, so I doubt their IT budget has changed significantly and is on the low end compared to other companies in the fortune 500 list.

Facebook and other technology companies might spend a lot more on IT, but that's part of them doing business.

Don't confuse the fact that technology companies do technology better with the fact that any company in the fortune 500 list have HUGE IT budgets. At a large company that I used to work for we had an IT budget of more than 1 billion USD and they didn't patch very well either. Our budget was actually smaller than many of our peers.

If you have a IT infrastructure/application budget of 1B or more and you can't patch, your doing it wrong and there is no excuse. For a full baked/managed solution, you can find folks to do an entire program for less than 12m/year for a 200k node network.

It's pretty simple to patch if you designed the system to be patchable in the first place and assign enormous cost to not patching. If you assume running an outdated version of struts for 3months is a $1B expense then you'll patch even if it means disruption to business operations, or will breach some conract.

And if the system is properly designed then anything can be rolled back if it turns out there were unforeseen consequences.

Maybe patching is hard. But the real question is why one vuln in their web framework allowed unfettered access to basically all of their data. There are much deeper problems here than slow patching.

the real question is why one vuln in their web framework allowed unfettered access to basically all of their data

The hack involved "primarily names, Social Security numbers, birth dates, addresses and, in some instances, driver's license numbers", that's a very limited subset of the data they hold on people.

I'm not defending them, but this was not "basically all of their data".

"I don’t dispute this point. It’s absolutely valid."

I do dispute it. Patching is not hard. It's incredibly easy to the point that some places and projects have the process automated from notification to application to testing for breakage to deployment. For reducing downtime, one can use clusters with rolling deployments of patches. Patching is only hard for things such as web applications when the company's application or processes are done in a way that makes patching hard. As in, they have to be incompetent or just not care about IT. Far as competence, your claim about fragile systems is a good example that might have happened.

" then you’ve implicitly made the decision that you’re never ever going to allow those vulnerabilities to fester."

You're thinking like an engineer that cares about quality. You instead should think like an Equifax CIO or something. To start with, this is a company that collects PII against people's will to sell to third parties for their main goal of huge profits. Politics plays more a role than engineering talent in people getting the senior, management positions in such companies. They also tend to chase whatever is popular among Fortune 500, esp with cheap labor or ecosystems available. Java was one of the fads that financial sector was all over. Combine all this to have a company whose fad-chasing CIO keeps costs down and profits up applying the thing he or she read in a computer magazine with the cheapest talent available on a tight budget. The result of their work is a pile of garbage they have trouble patching. If you doubt this, just look at the security of the web site they deployed for credit monitoring and apparently to help hackers get at people interested in credit monitoring. Or they just made mistakes so easily avoided that they're either inexperienced beginners or don't care at all.

"So what would those systems look like?"

Well, they would have built it some time ago. So, let's work our way from old, high-assurance security toward something commercial and affordable from at least 2000-2005 era. The original work in securing data involved security kernels:

https://www.acsac.org/secshelf/book001/19.pdf

Several of those are still available but expensive. Both security kernels and databases such as Trusted Rubix built for them. Today, those look more like the next link with companies such as Sirrix and Green Hills selling them commercially:

https://os.inf.tu-dresden.de/papers_ps/nizza.pdf

In any case, we'd need a robust combination of OS, database, and application code. Nothing hits the database without going through the app server first. So, we embed our security policy into app server. How to implement it? Ever since Dijkstra's THE OS (1960's), we knew to specify correct behavior with preconditions, invariants, and post conditions. Then use anything from formal analysis to testing to runtime checks to ensure that behavior is enforced. The security kernels did the former where Design-by-Contract in Eiffel used tests and runtime checks. Got to pick method with best bang for buck. Two links to illustrate safer languages of past with Ada 2005 since we're looking at earlier stuff.

http://www.adacore.com/knowledge/technical-papers/safe-secur...

https://www.eiffel.com/values/design-by-contract/introductio...

Given labor situation, we use DbC in a safe language. All prior work showed simplicity was necessary for security. So, it would be an app server done in Eiffel, Ada, or (if absolutely necessary) Java/C++ using Design-by-Contract methodology plus lots of testing. The web component would be a middleware done similarly that basically translates web actions to simple protocol the server uses for requests/responses. Any new types of problems with web apps are mitigated in the simple framework a la Airship CMS, caught with some kind of monitoring a la Spectre, or both.

This would have to run on something robust. You'd have the database/storage, app server, web component, management, and monitoring which is behind one-way link (data diode) w/ logs copied to write-only media. A company from likely mainframe background now run by wise technologists might first make the software target IBM but portable. Balancing reliability, security, and cost, the best targets would be AS/400 or AIX. The former is a capability-oriented architecture with few 0-days whereas other is a rock-solid UNIX. Portability means they'd migrate part or all of this to new OS's as they showed up. Eventually they'd notice OpenBSD's security advantages or buy one of those certified-secure kernels w/ POSIX layer. Depends on their viewpoint and budget. Probably throw some clustering software with rolling releases in there since OpenVMS and NonStop had high-availability in the 1980's. It was well-known strategy.

So, there was a straight-forward way to make a robust stack with a web front end which gets more robust over time. This could be done in high availability or just easy patches. They instead end up with fad-driven crud possibly running on other crud that's easy to hack but hard to patch. Just bad engineering that's typical in the market versus methods such as I described. Also note that there were (are) large institutions using some of what I recommended for those benefits. The tools and people just cost a bit more which doesn't align with incentives of companies as greedy and IT hating as Equifax. Sure turned out extra profitable, didn't it? ;)

Reply on news.ycombinator.com