Equifax Releases Details on Cybersecurity Incident, Announces Personnel Changes
investor.equifax.com
investor.equifax.com
For example, are the items in DB encrypted? Are database backups encrypted? Are different items encrypted using different keys? I don't think EFX did it right.
This is how it works at almost every company. Call it horrible if you want, but everyone stops short of describing in detail the exact architectural replacement. And when someone does, it inevitably has flaws that make it impossible to meet business goals.
It's too easy to handwave this away as "well, get a better architecture." Everyone knows that; the hard part is, what architecture? How does your architecture work in practice? Encrypting items won't help when the keys need to be stored in the same place to access the data. The attackers will just go after that box instead of the DB.
You can guard it more carefully, but the point is that some box somewhere needs access to a substantial amount of the data at any given time. It's the nature of a credit bureau.
Batch processing on encrypted data is absolutbly ok. For example, you can just pull row-level keys and run your audited and signed binary. Also the pipeline should be put away from the frontend.
I'm not an expert but I think it should already be documented in the compliance doc.
"Well don't let that happen!" Nope, not that easy. This threat model concedes that the hackers have RCE. They're on your network. That's why they can do this.
Simply put: if you have RCE into an app, then that breaks all the guarantees of that audited, signed binary. You can make it sing and dance and ask for a thousand rows instead of ten; you can do whatever you want to it. The modifications happen in memory, not on disk, so even if they don't persist you can still modify the app.
Monitoring seems like the only solution.
It can work, but evidence seems to imply this strategy makes the situation worse. I'm not sure what to do with this information other than to relay the experience. It's one of those tricky counterintuitive facts.
How do you read linux firewall documentation with the first example being old school public->DMZ->private? So your web server got hacked? That means what? Your database is across a single port through the firewall, and ACLs mean that any traffic arriving at the part from the DMZ CIDR can only do the limited set of queries that provide single-person-at-time. SSH from hacked box to database? Got a zero-day for the firewall (because 22 isn't open from there), and a zero-day for sshd (because its keys only). I know this and I'm not one of our hundreds of security people. (And as I'm not the security guy, if any actual security guys want to school me here, please do so!)
Sure, I agree that business peeps will say "Oh that's too expensive I don't care", but lets not pretend that we don't know how to do this.
Be clear, if you are the IT security guy, and you agree to bow down to "business" people (you're a business person) who tell you "No, that's too complicated", then just be clear that you are getting fired when the shit hits the fan.
It's much better, certainly. Companies should do that. But it's not obviously a panacea.
What I want to say is, it wouldn't surprise me to hear that the three main bureaus process millions of requests per day.
Oh, and also - all bureaus were replying to a request in less than a second. Quite impressive.
This underscores the importance of figuring out a better architecture. And not just for credit bureaus.
It is remarkable how resistant people are to the idea that you need to be patient while the world changes. Everyone is out for blood and gives not a fleeting thought about what would come after such a purge.
The first very likely scenario is given business logic says a web server application needs certain data, where "certain" is "pretty much all of it". So your application server in that DMZ has a database credential it needfully holds.
Which in turn is a credential an attacker can use to access "pretty much everything".
single person at a time
Overall, security people don't get the choice about "bowing" to business people in most of my experiences. I'm not saying scape goating never happens, but this
then just be clear that you are getting fired when the shit hits the fan.
I can present risks, and offer mitigations. More often than not, some get through easily, some never will, and it's those on the fence you work on constantly improving.
Now, sure, if the front ends query 10% of all records every day, then yeah, it only takes 10 days to get all the data, one query at a time, without triggering any alarms. They had 75 days. I don't know though, on any given day, what percentage of the population are getting credit checked? And did they have any alarms?
SELECT * FROM users WHERE ID = 10
SELECT * FROM users where ID > 0
But the development world has been turning on stored procedures for a while. You've got popular frameworks actively shunning the concept[0].
I've seen the topic on HN before and it's generally been the view that there is rarely a use case in modern times. We can't suddenly say everyone is expected to have everything in a stored procedure now that it's convenient.
[0] http://web.archive.org/web/20060418215514/http://www.loudthi...
The more modern equivalent, is to just have an "app server" that sits in between the web (DMZ) and db tiers. The app server can basically be just another web server that provides an API for the actual web server to use. So from the web server's point of view, it still has restricted access, but you get to write all the rules in your normal backend architecture/language rather than building them with stored procedures and triggers and all that.
Besides, the "rarely a use case" isn't the same as "never", and if you can envision any scenario where it would be worthwhile to go to extraordinary means to ensure security, I think "we have all the data needed to commit identity theft on half of all Americans" counts. If stored procedures is a convenient way for you to limit the impact of a breach from your web server/DMZ, then do it if you have data that important. Do 20 other things that are inconvenient too if it gives you defense in depth.
App server has no database access. No DB credentials, and no network access to the DB. It does, however, have access to a service that sits in front of the DB. It can ask the service:
GET /users/10
SELECT * FROM users WHERE ID = ?
It also means that we can use access controls on the API (the stored procedures).
In my experience, allowing anyone to hit a database with SQL leads to poor code quality, bugs and performance problems. And security issues.
And this is exactly where the problem comes from. A web application should never be allowed to talk directly to the database.
How many jobs are out there for people who tell the "business" people what they want to hear, versus people who tell them hard truths about security? How many businesses succeed despite lax security -- gambling irresponsibly, yet winning?
You can argue that Target is not in the "we need to secure this treasure trove of data" business and so them leaking 40 million credit cards is just an unfortunate accident.
But when you sit on the PII of 150 million people then you need to be top notch security wise. I am sure the eventually inevitable courts will appreciate a company culture where admin/admin is acceptable.
- PATCHING OF KNOWN CRITICAL SECURITY VULNERABILITIES IMMEDIATELY. This is an architectural decision! If you cannot put out a hotfix within months of a vulnerability being made public, YOUR APPLICATION & DEPLOYMENT DESIGNS ARE BROKEN.
- WAF rules for common attacks and known CVEs
- Least privilege enforced architecturally via stored procedures or microservices or something else so that a vulnerability in one application (the online dispute portal in this case) can't lead to a dump of everyone's PII.
- Data should be encrypted and encryption keys should be stored apart from both the application and the data. Upon startup, the application should make an authenticated request to read the encryption keys into memory. This significantly raises the bar to dumping sensitive data.
- Monitoring for connections transferring anomalous amounts of data
That's just off the top of my head. All of these can be designed into a system and still maintain 100% uptime.
I worked in system admin for many years. If anyone was actually foolish enough to do this, they would be dealing with multiple major outages every year.
> for every complex problem there is a solution that is simple, obvious, and wrong.
There's plenty of companies that are able to regularly patch and upgrade software, incident free. If you team isn't able to do this, they need to fix their process instead of putting their head in the sand.
I am not defending the totality of EFX's process by any means. I am just pointing out that "install every high severity patch right away" is not as easy as it looks.
Others have pointed out this patch was not available for all current versions, for example. So, to install the patch you need to upgrade. Oh, that breaks <dependency>. So we need to upgrade <other thing>. But that regresses a feature we use ....
The major issue with EFX was the fragility of their overall architecture. A single weakness should not be enough to lay the whole DBMS open.
Stand by my initial post, declarative systems will become more and more popular and will improve security.
"There might be an outage" is not one of them.
Companies are driven by profit; unknown patches can negatively affect profit and are usually treated as such.
Alternatively, there are sometimes political reasons. A director may not want to risk an outage for fear of being dressed down by their VP.
Any change to a production environment comes with risks, sometimes having a security risk is judged to be lower risk than making a change.
I've seen this from small startups to multi-billion dollar companies.
Ignoring these factors is also putting your head in the sand.
This is such a weird thing to me. I've never worked anywhere like this and so I have a hard time imagining it really. Wouldn't that same director get more than dressed down by the VP if on their watch a _known security vulnerability_ was ignored and the system was breached? I just can't understand this way of thinking.
The cost of an outage is very easy to quantify (revenue per minute the system is down), and the probability that something will go wrong while applying the patch is also somewhat easy to predict, and usually greater than zero. The director will be blamed with certainty for the outage, since he approved it.
The cost of a security breach is difficult to quantify; it depends on what gets breached and how bad. Note here, I say breach, not vulnerability. Even if there is a known security vulnerability, it's not immediately obvious in all systems what the consequence will be; there may be other mitigations in place outside of software that reduce the potential damage, or there may be unknown vulnerabilities that are exploitable due to the known vulnerability that make would make a breach worse. The lack of certainty about the consequences means it's also possible for the director to avoid blame if the breach is minor ("how was I supposed to know that other team is still using MD5?"). If there is no breach, then there is nothing for the director to be blamed for.
Given that the director would like to avoid being dressed down, director will be more inclined to delay patching over possibly causing an outage, because the costs of an outage are easy to predict and he will take all blame for it. The breach may never happen and even if it does, it may cost him personally less than the outage.
If this still seems weird, it might be because you are someone who views patching as an easy thing to do, because you probably work for a software company. Software companies are used to managing changing software, and have all kinds of practices around minimizing the risks of doing so. Non-software companies typically find patching to be hard and costly because their core business is something else; changes can disturb the "something else."
> without it leading to major outages
I think I would have preferred an outage over leaking millions of american's private information.
With full 100% hindsight vision, of course.
It's also a business decision. You need to invest staff time in testing and automating said updates, and will inevitably lead to more outages. Thus less profits, directly contrary to your assertion. Managers have trade-off between a small but relatively predictable cost, and a large but presumed highly improbable cost. This is something that neither data nor human intuition are particularly good with.
Best case scenario insurance lets you convert your large risk into a small predictable cost, and the premium difference decides whether you stay on the CVE treadmill or not. But its not like actuaries have a lot of data to model to the point of accurately imposing a cost per day of unpatched CVE.
Worse, many places have competitors, and if they're choosing to take on risks in the name of lower prices and higher market share, the customers of your secured app walk out the door. Not only do you need to invest in security, you need the competition to as well. I have a few ideas on how to achieve this, but they're not historically palatable to the HN crowd.
There are many vendors attacking this exact problem from many directions, including my employers[0], but pretty much every platform vendor has something here.
I do agree that automation and risk are at the root of the tension between security and stability. It's a feedback loop that can run in two directions. If you embrace stability, the loop becomes "no changes ever" and security becomes much more difficult to achieve in practice. If you embrace security, then at first you face instability -- but embracing rapidity and building out tooling for it makes you both more secure and more stable in the long run.
> But its not like actuaries have a lot of data to model to the point of accurately imposing a cost per day of unpatched CVE.
I'd be interested to know if this is actually true.
I recently got interested in the FAIR risk modelling approach. It requires a bunch of legwork, so in practice it won't catch on without determination to use it or something like it.
Usefully it breaks risk down into more easily-estimated, easily-tracked subcomponents[1]. "Probability" becomes "Loss Event Frequency", which further breaks down into Threat Event Frequency and Vulnerability, these break down further in turn. Similarly, the usually-vague "Impact" becomes Primary Loss and Secondary Loss, each of which has 6 specific types that can be estimated. Those estimates can be based on available data, whether your own or industry statistics.
> Worse, many places have competitors, and if they're choosing to take on risks in the name of lower prices and higher market share, the customers of your secured app walk out the door.
[0] I work for Pivotal. We've been building towards the vision of "Rotate, Repair, Repave" in Cloud Foundry: https://builttoadapt.io/the-three-r-s-of-enterprise-security...
[1] https://cdn2.hubspot.net/hubfs/1616664/The%20FAIR%20Model_FI...
At work we have a lot of customers now, many of them with large security orgs. Their approaches vary a lot, presumably because of their experiences and security maturity.
Beyond a certain level, there is always a security org and quite typically it has veto powers that override any local manager's budgetary concerns. Whether that veto is applied well is quite distinct from the way it creates a ceiling over potential spending on features.
That all said, I agree that there are fundamental properties of security that will cause it to be consistently under-attended relative to what a more actuarial or financial model would suggest is rational. It is only observable in its absence, but most feedback systems -- formal or informal -- rely on observable signals. Market releases may lead to a juicy bonus at many companies. Invisible improvements won't.
I've thought about this a lot in the past few months and published a bunch about it internally at Pivotal. I'm not alone, of course. We're streets ahead of many vendors, but I also feel we're only getting started on the deeper and more difficult process of blending it deeply into our cultural DNA. It's a culture which believes in doing the right thing, but also a culture which detects vetoes as faults and routes around them.
Interesting times.
It should follow the same processes as any other application update. Are you suggesting it's impossible to push updates to software without incurring outages?
Obviously this is fixable, but it's no free lunch.
Immediately? No real company can do this. 1-2 weeks after the announcement is the best you'll see, even with something of the magnitude and publicity of Heartbleed. I'm not saying that's how it should be but that's how it is. Most organisation's it's quarterly!
All of these can be designed into a system and still maintain 100% uptime
Sure. If you were designing from a clean slate right now. The vast majority of organisations in the world aren't.
And believe me: in 10 or 20 years your new design would have revealed flaws too, and look laughably primitive to the practitioners of the day.
What's the solution then? An architecture in which no single compromise can compromise the entire system. That's getting harder and harder to do as the landscape homogenises around Lintel.
You don't need to patch your software right away, but you should have other strategies for mitigation such as a WAP in front of your services that you can configure to block malicious traffic when you identify a remote code execution vulnerability.
patch: https://aws.amazon.com/security/security-bulletins/heartblee... (4/8/2914)
IIRC: 12 or 18 hours rolled out to all of aws ELB (main part affected).
https://usn.ubuntu.com/usn/usn-2165-1/
Patch was available at time of vulnerability release for current and LTS releases of ubuntu. apt-get update.
But you know... That's just too hard.
Isn't it interesting that one needs a license to drive an automobile and yet companies that handle this type of information get to do so without demonstrating the required level of competence.
Oh, it's the whole industry tho'. VMS running on DEC hardware was at a level of security and reliability that we can only dream of today, but it was abandoned on the scrapheap and replaced by glorified PCs running a hobbyist OS. If you wanted to build an infrastructure on old-skool principles now you couldn't, you can't get the components anymore. The truth is we're fighting a losing battle and the real enemy is ourselves.
They can, but it requires automation. I've worked on and used tooling for just such things. We can see an upstream CVE and have it patched in production inside 48hrs. Usually the same day.
Disclosure: I work for Pivotal. All this stuff (Cloud Foundry, BOSH etc) is opensource, but we sell commercial versions too.
This is just business being a) ignorant b) incompetent c) careless one or more of the above applied for a hack like this to happen. It's really sad that even ycombinator crowd doesn't get it, given that the crowd here is significantly more technical and sensitive to these issues than the average Equifax Joe..
Simple. Don't store a key with the data. A credit bureau only incurs liability by knowing the PII for hundreds of millions of people readable from one location - that they provide to everyone who asks!
How does a company create a better architecture? Separation of duties and responsibilities, defense in depth, authentication and authorization, cryptography, updating software, etc. Hire a security team to help with vulnerability analysis, design reviews, due diligence, and threat modeling.
This gets solved every day at many companies.
"You can guard it more carefully, but the point is that some box somewhere needs access to a substantial amount of the data at any given time. It's the nature of a credit bureau."
Yes, but the elephant in the room is that a credit bureau possesses that information in usable form. This can be taken care of by key management, where the individual owns the key and only one key is able to exist per individual at a time. Then, only those credit grantors who have been authorized by the individual are able to update or view the data.
In cryptography the operating assumption should always be that everything is known about your data except your key material, which is kept private.
This is what I mean by not meeting business goals. It is clearly a bad idea to give Grandma an irreplaceable key. But that's what this proposal is.
Uniqueness is a different property from replaceability. And, replaceability is not necessarily the correct property to design for, key rotation and recovery from loss are the properties. What you want is the ability to rotate keys and recover from the loss of keys.
Don't ever store multiple pieces of personally identifiable information in the same place.
The consequence of this is that Equifax's business model is no longer viable. I care about that as much as they care about me and my data.
Equifax had a trivially easy architecture to break.
Minimally, the data should be on a separate set of servers protected by an API with rate limiting, and that API should be implemented on top of a different framework than the front-end servers to minimize the chance that a single flaw would open up the entire set of data. This isn't rocket science. Equifax is a multi-billion dollar company. They could and should have done better.
Please stop defending them?
Disclosure: I work for Yahoo (er, Verizon) since late 2013. I don't know the details of Yahoo's data leaks, but it was not as trivial as getting RCE on Yahoo's public facing servers.
Now, I've seen nearly a hundred codebases and they all looked pretty much like Equifax. I don't even need to see Equifax to know the kind of disaster their codebase facilitated.
Equifax's codebase is your code, and my code, and Bob's code. The only way to make this situation better is to talk openly about it. And the best way to do that is to actively defend the unpopular side in order to call attention to uncomfortable truths. It's a calculated strategy.
Just because you worked at a place with good security hygiene, it makes no difference to the world at large. Everybody else looks like Equifax. We need to (a) acknowledge that and (b) actively seek out solutions. That's what's happening here.
Ask yourself: "Why do I disagree when I've seen one or two enterprise codebases?" Those big, plodding institutions you mention are similarly vulnerable. Everyone is.
Why aren't data breaches happening almost daily? Because it's illegal. It's no coincidence that companies with public bug bounties are among the hardest targets to hit. When you make it not-illegal to attack you, surprise: you find the vulns.
A good question is, "is it illegal for a white hat to attack my business?" If the answer is yes, seek to change that. It's a low effort thing that will increase your security posture. But it's no substitute for $40k pentests.
I don't doubt that there are other Equifaxes out there, nor do I doubt that there will be more leaks. But this wasn't a simple matter of a horrible code base. Heck, the leak wasn't even due to Equifax code, it was due to third party code. Other companies of Equifax's size apparently do better. Equifax should have done better, and I'll bet if the penalties for losing this data were more severe, would have done better.
Know what happens when you demonize something? People stop reporting it.
Look at the actual damage to actual people. You are arguing for prison time for the execs, if I remember correctly. Yet it's unclear that a single person has yet been materially damaged by these leaks. If you want to suck away N years of someone's life, you have a higher bar to clear.
I haven't commented on the Equifax incident, so you're confusing me for someone else. I don't have an opinion one way or the other on criminal negligence.
How do we incentivize (or regulate?) companies to do a better job handling confidential data?
A note of "Well, let's not be too hasty" would seem to be in order.
Part of the problem is that security breaches matter, but they don't matter. It's like the negative externality of factory pollution. One day perhaps the effects will become visible, but this disaster is about as bad as you can get and it's still not obvious that a lot of people are going to lose anything significant from the breach. If someone wanted to impersonate you, they can usually find a way already.
Of course, if the 140M SSNs become public, that will change the situation completely. Possibly even for the better: then we'd finally stop relying on them to be secret when they were never designed to be.
Believe it or not, the situation now is drastically better than it was a decade ago. People care about getting pentests now. You can grudgingly force a company to fork over the money.
One of the best ways to get companies to become more secure is to force them. Most of the companies who were being pentested didn't want to be; they were forced to, because some other business refused to do business with them until they produced a document stating they were reasonably secure. This special document is only obtainable from a security firm; hence, pentests happen, and the world is marginally more secure.
Mm, it can become that. It's important to get a pentest from a reputable firm. At the one I worked at, it was a point of pride to come up with clever attacks. We'd make a sport of it and try to one-up each other. But it's easy to imagine some other pentest firm just going through the motions, running Nessus and calling it a day.
Unfortunately, that's where money really does make a difference. $40k is at the low end of how absurdly expensive pentests are. Yet they make all the difference -- one app that I pentested was at a trading firm. I took that app from "typing ' leads to SQLi in their Node backend," and "there was SQLi in their Java backend if you were clever" to "you'd have to be more clever than me if you want to breach this app."
Most attacks come from unsophisticated adversaries. This implies that raising the bar higher than these guys gets you 80% of the way home. The other 20% is much harder, but it's doable if your company has a culture of caring about security.
> If a CISO can come up with a list of controls that he/she is comfortable with, then by and large the evidence proves that those controls are working effectively and are going to satisfy the elements of any framework that you use.
These are two soundbites from an interview with Susan Mauldin from over a year ago. Someone didn't want us to hear them, according to the timeline[1] of when someone remembered and reported that these interviews happened on September 9, and how soon after (on September 10) they were taken down (and then late last night, at least partially recovered again, further on down in this very HN thread!)
They are not even especially damning soundbites if you put them out of context and honestly evaluate them in a vacuum, but it does put some additional context on "it was due to third party code" and in the context of this whole discussion, it is clear why someone did not want us to see this interview.
If the job of the CISO is to deliver the project under budget and on time, rather than to do the job correctly as you and I understand it, then we are certainly more likely to find ourselves in this position. So let's keep having these discussions and make it better! It is absolutely your codebase and my codebase, we are all in this kind of trouble (or, we could be... if we all similarly had 140 million involuntary "customers" entrusting us with their SSNs and credit histories.)
[1]: https://www.hollywoodlanews.com/equifax-chief-security-offic...
The same fault is responsible for the Space Shuttle Columbia crash. It's called "leadership without vision." If you put a manager in place and give them tight constraints on how they deliver the product, they will almost universally deliver a critically flawed product ... albeit, on time and under budget.
The logic you have laid out makes it sound like only white hats would ever try to attack a credit agency.
I'm pretty sure by the definition of black hat, and criminal organization, if these companies have something they want, and they have a technical means to get it, the fact that it's illegal is not a deterrent, it's actually their business model.
Look at it this way: If you get a pentest, you will immediately discover there were glaring security flaws that an attacker could have used to get into your network. This isn't true in every case, but for example I can count on one hand the number of companies I didn't get at least an XSS against.
I had no magic. I'm just a guy using Burp suite, working together with someone else using Burp suite. When two talented people work together to break your stuff, they pull out rabbits that nobody knew existed. Especially when they've been doing it full time every day for a year.
I'm sure black hats have talented people at their disposal. But they are selective. Launching a real operation against a US target is risky unless you live in Russia or work in China's army. And while it's valuable for Russian black hats to swipe Equifax's PII, other data troves typically aren't valuable enough to target.
To put it another way, if Russians got into Facebook, so what? They'd get everyone's sultry private messages. Yet Facebook cares more about security than almost anyone else. That's why it's almost impossible to swipe their data.
The vast majority of companies are in the "If Russians got in, it doesn't really matter; also we don't care" category. These are the mom and dad shops that use duct tape and glue for their payment processing. Above this are our companies: If Russians got in, it would be a minor disaster, but we care about preventing it. Except usually we don't care enough to put down $40k.
Then you have companies like Equifax. The worst of both worlds: They weren't careful enough with their security, and it was an unprecedented disaster when someone got in.
Yet even here, it's unclear how much this unprecedented disaster is going to materially harm people. Will those 140M SSNs show up on the black market for sale? Who will pay for them, and how much? (Hopefully it will be posted in full, in clear text, so we can all switch away from the stupid SSN system we've been relying on.)
So when it's a combination of not too valuable to attack you, very costly to improve your defenses, and not too harmful if you screw up, you get the present world: ~everybody is vulnerable, but few companies care enough to pay more than lip service. Why would they? Even if they lost everyone's data tomorrow, it would be little more than egg on their face.
This situation is changing, slowly. If the diff between 2008 and 2018 is like 2018 to 2028, we will be in excellent shape. But we have to avoid falling into traps like mandating regulation at every company, or sending people to prison. Those are tempting but misguided efforts: regulation would reduce the effectiveness of pentests, and sending people to prison will cause security breaches never to be reported.
It will be hard, particularly when the world reacts like this: https://i.imgur.com/g01f9vV.png But companies like Apple are paving the way forward. Even if they help no other companies, they are proof that security really can work at scale. The trick is to make it not cost Apple's resources to get remotely close to their security posture.
(I hate to pick on Russians specifically, but they have no extradition treaty with the US. It's uncontroversial to point out that many black hat attempts originate from there.)
I don't know... I think companies get penetrated often and are not aware. Or when they do know, they don't tell the public.
Equifax did have good security hygiene. But they didn't have perfect security hygiene, and this particular gap cost them big time.
Most security incidents are like terror attacks that are averted. You read that authorities in Spain stopped an attack, and it means the system is working and people are safe. Then something explodes in London, and people are hurt, and you realize that there's no such thing as complete security.
sillysaurus may be overgeneralizing a bit but he is correct about the vast majority of companies. To me, the key differentiator between companies with serious security projects v. companies with me-too, mantlepiece-style security projects is active, engaged technical leadership at the very top.
This is not to defend Equifax as much as it's to desecrate the widespread dogma of the managerial class that an MBA qualifies a person to lead any project. Time after time after time, most of us who've worked in corporate America have seen security pushed to the bottom of the stack, only becoming a consideration for about 18 months after the last close call or underpublicized breach. There is virtually no appreciation for the problem itself at high levels, or the complexity involved in reasonably subjugating it.
CISO is something of a nightmare job because you're essentially the company's designated patsy/lightning rod. Most of the time, CISOs and those in analogous roles get stuck with the blame for the company's security without ever really being given the authority to do anything about it.
I knew a CISO who set up camp, including prop tent, outside of the CIO's office until he finally agreed to get his guys to patch a server. The CISO took a picture of himself doing so, because he knows the game; if the intrusion occurred, he would have apparent visual proof that would allow him to deflect blame onto the CIO.
Reckless disregard like this is absolutely common and routine, and if you have any doubt about that, a cursory inspection of the sites of companies that are not tech pioneers will easily disabuse you of any contrary notion. Equifax is not alone here, not by a long shot. Not even among large companies dealing with sensitive financial data (don't ask how I know).
The only reason we haven't seen (or, more likely, haven't learned about) widespread cyber-intrusion-apocalypse is because semi-competent people just have other things to do, and there is substantial legal risk if caught. I fully believe that 95% of America's biggest companies have systems that are easily penetrable.
Again, let me reiterate that this is not making an excuse for Equifax. It's just acknowledging that the problem supersedes this single instance. We need a revolution in system admin and application security before we'll solve these issues, because the plain fact is that companies simply can't be trusted to do it under the current conditions.
I don't believe those systems are necessarily any more secure than Equifax's. It may be that Equifax got breached first. But it's also partially because it's actually incredibly difficult to steal funds from traditional financial institutions. Not because of technical security measures. But because of procedure and regulations.
As an example, money illegitimately moved from bank A to bank B can (and in most cases must) be reversed. Even if done digitally.
No, it's not how almost every company works, and yes it IS horrible. Do you know why people stop short of giving details of what should have been done? Because that is a full time job for a entire team of people in any place that needs real security, and part of that job is to be constantly rooting out flaws in both design and implementation and fixing them. And guess what, if your business involves storing and retrieving sensitive personal information on millions of people, then security needs to be one of your business goals. If it is not, then that is negligent.
> It's too easy to handwave this away as "well, get a better architecture." Everyone knows that; the hard part is, what architecture? How does your architecture work in practice?
It is also easy to handwave away the real work that real security professionals do. Security is not a just some module you can strap into your systems, it is an ongoing process that requires vigilance and competence. Yes it is hard, but being hard is not an excuse
> Encrypting items won't help when the keys need to be stored in the same place to access the data. The attackers will just go after that box instead of the DB. You can guard it more carefully, but the point is that some box somewhere needs access to a substantial amount of the data at any given time. It's the nature of a credit bureau.
So you are implying that data encryption is not useful for protecting sensitive data because you have to store the keys somewhere. If that were the case then data encryption in general would be useless, we know that is not the case. This is a design or implementation problem, not an encryption problem.
You're playing IT if you can't deploy a CVE fix in 60 days.
Finally, they only did this because of greed. They encourage online disputes because it saves them money and gives them greater rights vis-a-vis paper disputes delivered to them via snail mail. They could simply have taken the service down if if they couldn't deploy a security patch in two months.
Amateur hour everywhere.
ps -- the data controls I mentioned above were implemented in a company of ~15 eng dealing with similar PII. If you owned front end servers you could query individual records, but you would have to own more services to be able to "\copy (select * from data) to /legit.file; gzip /legit.file; scp /legit.file.gz $MY_SERVER". And if you tried pagers would have gone off all over the company.
I don't think "better architecture" is handwavy. If you have old legacy apps that can't be updated and maintained, don't provide them with a path to the public internet. Or front then with some infrastructure and adequate alarms to detect something going wrong.
I think it's reasonable to respond to a CVE with an update to your app or infrastructure within one week. Most of the time rapid testing will be fine as your patching a library or component, like Struts or Tomcat.
These companies are hardly providing 100% uptime, in the case where an update like this needs a rollback, it won't be a big deal. If you have a good team and you spend the money on them.
Companies that store personal information at that level should be required to implement PCI-DSS level security. This includes going through the auditing process.
Here's a quick overview.
https://www.pcisecuritystandards.org/documents/PCI%20SSC%20Q...
You can work to not implement the security standards, and then try to fake your way through the audit. At that point you are not ignorant of proper security, you are actively endangering the users, and your right to hold PII (Personally Identifiable Information) should be revoked.
- Equifax login form does not use STS header.
- equifax.com/business exposes full server version (Server:Apache/2.4.27 (Win64) OpenSSL/1.1.0f mod_log_rotate/1.0.0)
Like:
https://www.fastcompany.com/40468811/heres-why-equifax-yanke...
I don't think they are competent enough to realize how damning this press release is. This company is finished.
I think its about time we had the equivalent of the NTSB to review and publish, publicly, what happened and what the failures were. Nobody died but certainly many people will be affected by this for years to come.
I watched them before they came down on Sept 10, and they were eye opening. I can say with certainty that the transcripts are not complete, because I remember "resistance to cloud is futile" and other such gems which are nowhere to be found in the partial transcripts that you can still find on the linked archive.is pages.
[1]: https://hollywoodlanews.com/equifax-chief-security-officer/
Some of the best software developers I've known were music majors. I know nothing about Susan Mauldin, including any of her other qualifications or lack thereof, but implying someone is only qualified to do what their college major was tells me this author is a complete idiot.
I personally believe that she was backed into a corner, based on the interviews I watched from some time before the breach (that I can't show you, and I won't link to the transcripts because I know they are incomplete and by far less impactful.)
She sounded to me, exactly like a person who was given a budget that was effectively no budget, and then put into this role because people with a larger stake were sure that she would comply when they said "this is your budget, and not a penny more." I know exactly what that is like, because I have been made CIO and put in that position before. (I'm sure she was better paid than I was...)
There is absolutely 100% a coverup going on. I am so bummed that I did not save the interviews when I first watched them. You would agree too, if I could show you.
Edit:
Here is the transcript, but with the disclaimer that it is 100% not the full transcript. Although it is in her own words. Make your own judgement about her qualifications please.
It's really a pretty balanced interview. This is what she said. There was a second interview, but I'm a lot more incensed that this information is being suppressed than at any one thing she says. I want to be furious when she says things like
"If a CISO can come up with a list of controls that he/she is comfortable with, then by and large the evidence proves that those controls are working effectively and are going to satisfy the elements of any framework that you use."
but all this interview really can show is that she knows some jargon, and her mindset. And I think it's really (truly, not in a figurative sense but literally) criminal that this information is suppressed. This is the case study of an interview that reveals insights from the mind of a CISO before disaster strikes... I literally don't even care that she was a Music major, how can anyone justify taking this down?
It's not just potentially criminal, it's also unconscionable. This should be preserved for posterity, I want to tar and feather the company, but I want to hear more from Susan Mauldin about what went wrong at Equifax.
I don't just want this interview back online, I want there to be a follow-up to this interview! And if it takes a pardon from Trump to make that happen, let's start the conversation.
-----
Edit: Well, I found one video (11m30s), were there more?
-----
Edit:
From a quick check, this seems to be the same interview that was transcribed, though I'm not sure if it's the entire interview or even the one you were looking for:
80MB, M4V: http://evilrouters.net/media/susan_mauldin_cazena_interview_...
I'm uploading it to YouTube as well but I wouldn't be surprised if it gets taken down at some point.
Edit: Bless you for doing this, however you found it.
If it matches the transcript, the one I'm looking for has the expression "resistance to the cloud is futile" in it (which I didn't see in the transcript.) It might have been the follow-up interview that had this phrase in it. From reading the first transcript, it looked pretty accurate from what I could remember.
It's too bad we don't have that transcript. I honestly can't remember much about that second interview. Didn't think I'd be here in this position today, trying to retrieve it for the public interest.
Edit: This one does have the quote about the Borg in it. So I remember absolutely nothing about Part 2 of the interview, and we still don't have a copy. But this is something I can't find anywhere else on the internet for now. Thanks!!!
They also gave you credit for finding it! ;)
Looks like the embedded version on their page is missing about 1m30s of the interview, though.
https://www.youtube.com/watch?v=vUskCtFOKdg
I am so ticked about this hack. These stooges make billions off of us.. where are all those billions going? Not towards security!
Overall we can no longer use our socials for private identification. A different system is needed and is needed now.
Further freeze your credit and force these stooges to lose massive amounts of revenue!
That's the root cause here. The same thing happens over and over. They find ways to save more money, get the rewards, and the rest of us get screwed.
- Senior Vice President of Enterprise Security and Chief Security Officer at First Data
- Senior Director of Information Security and Audit Compliance at Hewlett Packard
- Group Vice President at SunTrust Banks
- Vice President of Information Technology at First Commercial Brokerage
- Vice President of Information Technology at E&I Cooperative Services
She presided over a total security flop
This means she had no job being a chief security officer.
Either she is the scapegoat, or she knows who is responsible for this. I'd love to have her on record to testify about the culture and what happened some time before they've effectively paid her to go away, and stashed her somewhere off the coast of Bermuda.
Equifax "believes" that the hackers got in on May 13. They had some kind of intrusion detection system that finally detected the intrusion on July 29. 5 months after the "Critical" CVE alert went out. During that time security vendors were adding firewall rules to stop the attack. But apparently Equifax didn't have any other security in front of the Struts server.
That just seems like unconscionable incompetence and malpractice for such a high value target.
If they had some sort of intrusion detection system, I gotta think they would have picked up on this in days, not more than 2 months later.
I feel like this is one of those situations where a dev is saying something like "where is all of the CPU on this database going?" or "Why is the network connection so slow?" and digs into it and finds some strange behavior...
This looks really bad. I don't know what's the point of letting them operate anymore. They utterly failed on the single thing they were supposed to do.
I don't want to downplay their security failure. It's obviously really bad, but saying that security was the "single thing they were supposed to do" is false.
Never mind, I guess too big to fail applies here as well.
Also knowing how JNDI usually gets configured on app servers (sometimes with credentials and all) would have made recon ridiculously easy.
I think some have alluded to it, but what some people in the comments don't seem to understand is that RCE is an "all bets are off" type of situation. DEFCON 1 to be sure. Prevention is really the only good answer.
-asking as a relatively inexperienced dev
Don't be so quick to judge.
I have known people without degrees (or without relevant ones) that learned on their own and were great. I have known people with a CS degree that were terrible.
If you're working on something important, say critical national economic infrastructure, you do the equivalent with automated staging and testing happening before any potentially breaking changes are made to live servers.
Or... you do nothing, as the case may be...
There are services that monitor your package configuration(s) and let you know when something has been updated.
There are also mailing lists. Unless you're a Node developer, you probably only have a couple dozen dependencies in your app. Subscribe to them.
Finally, you can just check in your lockfile and update packages as part of your dev builds, then commit it whenever something changes. Your CI/CD will make sure you are always running the latest version of every application dependency in production.
1. Reliance on a consumer-grade component in a security-critical system holding high-value data.
The portal should have had a small, audited code base with secure coding techniques and minimal reliance on third-party components.
2. Excessive attack surface on a system holding high-value data.
The machine hosting the portal should never have had read access to SSNs. Sensitive data should have been "thrown over the wall" to a secure backend with a constrained interface. This would have greatly reduced the scope of the breach.
"The particular vulnerability in Apache Struts was identified and disclosed by U.S. CERT in early March 2017."
"Equifax's Security organization was aware of this vulnerability at that time, and took efforts to identify and to patch any vulnerable systems in the company's IT infrastructure."
?????????!!!!!!??????
"While Equifax fully understands the intense focus on patching efforts, the company's review of the facts is still ongoing."
That is not even close to an excuse. A remote code execution vulnerability has the potential to destroy your whole company.
> The company's internal review of the incident continued. Upon discovering a vulnerability in the Apache Struts web application framework as the initial attack vector, Equifax patched the affected web application before bringing it back online.
That bullet point lies between the "July 30th" and "August 2nd" bullet points. Based on that timeline, the vulnerability took days to patch.
Equifax's core business is about giving out credit scores. I'd bet their biggest fear is giving someone a high score when they deserve a low one. Data breaches, moderately inaccurate information... a nuisance, but a sideshow.
MARCH 2017 - Vuln in Struts is disclosed by CERT.
MAY 13 - Initial intrusion happens according to FireEye's later analysis
JULY 29 - Equifax notices weird activity.
JULY 30 - Equifax notices more weird activity.
JULY 30 - Equifax takes down affected web site
JULY 30ish? - Equifax realized vuln was Struts. Patches. Puts site right back up.
AUG 2 - Hire FireEye to check things out
(weeks) - FireEye assesses situation and presumably Equifax panics
SEP 7 - Equifax makes intrusion public, offers self-described "comprehensive package" including the web site "so that consumers can quickly and easily find the information they need".
At SOME point, they kind of shove in the statement that following entities were notified:
* FBI
* all U.S. State Attorneys General
* other federal regulators.
My question is WHEN did this happen? It's my understanding there are rules-- state and federal-- about when law enforcement (and affected parties) need to be notified on a breach of this size involving this type of sensitive information.
Anyone have more insight on this?
They might as well be running their business on a cluster of tomcat servers sitting atop sqlite.
Hopefully they don't recover from this - they should not have the data they posses if they cannot mitigate risk.
Yes, developing against such a system might be annoying (think about updating a new piece of data for all records).
But it feels to me that we need a way to rate-limit access to sensitive data, to prevent wholesale dump in a short time. But you still need other systems in place, to prevent a hacker lurking around for months until it gets all the data.
"Hey... Splunk tells me there's been a 34x increase in record queries. Page page"