Fresh Horrors from Equifax CEO Richard Smith's Congressional Hearing

Right. I think it's code for "I think domain means website, and our technical folks said our existing corp site didn't have the capacity, so we told them to setup a 'completely separate site'".

Ya, that solution alone says a lot about the technical decision-making taking place.

Especially given that random numbers input to the site returned that data had been compromised.

Most probably their marketing and legal team came up with the idea. Once the outcry blows over turn off the site and all incoming external links become dead. After a while general public will not remember and wouldn’t associate their main site with this incident through search. Legal might have encouraged to have two fully independent sites. I will not be surprised two sites belong to two separate legal entities. Just a speculation on my part considering how US corporate behave.

Seems to me to be dead simple. One dns entry for equifax.com, another pointing off to different hardware for equisecuritybreach.equifax.com. Nothing fancy at all. Make the page have a different look and feel.

Not me :(

I'm just a .NET developer, I have no idea what my code runs on.

That seems to be indication that decision process there is as broken as security process. Either they did not have a professional that could explain the management why the claim above is baloney and what are the solutions to make it work, or that professional wasn't consulted, or his opinion was dismissed by the management. In any case, the symphony of fail continues.

On the positive side, their chief security officer can compose one heck of a musical composition.

>the company's main domain was not architected to process the enormous traffic

I understand all the sarcasm expressed here, but this actually may be true if they run their own DNS service. Subdomain isn't a solution in this case, but separate domain is.

from a security point of view it does kind of make sense to have non-trusted content hosted under a separate domain. so if they were using a third party to host the breach page it may have been risky to have it hosted under the .equifax.com domain. this is why you have .github.io and googleusercontent domains, etc. though, probably the risk from confusing users is worse than the risk from the third party playing silly games with cookies in this case.

The solution to stopping stupidity like this is to take a 12 month tour of duty at the US Digital Service:

https://www.usds.gov/join#tours-of-duty

Yes the bureaucrats, lobbyists, and special interests will continue fucking everything up.

But the USDS is set up so that at least competent technologists can throw up some objective defense vs. the government officials who really think Equifax is a viable solution aka "Nobody ever got fired buying an IBM."

That is uttlery insane!! I thought the best possible outcome for Equifax was to avoid getting sued into oblivion, but now that get a 7 million dollar contract with the IRS?!

There are clearly two sets of rules - one for large companies where they can brazenly flaunt laws whilst going from strength to strength. And another set of rules for everyone else - where if you take a wrong step it's all over!

I don't disagree with any particular assertion you make.

Let me ask the obvious question. What do we do then?

(If this doesn't spark discussion I'll gladly contribute my own ideas, which involve finding methods of partially-civil disobedience to even the scales of inconvenience a little bit. I honestly believe a large issue in the current system is that there's no real pressure being held against any of the people with real power. Why would someone who can bring in tens of millions _FROM FAILING AS BADLY AS ONE CAN FAIL_ in their job, give a damn about anything but optimizing that number and minimizing time/effort spent to achieve that? With senate and house incumbency rates hovering around 90%, why would a politician ever deviate from the status quo? The wellbeing of the people is so far separated from any of their driving incentives, we must find ways to realign that.)

Agreed on #2. Quarterly meetings with CEO about security don't seem so bad in the general case (although in Equifax's case, security is a way bigger deal), but that shouldn't be all. Only 1 person knew about the patch? Even a medium-size enterprise where security is even a moderate concern should have more people than that focused on security 100% of the time.

Quarterly is possibly fine, but when sparks are flying in the back end, and fluid is pouring out of the transmission on the running engine, CSO needs to be able to access CEO promptly.

First, no one gives a crap about security until something like this happens to them. The incentives are not in place to require it. Time and resources spent on security are time and resources not spent on beating the competitor. The best outcome you can hope for if you _do_ spend on security is that your competitor will have some apocalyptic breach similar to this one, and that doesn't happen most of the time, at least not in a way that results in any accountability or visibility.

It's important to understand that because it's the reason why your bosses will never care unless/until they have an experience like this, or until they expect someone important to be looking for an opening (audits, due diligence, etc).

Second, there are indeed some clowns and cronies hired at these sort of places, but a lot of the people are decently skilled. I'm speaking generally about the workforce at these larger enterprisey companies, as I don't know anyone at Equifax afaik.

Technical competence is less of an issue than intra-managerial political games like "shift the blame" and "silo defense", at least some of the time. Part of "intra-managerial games" is having the least-skilled people in the most authoritative positions, leaving decision authority with those least equipped to evaluate a situation and determine a responsible response.

I am sure that there is someone at Equifax who saw the news on this vulnerability and registered it as being something they needed to patch. I wouldn't be surprised at all to learn that this person didn't say anything because they learned long ago that there's nothing to be gained by actually trying to get the patch done in any non-routine manner.

I also wouldn't be surprised to learn they did try to raise the alarm and were mocked and/or shut down, both by other technical groups trying to avoid the appearance of an issue with "their" section of the system, and/or by management, who are likely to interpret such attempts as alarmism, if not plain political malfeasance.

Hate to say it but we're on the brink of a formalized licensing program for security, servers, etc. I am honestly surprised that the mainstream outlets and politicians have not been touting this yet, with all the high-profile and extremely costly data security breaches that have been occurring over the last few years, but I really expect it to come soon. I wouldn't be surprised at all to see Equifax become the Enron-like scapegoat for such legislation.

#2 is more than just production scanning.

Open source vulnerability management is a technology process that has levels of maturity. That being said, a freeware tool could have spotted this right away.

Apparently the right user:security engineer ratio is 143,000,000:1. /s

Maybe he was talking about himself?

We do not know anything about the number of people who could have known about this issue and escalated it, but it does not surprise me at all that there is one primary person who is responsible for it. In fact, if there were not exactly one person who is responsible for the security of the application that would be a different sort of problem as diffuse responsibility can be neglected quite easily in large organizations. Still, at my own company - A LOT of people were focused on the timely remediation of this issue and there was no chance of it being left to one person, even for development environments of internal facing applications.

The scanners do work, we used them at my company and they found this specific vulnerability. It surprised some development teams because there are libraries that use struts without really advertising it, so even applications that don't "use struts" had a transitive dependency on it, it was running in their container and showed up on the lists that were circulating.

> Do these scanners ever work?

See https://www.tenable.com/blog/apache-struts-jakarta-remote-co...

Yes, they do. Detecting the flaw is not a problem, but often operationally patching can be a big deal.

> How is this one individual responsible for tracking all IT security vulnerabilities in all the technologies that equifax uses across its stack? How do they not have an admin team whose job it is to do these things?

They do have a vulnerability management team; the responsibility does not fall on one person.

For all we know this unnamed individual is just a low-level developer, analyst or pentester who had some knowledge of the exposure and failed to act.

Having personally seen the sausage factory on one of the "scanner" companies, I would be very careful and do due diligence on any such thing I contemplated buying.

Compliance & Security "in the enterprise" is a smoke-and-mirrors game full of weird terminology divorced from the nuts-and-bolts issues with people who don't know what they're selling engaging with people who don't know what to buy, playing the industry-form and trade-show and junket and relationship game (probably no different from the DB market, the network market, the ERP market).

The DKs that fail at technology and don't really know it move on to these segments of the security market, to everyone's detriment.

> Do these scanners ever work?

Yes they do. But you have to have a process to triage what the scanners produce, and have a team whose job it is to keep the ops/dev side accountable.

They are quite useful when scanning all the internal parts of a datacenter. There's a fair amount of nitpicking but it helps weed out the obvious (like installing some open source package which defaults to some bad cipher for SSL, or leaving internal links unencryped under pretext that it's "safe behind the firewall").

Often, though, the issues flagged by the scanner trigger deeper conversations about security. That's where the real value is, but that requires an organizational culture that actually cares about security. Instead, many companies just throw money at the problem of "security" and consider the scanner will fix all their issues with zero effort.

> the only reason we used this at a previous company was to kind of handwave around hey we have this tool doing regular security checks

That's the only reason for most of their clients. "We get periodic security scans from a respected vendor, yes sir!".

I can't rule out that the scanners might work, but if the vendor isn't naive, they'd be optimizing for their real target audience.

>Do these scanners ever work?

For this particular vulnerability? I doubt it. (Source: I worked on building such a scanner recently).

Let's begin with a vulnerability that is obviously detectable with an off-the-shelf scanner: "Buffer overrun in Linux CIFS Server". We can detect a host with this vulnerability by simply scanning the local subnet for live IP addresses then fingerprinting the host to determine it is Linux, checking if it responds on the SMB port and finally sending it a test exploit payload and see if it responds in the expected way (or crashes). This all takes a few ms.

But now consider what if the vulnerability is only exploitable by an authenticated session? Well we could have our scanner ask the operator for a set of credentials for each CIFS server it finds. But what if the vulnerability requires a mounted share? Well the scanner can ask the operator for the name of a share, or if it is lucky it could try to guess one. Perhaps we could be happy with the scanner identifying the version of Samba running on the server and concluding from patch history knowledge that it is vulnerable. But boxes get locally patched and often there isn't enough information in the externally visible version info to tell one way or the other.

Now think about trying to do this in the context of the vulnerability "If your application uses Struts and allows file upload then it might have this RCE vulnerability". We don't even know if we're using Struts in any of our applications. We may have hundreds of applications. Is it possible to tell from the outside that Struts is being used? (I'm not sure but probably not). You could note that the web server is Tomcat and therefore the application is written in Java and therefore it might use Struts. There will be hundreds if not thousands of potential CVEs to check for given only that information.

You don't know if a given web application even supports file upload. How to you tell? Look at pages for the string "Click to upload"?? (Yes I have seen this done).

Given that it is often a hard task for a human to figure out how to use one of these applications, and that they would need to possess all kinds of valid data to even get the application into the state where it permits file upload, I think you can see this is not going to be easy.

Add in the fact that each state change may take a sizable fraction of a second and there could be thousands of plausible vulnerabilities to check for. The driving of the application has to be done inside a headless browser process which will often sail off into space paying you no heed..

Even if the 10,000 monkeys happen to type Shakespeare, there will be a mass of false positive results in the report which a human has to trawl through.

And the scan will not complete in finite time.

Not so easy after all.

What is relatively easy is to have humans keep an eye on the applications for which they are responsible, reading the security mailing lists for the dependencies and taking appropriate action.

Why not?

Yeah that's overly dramatic. What is probably "we used row count from the old monthly backup instead of current row count of the database" turns into "OMG millions of lives!" And, of course, these lives are not "upended" in any way - some of them potentially might be if they suffer from identity theft, but surely not by a minuscule counting mistake. Bad journalism.

It depends on how they get access, and it can prevent some types of access. For example, they can’t steal backups, and they may need access through a web app server that has the database key rather than being able to go directly to the database. It’s not a catch-all solution but it is correct security practice and they should have been doing it.

Depends on vulnerability. If you have RCE (like I understand Equifax had) then encryption at rest is useless, the code can access the data, obviously, so the RCE code can too (maybe with a little more work, but ultimately it is not a barrier anymore).

THIS:

> "who is responsible for patching applications with no development team anymore?"

Struts 2.2.3 (oldest vulnerable version) was released on May 7, 2011. Personally I find it easy to imagine an app depending on that, going into prod, and just chugging away, forgotten for 6 years on some obscure corner of an enterprise's public (or internal) web assets.

You aren't their customer.

Actually it is not. There are good reasons to call outside counsel first.

Not really. A specialist law firm can give you a risk assessment, give you an overview of all relevant regulations that you need to follow, recommend security firms, connect you to any federal agencies that can help or need to know and all discussed in layman's terms (or at least exec-speak). If your CSO is good, then it's a second opinion. If he or she is not, then it's the only way to get the help you need.

It was encrypted.

(HTTPS/TLS)

Encryption is not a panacea.

If the vulnerability got to admin level, then since the database can read everything, all is lost.

Encryption at rest essentially protects the disks from being compromised if they are physically stolen. Or if the attacker manages root on the system and reads at the sector level. But even then, if you are root, you can find the key, and you are in anyway.

With cloud computing so cheap and easy, I believe no encryption is safe. Once you have the data in your reach, you can run any dirty algorithm, if the data is worthy enough. Which in this case is millions of credit card details.

> It wasn't a phishing site, Wired. It was a site designed to point out that their approach is susceptible to phishing. Entering data in the form gave you a scolding.

I'd argue it was a phishing site. Just fortunately it was controlled by white hats.

What a bizarre excuse. Do their IT teams know about subdomains?

The developer who set it up tweeted that it was a "phishing site" (albeit one that didn't exfiltrate any actual data), so I'd say that was accurate reporting.