Apache Struts Statement on Equifax Security Breach
blogs.apache.org
blogs.apache.org
Having said that, definitely keep up on the vulnerabilities of software you use. It's hard though, especially when you're relying on a great deal of dependencies. A company the size of Equifax should have had a team dedicated to this. A team. It doesn't seem like they had anyone who knew anything about basic security at all.
That doesn't mean we should be letting Equifax off the hook. But nobody should pretend that RCE on an on-prem webserver wouldn't be game-over, for the entire internal network, across the majority of the Fortune 100.
We have as a society chosen to trade the security of our personal information for greater and cheaper access to products and services. Nobody on HN will like that fact (at least, as stated bluntly like that; plenty of them do like the increase to their earnings capability that results from that tradeoff), but it's generally true.
I'm not saying you should like the tradeoff, or that the existence of the tradeoff doesn't have implications for how we should regulate banks and credit ratings agencies. I'm just saying that if you think there's some other CRA that has reliably protected its IT assets from threats like this, you're wrong.
this seems like goalpost moving. first you said 'we as a society', yet there isn't much 'we' in this. it doesn't seem to me that people (the general public) are properly aware at just how vulnerable they are. they haven't therefore 'decided' that this is okay.
> the expense that would go in to making a large enterprise network reliably secure would be extreme, and would drive prices up, retard economic growth
what is this based on?
To more directly answer your question, though, consider a regime of general commercial liability for security incidents caused by failures of engineering (which Equifax is an example of). What would it be like to bring new products to market under such a regime? How much more difficult would it be to enter the field? How much more difficult to hire qualified people? What would that do to the prices of things made possible largely through software engineering work? When the prices of things increase, they're put out of reach of more consumers; markets constrict. What does that do for employment? What does reduced employment in a sector do to wages?
I'm not saying this to justify insecurity. I'm ambivalent about the tradeoffs our society has made on these issues. I just don't think we tend to grapple with them honestly on HN.
My point is: the most savvy companies contract annual sitewide pentests, in addition to all the other stuff they do to secure their networks, and virtually all of those pentests "succeed" (from the perspective of the testers).
Further: not only have I never personally been on an internal network pentest (the kind where you start with an IP "behind the firewall" and nothing else) that failed, but I'm not sure I've even ever heard of one that failed. I'm certain there are some networks that are so resilient that they've stymied a competent internal pentest --- but again, I wouldn't know which one, because I've never heard of that happening.
And they aren't perfect. They will eventually screw something up.
Comparing Stripe to Equifax is unreasonable. Equifax has about 10,000 employees. Stripe has less than 1,000.
And, once again, please don't try to win an argument by putting me in the position of defending Equifax. That's not what I'm doing. My point is that practically every company of Equifax's size has similar problems.
Apple, Facebook and Google have far more data under their control than Equifax and they seem to take security far more seriously than Equifax. The Apple InfoSec team is extremely aggressive — InfoSec is ingrained in Apple culture. I am skeptical that Equifax takes security as seriously.
Being “big” or “other large companies have the same problem” isn’t an excuse. They have no problem fucking people’s credit by reporting inaccurate information — they should be held accountable since that information has such severe consequences.
My hatred of Equifax runs deep because they reported 7 state tax liens on my credit report that weren’t mine and were from a state in which I never lived. Despite providing all kinds of documentation, I ultimately had to sue them. I won the case and a bit of money, but the point is, they were willing to have a team of lawyers go to court against me and tie it up in court for several years — to defend something they knew was wrong. I even had letters from the state tax agency attesting to the error and they still had to be sued to fix the information. If their IT security is as arrogant as the rest of the company, then this breach doesn’t surprise me.
How is that relevant? It’s highly relevant because they have demonstrated a willful disregard for facts. They demonstrate an arrogance based on their near-monopoly power over people’s lives. They don’t care about accuracy — they care about the appearance of accuracy. It follows that they aren’t good stewards of the information with which they have been trusted.
The fact that they are too big and IT security is “hard” — that’s an excuse. General Motors might have bad IT security (I don’t know,) but they also don’t control the financial lives of almost every American.
If they are so big that they can’t handle it, then they ought to be shut down.
To be clear, I don’t have a problem with credit reporting. I have a problem with the fact that Equifax is going to survive this. Some big judgement happens, they’ll file for bankruptcy, they’ll reorganize and be right back at it.
Apple is great. I like Apple's product security team and I like Apple IS&T. But so what? I'm not denying that there are 10-20 firms in the world that have world-class security teams. I'm denying that those teams are in any sense an industry norm, even in the technology industry.
Your car kills people? Your biscuits are mouldy, or full of lead? We have strict rules, and companies that don't follow then end up in serious trouble.
The system/network I inherited at new job it absolutely would be, it won't be in a year though.
We aren't a fortune 500 but there are simple, practical things you can do to limit the damage an RCE on a webserver can actually do.
But I disagree, security teams can and should manage large networks with large numbers of stakeholders.
What's needed is an architectural security review, which is even rarer than people conducting an annual pentest.
If they can, why don't they?
Think of the standards of network security and software validation used by, say, a Certificate Authority to keep their root certificates safe. Or by Google to secure GMail. Equifax just hasn't made reaching that standard their highest priority.
:)
Google would be most people's exception to the rule I'm talking about here, having as they do one of the world's greatest information security teams. And Google has had major incidents!
You're too optimistic :(
You certainly have more experience with this than the vast majority of the rest of us, but with that kind of data on millions of people, shouldn't it be, and why isn't it, standard practice to put servers storing that data behind an extremely limited API, with a firewall in place to monitor queries and responses and severely rate-limit or stop additional traffic if something suspicious happens, until the monitoring people can take a look?
Those with data on Equifax's servers aren't Equifax's customers, so other than "not having a PR cluster&#@_" economic incentives are not aligned to prevent this.
Also... even with multiple layers, what do you think the backend stack would be? Unless it's built by a separate team deliberately using a different technology platform, you're stuck with the same Struts (or whatever) vulnerability.
Agree, and it's one of the reasons I generally try to avoid using products that are low-cost or free (the other reason being questionable sustainability, regardless of funding). At some point, products need to break even or they will go out of business. As a customer, you are always paying for the product in some form or another. For something that's free, you might be paying with your personal information now... or later if running on a low budget has caused the product developer to cut some corners when it comes to infosec.
[0]: https://web.archive.org/web/20170908214344/https://qz.com/10...
[1]: https://qz.com/1073221/the-hackers-who-broke-into-equifax-ex...
[2]: https://baird.bluematrix.com/docs/pdf/dbf801ef-f20e-4d6f-91c...
Can you please link to your preferred tutorials (or other sources) which explain how to set up the most secure server architecture, with regards to this story/subject, for those of us who would like to do a better job?
Thanks a lot!
You can do layered security, but how? Think about a customer service rep. They need access to ~everybody's data at a moment's notice because they're a CS rep. That's what they do. That means every CS rep machine has access to all the data.
I'm not sure this is solvable. SSN's are the problem, not the way that we store SSNs. They still need you to read off your SSN when you call in, so you can't just encrypt it.
Think about Equifax in particular. They're a credit bureau. Of course they need all the data available all the time -- that's what a credit bureau does. The question is, how would you partition the data so that a failure in one place wouldn't lead to exfiltrating all the data everywhere?
The trouble is, it wasn't just "they popped Struts and then ran an scp." Even if you partition the data, they popped Struts! They have RCE inside your network. It's absurdly easy to pivot from one machine to another in a given network. You keep a helpful ~/.bash_history file that says exactly what's important and where everything is installed. Stuff like this is why I argued for deleting it, since it was the #1 most helpful tool I had that helped me pivot inside a target network. But then you get people yelling about taking away their bash history.
You can imagine how terrible this problem becomes in practice. Nobody here gives a crap about Equifax; they just want their heads to roll. Unfortunately you'll be next on the chopping block because no one has solved these issues. As soon as the world goes crazy over the lack of security, it's game over for us all because it leads to a highly regulated world that won't be much safer.
See how resistant we all were to the mere mention of increasing security procedures: https://news.ycombinator.com/item?id=14104156
I was snarky in that thread which certainly didn't help, but you have an uphill battle if you want to implement anything on a large scale.
The main data store that contains everyone's credit reports (high side), should have physical and network isolation from the public site (low side). Only copy from high side to low side when someone becomes a customer.
(disclaimer: IANA security expert, but the perfect is the enemy of the good)
That is true. But you can observe that a typical CS rep accesses maybe 10 customer's data sets per hour. If they work 8 hours a day, 200 days a year, that's 16k customers they could leak per anno without arousing suspicion Maybe 60k if you allow a fudge factor of 4 to prevent false-positives from highly motivated CS reps.
But only if you monitor for this kind of thing. (And better yet, you should rate limit, and monitor).
Most security problems don't have perfect solutions, but they do have some solutions that get you 80% there. And if you invest in sufficiently many such 80% solutions, you get to a point where it's not economical for an attacker to continue hammering on your application.
It wouldn't have helped in the face of RCE in struts. RCE in struts would let the adversary scp off all the files that make up the database, for instance. This would completely bypass any application-level checks or monitoring.
https://www.computer.org/cms/CYBSI/docs/Top-10-Flaws.pdf (PDF warning) might be an interesting start.
The principles below it aren't too hard though: encrypt data in transit and in rest, think hard about key management. Keep attack surface minimal. Validate at all layers where it makes sense (for example in a multi-tenant application, you should authorize access to a row by tenant both in the application layer, and use row-level security in the DB as second layer of defense). Make your assumptions explicit; write them down. If possible, write automated tests that check if they still hold up.
While the principles are not hard to understand, designing an application with them in mind can be quite a different beast, and keeping stuff secure as features and integrations creep in requires quite some dedication.
At some point, the front end of any web application has to go to the back end database to get information. So there's just no way to cleanly and completely isolate the front facing parts of an application from possible breach.
Maybe segmentation is the only possible defence - customers A-C served out of this DB with a different stack to limit the damage.
But once they get in, they get in, right. Its just a matter of finding the next vulnerability.
As long as you are providing a rich customer experience, with the ability to generate reports based on historical information, you have risk.
I just checked some Equifax domains against SSL Labs, and while their Canadian site (https://www.econsumer.equifax.ca) scores an A-, it has no forward secrecy. I'm surprised to see a modern web server not supporting FS today. Worse, the main entry point to their Canadian site (http://www.consumer.equifax.ca) as indexed by Google does not redirect to a TLS enabled page, although they do seem to have a TLS endpoint for that domain -- but not sure how people are expected to get to it.
Edited to add: The first link is only accessible through a redirect by clicking on the "Get Started" button on their main Canadian site. Furthermore, even selecting Canada from the drop-down on https://www.equifax.com/personal/ redirects to the insecure non-TLS site.
https://observatory.mozilla.org/analyze.html?host=equifaxsec...
The attack could be way more sophisticated and involve several hacks through the stack.
Injecting a JSON blob to the Struts REST deserializer that effectively results in executing payload code would indicate high investment in the attack.
The hacker(s) possibly had insider information. They possibly used social hacking techniques to acquire data decryption keys.
Given Equifax stores more than 143M records, the initial report would imply some sort of data segmentation limited the scope of damage. If the doors were wide open, why didn't the hackers take 100% of the records?
None of this excuses Equifax or removes liability (I've personally initiated a credit freeze and will likely need a LifeLock Plus account for the rest of my life thanks to this breach).
Why does it indicate high investment? IMO it's similar to SQL injection attack. People, just take a look at points where complicated data can be deserialized and check whether adequate checks can be overcome.
As I see from their report here: https://cwiki.apache.org/confluence/display/WW/S2-052 they use XStream to deserialize REST data. XStream is basically Java serialization/deserialization to XML instead of binary. As far as I remember from my experience of working with XStream, XStream can deserialize almost any object, even the ones which aren't serializable by default. It's quite trivial to go from here to execution of arbitrary code.
Code that does what? Dumps all environment vars? OK, maybe that produces a DB2 connection string. Refactor code, re-inject, does the connection work? OK. Refactor code, re-inject and dump all tables. etc...
This attack requires an iterative trial-and-error probing process, typically done behind a proxy for many weeks/months.
From Equifax's PR: "the unauthorized access occurred from mid-May through July 2017.".
That's a reasonable amount of time to probe all systems involved and develop sophisticated code.
Of course this attack is quite laborious, but it doesn't require anything special or unique.
For most attackers, it is vastly more efficient to run automated sql/wordpress-vuln/ssh-user scanners dropping bitcoin miners and botnet nodes. Those require zero human effort and self-replicate. That gives you less publicity than a hundred million personal finance records, but it has a much higher success chance and a more reliable income.
Now a different thing is that should this information be available through a server that is accessible directly through the Internet (and I don't know if it was) or if they should require for example a dedicated VPN connection.
If you don't have a full team for this, another strategy is to rely on a vendor, and try to obtain as many dependencies as possible from this vendor. Of course, you still need a process for updating the vendor-supplied components as quickly as possible.
1. Understand which supporting frameworks and libraries are used in your software products and in which versions. Keep track of security announcements affecting this products and versions.
2. Establish a process to quickly roll out a security fix release of your software product once supporting frameworks or libraries needs to be updated for security reasons. Best is to think in terms of hours or a few days, not weeks or months. Most breaches we become aware of are caused by failure to update software components that are known to be vulnerable for months or even years.
3. Any complex software contains flaws. Don't build your security policy on the assumption that supporting software products are flawless, especially in terms of security vulnerabilities.
4. Establish security layers. It is good software engineering practice to have individually secured layers behind a public-facing presentation layer such as the Apache Struts framework. A breach into the presentation layer should never empower access to significant or even all back-end information resources.
5. Establish monitoring for unusual access patterns to your public Web resources. Nowadays there are a lot of open source and commercial products available to detect such patterns and give alerts. We recommend such monitoring as good operations practice for business critical Web-based services.
http://blog.sonatype.com/2017-struts2-vulnerability-exploit-...
This issue is solved by having your server OS download and install security updates automatically, which amounts to more or less uncommenting 1 line of config.
Edit: Downvoter(s), please comment.
What are your best sources/tutorials explaining the perfect server architecture setup which address this point?
You don't come across this too often these days in OSS or startups. It's a pretty big explosion in complexity. What hot N-tier enabling frameworks have you read about recently?
Credit rating is an instrument of control used by banks and credit card organizations to force us into debt we don't want.
This system is CRAZY and I don't understand why normally anti-establishment and "leave me alone" Americans so sheepishly agree to be involved in a system that hurts our individual liberties every day.
When I came to this country the "credit score" system was the single most incomprehensible thing about american society. Still is.
When I came to the US from Europe, I had no credit history and that made it difficult to get any type of credit card, or loan, or anything. I thought it was a catch 22 since I couldn't build credit because I didn't have credit, but there is this thing called "Secured Credit Card", where the bank lends you your own money (that you deposit) to see if you can repay yourself. Had to do that for about a year, and I also managed to get a Macy's card with a $100 credit limit (yes, one hundred) that forced me to shop there to "build credit".
After that the climbing of the ladder started with credit limits increasing slowly, etc... up to the point that I was able to get a loan for a house.
There's a bit of strategic planning to building credit worthiness, and that requires you getting into some "managed debt". I think that was the spirit of the comment you are asking about.
A large part of the credit score concept is to flag people for whom paying a trivial bill on time is difficult.
Here you get full credit worthiness based on your income, savings and loans.
The income and (by tax proxy) savings are public, so no loans and some income and moderate savings will give you full credit.
Wait, how is that?
Because it's a private system, and it's at least theoretically voluntary. Nobody forces you to get credit cards or borrow money.
Credit cards are immensely convenient. I hardly ever carry cash anymore. I buy everything from coffee to lunch to groceries to gasoline using credit cards, and pay the balance every month. If I had to make arrangements to always carry cash to cover those sorts of expenses, it would be frustrating.
I've seen a slight increase in debit card rewards, like with Yelp, airline company dining rewards, and BoA rewards, but those are for the most part random and need to be planned spends. My credit cards will always reward me for using them.
Heck, any JITing language requires process images allowed to execute code in dynamic memory segments, such that basic NoExecute hardware features can't be used. Combine this with Java server-side apps being run in a single process/address space, and I hope you can see that, if nothing else, from a security PoV Java is a dead end.
I would like to understand the thought process that resulted in hiring a Music major to run an organization in charge of protecting one of the largest troves of PII in the history of mankind.
Most practicing programmers that are called engineers are not trained in Engineering either. Computer Science is not Engineering.
There was no line of reasoning, only observation. You asked the question, I answered. I assumed you work in different place/industry and your question is honest quest to find how things work.
Yet moreover, corporate security is quite uncool to people who do academic anything (including security) or worked as security researchers. Corporate security is more about negotiation, enforcement, standards compliance and other significantly less interesting things to someone who like research or low level details.
The EE eco system while not perfect is basically a best in class for framework for everything we've learned about building business apps over the last 20 years.
Too many knobs and dials can be a big problem, and verifying you have everything set up optimally for production is tough with all that configuration present.
this seems like a tautology. too many knobs is too many.
(Whether it's true or not is another matter, and not a point I'm qualified to argue)
What style is that? The style of composing persistence technology, Authentication and Authorization, dependency injection, messaging, web communication etc to make a business application? If you know of a magical tech that doesn't aim to solve these problems in a standardized modular way so that the developer isn't socket programming from the ground up then please chip in.
"There are many moving pieces and many permutations of how they interact together."
Agreed. The problem space isn't todo apps or "Hotdog/No hotdog".
"It's practically impossible to understand it in its entirety, and thus impossible to guarantee that it's secure."
I don't understand the finer points of the myriad of TLS cypher suites etc, probably means I can't guarantee that they're secure. But what other option do I have? roll my own? Nope I'll take sensible defaults and twiddle the knobs where the client's requirements demand something different.
"You're basically plugging holes as you find them, but you're never sure that there aren't more holes you don't know about." - this is FUD.
I'm not talking about understanding the finer points of the internals such as the implementation of the TLS cypher. I'm talking about the complexity of how different moving pieces fit together in the stack.
This isn't FUD, it's the reality and it causes security breaches all the time in the real world.
For example:
https://www.google.com/search?q=rails+rce&ie=UTF-8&oe=UTF-8&...
There is a massive difference between an over-engineered WebSphere-based deployment using EntityBean EJBs (1.0 no less!) and a tight deployment of Jetty + Struts 2.0 (or more recent frameworks). Anyone who has done real Java development knows this difference. The embedded Jetty + web framework movement was the start of today's modern frameworks, such as Play and various microservices toolkits.
The Servlet spec was probably the only good thing to come from JavaEE and Struts falls into that camp, even if it's fairly dated these days. Painting it with a broad "Java EE" brush, with all the bloat that went with it, isn't right.
Sounds like you didn't even bother reading the article and jumped at the opportunity to criticize something you don't like.
The breach has nothing to do with Java EE and everything to do with Equifax' absurd network configuration.
>The bug specifically affects a popular plugin called REST, which developers use to handle web requests, like data sent to a server from a form a user has filled out. The vulnerability relates to how Struts parses that kind of data and converts it into information that can be interpreted by the Java programming language. When the vulnerability is successfully exploited, malicious code can be hidden inside of such data, and executed when Struts attempts to convert it.
>That means intruders could easily inject malware into web servers, possibly without being detected, and use it to steal or delete sensitive data, or infect computers with ransomware, among other things.
https://qz.com/1073221/the-hackers-who-broke-into-equifax-ex...
This has nothing to do with Java EE.
Where a particular set of companies falls on such a scale is also independent of this, isn't it? I feel like I'm missing something (which admittedly may be because I haven't read the linked PDF), so if you'd be willing to take the time to expound, I'd appreciate it.