Apple Confirms That Its Dev Center Has Been Breached By Hackers
techcrunch.com
techcrunch.com
This is a serious exploit on Apache Struts 2, a popular Java Web framework known as SSH (Spring, Struts, Hibernate). If you visit many websites, and see URLs ending with .action , it's probably written in Struts 2
S2-016 / CVE-2013-2251, effecting Struts 2.3.15 or lower
http://struts.apache.org/release/2.3.x/docs/s2-016.html
The exploit was deadly easy to weaponize as the Apache folks blatantly published PoC on their own bulletin
Example of server side arbitary Java code (OGNL expressions) execution
http://wp3.sina.cn/woriginal/761d2801jw1e6pqfs8hrbj20c70gomy...
Apple.com hacking was published on a popular Chinese security bulletin website
http://www.wooyun.org/bugs/wooyun-2013-023444
The issue was submitted to APPL on 2013-05-10 but ignored, and went public on 2013-06-24.
A Chinese blog on history and technical details of the exploit:
http://www.inbreak.net/archives/507
As a side note, rumor is that about 60% of Chinese goverment, e-commerce, banking, gaming websites was hacked using this S2-016 exploit, database was dumped, and exchanged in underground market. Also past records shows that the Apache Struts team is incompetent at security:
http://taosay.net/?p=611 (Warning: rant in Chinese text)
https://news.ycombinator.com/item?id=6082212
This link points to the top comment on another thread (which has more points and comments as of now) about the Dev Website outage.
I remember the last RoR exploit was quite a thing on HN few months ago. But not this Struts 2 one.
Tip: if you see some URL end with .action, it's propably Struts 2 and vunlerable to S2-012, S2-015 and S2-016 such.
Besides, apple.com was hacked as early as in May using similar Struts2 exploits.
I can't conceive of the specific use-case that justified adding this, but it's so clearly a bad a idea that it doesn't appear bad only in hindsight. It appears bad in foresight.
"I'm going to accept and execute arbitrary expressions from clients, and these expressions can interact with arbitrary Java code, is that cool?"
"No."
As we realized it could, say, invoke static functions and execute extremely complex expressions, we realized the hole. Since then the team has been patching them piece by piece, but really OGNL should be thrown out and replaced with something that is far more limited in it's capabilities.
It's been a long time since I was an active contributor, but that'd be my recommendation if I was still hacking away on it.
He said it used to take days or weeks for that to happen. Now it's hours.
Compared to Rails, PHP, what?
The sad thing is I know several competent people who work at Apple on security, although I think they do OS security not online-services. Apple has proven over and over again that they're utter crap at online services. They're going to be totally pwned at Blackhat w.r.t. iCloud in a week, too. I don't understand why they can't just drop $500mm to $2b on buying one or more competent saas/ops companies to get some real expertise in house, rather than relying on 15 years of accumulated contractor/vendor built crapware.
I don't know. Look at how huge iCloud and the iTunes store are. Every iOS device sold uses those services. They'd need to buy twitter or facebook to get people that are experts on that scale.
If you take the scale into account I don't think that their web services are 'crapware' - especially for a company where those services are only complementary to their main products.
Jobs was right -- in 10 years, iCloud is going to be central to the success or failure of Apple. There's no way I'd pick Apple based solely on iCloud so far; it's markedly inferior to what Google has put together, and actually basically inferior to what a modernized RIM BES could have been.
As far as I'm concerned, the gold standard for developer relations, including high-dollar/enterprise customer support, (at least at a large company) is Microsoft. It would probably be politically infeasible for Apple to license MSDN though.
Which is weird, because back in the 1980s (and maybe early 1990s, still), Apple was pretty awesome for developer relations. I only faintly remember it (I had Macs from 1985 at school, and from 1990 at home, and only did the most basic of stuff with ResEdit and checked out the books from the library), but stuff like their UI/UX guidelines were way ahead of their time.
What "scale" has to do with "crap" ? Considering both vertical and horizontal scaling - there is possibility that "crap" can be scaled too.
I do expect them to give some basic initial notification to developers that a security breach has happened, it was limited to x, y, z (which they should be able to determine within a day), and what initial steps are, and have some independent way for developers to contact them.
Due to the price premium for being a "winner", buying a $100-200mm (in a down market) really well executed SAAS company for $1b (now) isn't that big a deal when you're sitting on hundreds of billions and tens of billions a quarter in profit. Online services will ultimately be that important to Apple's success.
Yes. They only have the most successful online app and music store service on the planet.
And one of the biggest backup online service (iCloud) too.
Oh, and the most popular online computer shop.
Utter crap indeed.
>I don't understand why they can't just drop $500mm to $2b on buying one or more competent saas/ops companies to get some real expertise in house, rather than relying on 15 years of accumulated contractor/vendor built crapware.
For one, you have no idea what saas/ops people they have in-house. Second, you have no idea how their systems are setup.
Second, you have this baseless idea that throwing money at an engineering problem solves it (yeat, it worked great for Brooks, Mitch Kapor, and tons of other multi-hundrend million failed projects out there).
Third, who told you it's "15 years of accumulated contractor/vendor built crapware"? From the little we know, their current foundation is a cloud on top of Azure. Which is anything but "accumulated".
Like how when you launch a new app on the App Store, for the following 6-8 hours users get random errors ("not available on store", "not available in this country", sporadically missing from search, etc) when they try to download it as it propagates through the app store CDN. This even affected the recent OS X updates launched on the Mac App Store. Imagine if Amazon CloudFront worked like that!
And imagine if Amazon had to shut down their whole site when they updated one product like the Apple Store Online.
Most of that is just hype machine on Apple's part. I doubt there is a technical reason to do so.
They also use a lot of Sun/Oracle in general (especially in the early 2000s). Look how well that worked out for eBay.
Given the runaway success of the iPhone for hardware reasons (and I guess iOS, and third party developers), you can't really claim the success of the App Store is due to the quality of the App Store. The iPhone was successful first, then demand for apps, then Apple built the App Store once regular people were jailbreaking their devices and doing their own development.
I'll concede iTunes was successful for music on its own, but that's more for licensing reasons than anything else; I find what.cd a vastly superior experience as a user, even independent of money, and Gazelle/BT are open source. Both are better than what the RIAA came up with for sure. Arguably Spotify, Rdio, Pandora, etc. are better.
Netflix has done a way better job on video than Apple, too.
Anyways ignoring that, the "App Store" was built alongside iPhone and released at the same time. So I'm not sure where you're getting yo facts from.
But Netflix has done a much better job on video. And I agree that iTunes is a little weak.
I wish there were more folks who could see that. Your comments about Apple on this thread are worth contemplating.
There seems to be this silly idea that Apple is now "more than a (proprietary) hardware company". But if we took away the design-patented, hermetically-sealed enclosures, what is the value of Apple? What's left that we can use? Can any of it stand on its own?
It should be obvious, but only after you tie users to a particular piece of hardware do the other opportunities like playing middle man to digitized consumables like music, software or books arise. I surmise it is those opportunities that cause people to believe Apple is more than a hardware company. But maybe Apple is just a hardware company that, outside of their area of expertise - hardware, is very opportunistic? "Good artists copy, great artists steal."
How many times have we seen Apple "borrow" from third party software that supplements Apple's OS functionality and proceed to incorporate others' ideas into their next OS version?
Are all the components of Apple's allegedly diversified business dependent on the success of Apple hardware? Try the following thought experiment:
Subtract the fact of the Apple hardware with its attractive form factor and imagine other companies designing the world's most popular mp3 players, phones and tablets. Imagine Apple does not make, sell, or have made hardware enclosures or the cheap, Chinese-made electronics that are hidden inside. Now, ask yourself, "What is the value of Apple?"
The value of iTunes without Apple hardware?
The value of the AppStore without Apple hardware?
The value of Apple's latest Mach/BSD hybrid OS without Apple hardware? OK, but how easily can Apple's graphics layer run on other hardware?
...
Filemaker Pro for Windows? Eureka!
Yes, Apple is much more than just a hardware company.
> Yes. They only have the most successful online app and music store service on the planet.
> And one of the biggest backup online service (iCloud) too.
> Oh, and the most popular online computer shop.
> Utter crap indeed.
I really don't understand this argument. The quality of the code powering something is completely orthogonal to the amount of money said thing can generate. As long as the transaction goes through, money can be made. That doesn't mean that behind the scenes it's not a complete disaster.
Completely orthogonal my ass.
Where does any idea about the quality of code comes from? From an operational point of view, not only it WORKS, but it works in a CRAZY scale.
Hundrends of millions of customers with hundrends of millions of credit cards, serves multi-TB of stuff every day, handles music, video, apps, updates, etc. (Oh, and it's not even that which is down -- it's their developer portal).
So where does the ideas about the "quality of code" come from? Have you SEEN the code?
If not, we just have the "works, serves more than half-a-billion, at a Facebook like scale, with more credit cards than even Amazon" to go by.
Seing that the other alternatives to "standard car's internal combustion engine" are either off the market or marginal/niche.
People complained in this post that it wasn't good. As in: crap compared to the standard.
For the analogy with cars to hold, it would have to be like a seriously brain damaged internal combustion engine -- not like the "standard car's" one.
(although I think it was mainly other British marques who had horrible electrical systems at the time, and was more a 1950s thing)
Apple's developer and support forums are pretty worthless compared to other vendors. A lot of information is missing or hidden, and finding anything worthwhile requires a lot more clicks than finding something at Google.
The whole iOS approval process is...Kafkaesque.
We only know about SOME of them (e.g WebObjects).
>Those suck (which is a personal evaluation, sure, but the marketplace has pretty much validated my opinion on this).
In the same way that the marketplace has validated DOS and then Windows over UNIX? Javascript over LISP? VHS over Betamax?
Actually, WebObjects was one of the finest web frameworks (including in it's Java incarnation). Sure, there are newer things now, but nothing extremely better. Not to mention even giants like Facebook and Yahoo use PHP for christ's sake. It was also one of the most popular in its range, when it was available. It just didn't make sense for Apple to participate in that market.
>The performance/feature characteristics visible to the end user also suck.
Compared to what? In the same scale? Never hard any real issue with ITMs (at least none that I didn't have with online services 1/10 it's size).
Post a new app to the App Store. Poll various friends to see when they can actually access it. You can watch their database replication slowly happen in real time over the course of hours, as the app randomly appears for more and more people. People will often be able to access the app through a direct link but not through search for a few hours before everything synchronizes. You have to give it about twelve hours before you can reliably count on all users being able to access it.
Let's compare this to, for example, Google search. The scale on Google's end is much larger. What's worse, they're indexing external content that they have no control over. Despite these handicaps, they have no problem with rapid, coherent updates. More than once, I've typed up a reply to a comment like this, posted it, then hit up Google for some additional information and found my own comment posted just minutes ago among the top results.
Google, working with external data they don't control, serving vastly more requests of vastly more complexity, is able to provide coherent results within a few minutes, while Apple takes hours to roll out new data that you explicitly send them.
That kind of stuff is where ideas about code quality come from. No amount of "but they make lots of money and are popular" can counter this.
Whereas Google is going for absolute fastest delivery of content, Apple is trying to deliver a full ACID database system with guarantees their licensing vendors will sign off on. This likely means massive replication and it likely means that Apple is enforcing a lot of service guarantees to ensure these are atomic transactions.
On the other hand, Google just really wants you to have blisteringly fast load times. Different problem domain in my opinion. One is an index with few guarantees necessary, the other would not be obliged to sell you content if one of its guarantees failed. Different strokes for different folks.
Granted Apple could be faster, but I don't think it's fair to compare their propagation to Google's.
I could kind of sort of understand if apps took twelve hours to show up on the store, in general. But no, they take twelve hours overall to show up after a long period of bizarre, inconsistent rollout.
The speed difference is much less important than the fact that Apple presents its users with a wildly inconsistent view of their database for any recent changes for a period of hours. Of course, updating changes quickly would be one way to fix this, but the speed is not in itself the problem.
If it manages to push 1 billion apps and 1 trillion notifications, billions of songs, TB of video and such, with no major complaints other than "it takes 12 hours for an app to appear for everyone" that's doing great in my book.
No, they don't. Amazon is bigger.
The Apple Retail Stores are innovative (sort of; not that uncommon outside of computers, even in design, but bringing that to computers, sure). The Apple Online Store is pretty boring and simple -- limited SKUs, wasn't international for a fair bit, etc. The backend infrastructure to customize orders and manage manufacturing is somewhat less advanced than Dell was. It's just a question of making amazing hardware that people will do anything to buy. I don't think a random startup would have a hard time with the online store portion.
I don't think so. How many people buy computers off of Amazon?
But even if so, that's a negligible pedantic correction. It's still in the top two.
It is really unfortunate and irresponsible that data in the video is not obscured.
Taking all the data and publishing it, and then bragging about it on youtube (!) just leaves him open to prosecution, and I imagine Apple will go after him now for publishing the data of their developers.
I'm not sure I understand the logic of publishing this, but there seems to be a mentality of braggadocio among wannabe security researchers -
1. Hack high profile website and publicise it
2. ?????
3. Profit!
I'm not sure what step 2 is.
Not the brightest idea he had there but neither are the responses here auto-assuming that if the video is in English then that person must be bound by U.S law.
http://mashable.com/2013/07/22/security-researcher-apple-hac...
Says alternately that he's taken 100,000+ user records, or just 73 Apple worker records, or no user details at all. And that he's keeping all the "evidences".
Downloading 100k userdata records seems quite extreme, but is it unethical for a security researcher to do so?
It usually doesn't protect you even if you don't take anything.
That's why there's a huge backlash against "responsible disclosure".
I don't know the full details of the case but didn't the hacker weev get imprisoned for downloading 100k user records from AT&T?
I think part of weev's downfall was the subsequent conversation he had discussing what he could hypothetically do with the data though.
That likely depends on the laws of the country he was in when the incident occurred.
What really matters in cases like these is whether the country in question will arrest and extradite the individual to the US - this is far less clear cut - see the Snowden case for an example of someone attempting to evade US extradition law.
Some countries (notably China) have been fairly opaque from a US judicial perspective - see hacks on Google, the New York Times blamed on the Chinese - which have ended up with no visible action.
He refers to Facebook's whitehat list too. Facebook does allow people to try to break parts of their application if they're responsible about the disclosure. To my knowledge Apple doesn't have such a policy so can't have given any implied permission to attempt to attack them.
I wonder how out of date it was?
>updating our server software
Why now? This should be done as soon as new releases are available. There's even packages like unattended-upgrades that do it for you.
Anyone deliberately installing such a package on a production server is dangerous. Their profound lack of judgement disqualifies them from any form of access to servers I control. Updates break things. They get approved by a knowledgable human familiar with the system and installed with suitable engineers standing by, or they don't get installed.
Given that, your question of "how can we completely avoid zero-day attacks" is nonsensical.
The manpower necessary is not enormous. Any particular security update has a relatively small chance of requiring prompt installation on a particular production service. On the unusual day you're struck by lightning, you pull out the relevant emergency plan and begin executing it.
In such a situation it's probably safer to at least have automatic scheduled patching against deadly vulnerabilities and accept that occasionally that might break something.
Of course that wouldn't apply in apple's case.
If you want to completely mismanage a server you depend on for your livelihood, I can't stop you. All I can say is you're doing it wrong.
There are a lot of poorly managed VPS out there, these would be better served applying security fixes automatically.
Funny man.
More seriously, even if you have complete test coverage of your code, tell me, do you do realistic load testing as part of your automated tests? Does that include making sure the results returned by that load test remain correct? What about verifying that your backup system continues functioning correctly after installation of an update? Your monitoring system? Will your tests catch the fact that your SSL configuration just broke? Do they test your load balancer?
I could go on for a while. My career basically started with production service operations, and it's never stopped being a part of my life since. I've seen things people insisted had to be impossible even while I was staring right at it.
I have a favorite story I sometimes tell people in another context. I once wrote an email that, between my explanations of what happened and the SQL dumps proving it, spanned something close to 10 printed pages. It was, at last, real proof of a bug that I'd suspected the existence of for months, but was told had to be impossible, and couldn't be reproduced.
We were days from deploying a change in production that would have triggered this bug in a catastrophic way, and if we didn't know exactly what was going on, we would have had no way of knowing until customer complaints streamed in.
Guess why the developers and server QA never saw it?
Their machines were in the Pacific time zone. It only affected non-PST8PDT machines.
The bug was a confluence of factors, some of which were in third-party code. If it hadn't existed to begin with, it very easily could have been introduced in a package update, as this particular behavior was not a well-specified part of the package's intended behavior. Automated tests would never have caught it.
And at this very moment, somewhere in the world, there's an HN reader rushing off to make sure all their development and production machines are set to the same time zone.
Also agreed with your position in respect to the person you're replying to. :)
On my way out, I recommended one of them to replace me. Shortly thereafter, he did. Years later, I think he's actually the ops manager now. I wouldn't mind working for him, but political BS drove me out of that company, and pretty much any other company that big.
1. He unnecessarily downloaded so many records
2. He made a YouTube video to brag, showing off names and emails in the process.
[1] http://techcrunch.com/2013/07/21/apple-confirms-that-the-dev...
These tools and other static downloads (SDKs etc.) are clearly not affected by this breach - can't they rehost them somewhere in the short term?
Why couldn't they notify people about the hack, then alert them to it's specifics in due course? Piss poor excuse IMO.
While there was still wild speculation, security was only one of many possible scenarios being discussed, and it was mostly treated like a regular outage.
I'm not saying this delay in disclosure was "right" (what if it had ended up worse in scope?), but I agree with sibling post (dave1010uk) that it seems to have worked out better for their brand.
> If you weren't trying to reset your password, don't worry – your account is still secure and no one has been given access to it.
Ironic.
So? Who cares about credit card data - it so easy to just send a list of compromised numbers to the banks and get new numbers. I've been sent new cards at least twice without being told why they are changing the number, except for some nebulous "security measure", so they clearly do do this.
Of all things to worry about credit card numbers hardly rank.
I don't know how it works - but can the breach allow attackers to upload modified apps in the name of the developer?
Of course, YMMV, this is just what my bank does.
Yes. Anyplace that bills you regularly on the old number will automatically get the new number.
> You can't continue to use the old number for new transactions.
It depends on how you define new. A new biller, then correct. But if it's a biller that you have an existing relationship with, i.e. they bill you every month, then they do send them the new numbers (and/or allow them to use the old number).
I can't say if this is a global policy of all issuers, but I can say that I've experienced it. I've also seen it in the fine print when I signed up for repeated billing.
Here's an older Braintree blog post about it: https://www.braintreepayments.com/blog/automatic-update-of-c...
It doesn't look like they can. Uploading apps to the app store is done via iTunes Connect. That's always been a separate login from the developer center, and it has not been down during the current dev center outage.