Apple shut down my iCloud account for five days, no warning, no explanation
zdnet.com
zdnet.com
You build a new data backend for a service, you test it a million ways, and finally you migrate all the user's over.
Except it goes wrong. 999,999,999 user accounts migrated. 1 account failed.
Leaving the old infrastructure up for that one user is costing millions of dollars a day, hurting performance reliability and failover and holding up hundreds of other engineers projects.
Turns out the failure is because the user managed to create a filename with 67000 byte order marks in and nothing else, and it trips up some parser in the new system. They probably did it using a custom client or some kind of exploit checking tool. It's probably some security vulnerability researchers test account, but we can't be sure because we can't look into the account for privacy reasons.
Yet fixing the bug for that 1 user account is going to take a while - a quick bodge solution breaks other user accounts, and the proper fix is part of Jane's project due next quarter.
The decision is made. We're going to treat the needs of the many over the few. We're going to disable this one user account and move on. That user will have to re-upload their data, and this time they won't manage to upload a file with 67000 byte order marks in the filename.
It's not possible to anticipate every need, nor is it practical. You do the best you can to build something useful with the resources you have, ship it, and know that something will go wrong at some point. When something breaks, and it will, you fix it as best you can. If suspending your account for 5 days is the best you can do, it's the best you can do.
He has a right to complain about his horrible experience and the way Apple customer support treated him.
Testing on real user data could have been done but has privacy issues (now we can't inspect logs to check stuff is working), but even so, the user could have put the offending data in after the testing was done.
All pretty reasonable; The only thing wrong with your suggestion is it's missing the part where you contact that user and let them know what you are doing and why...
VERY FEW (two?) companies in the world have operating expenses of 1B+/year on data infrastructure. I find your story quite hard to believe after reading that.
Also, for reference: https://www.ibtimes.com/apple-amazons-biggest-customer-spend...*
2. There were security issues in some cases. But you can tell because Customer Rep would not be showing much empathy, they will tell you they have nothing / update.
3. Engineering on iCloud has been a bag of hurt since Day 1, that is nearly 10 years ago. ( Remember when Steve said why trust them when they brought us MobileMe? ) iCloud Backup corruption is still a thing even though it is extremely rare. I still dont understand how the backup could be corrupted. On the assumption they have multiple copies stored on different hyperscaler. ( I think it is Azure and AWS, not sure if they are on GCP now )
4. TimeCapsule for iOS. For Pete Sake this needs to be a thing.
5. Things like Project McQueen [1] or iCloud team infighting [2]. You could probably smell a thing or two if you were keeping an eye on turnover and hiring with Cloud infrastructure. Although that is often not a reliable indicator.
6. And in 2020, it seems Apple were finally doing something. [3] At least on paper.
7. Time to Watch WWDC.
[1] https://venturebeat.com/2016/03/17/apple-cloud-project-mcque...
[2] https://appleinsider.com/articles/16/04/21/icloud-team-repor...
[3] https://www.protocol.com/apple-hires-cloud-open-source-engin...
It will never be, because iCloud is a cash cow.
Unless something changed, they still are.
You have 500gb of files at t0, that gets backed up as Backup1.backup, then you download a 15kb text file and kick off another backup. Instead of completely regenerating a backup of all the data, it will instead recreate a new Backup snapshot, but it will still use that original Backup1.backup file and then have some data structure in place that allows you to just incrementally backup the new data while still preserving the integrity of the old files and the integrity of the previous snapshot.
Does anyone have a TM backup that has not corrupted in over a year of use?
The subtext is if you are of little to no value to the opposing party, you always have the risk of getting screwed. You will experience this a lot if you try to deal with governments.
Google, PayPal, Facebook, do it routinely...
The kind we have. This sort of thing happens all the time. Large companies -- especially large companies with monopolies -- screwing (at least some of) their customers is very much the rule, not the exception. In fact, I'd be surprised if there were even a single example of a large company that did not have customer service horror stories like this one.
Apple is particularly bad about this because they almost never acknowledge problems of any kind. They just issue regular updates that, if you're lucky, fix more problems than they introduce. Most of the time most of the features work for most of the people. But if you're among the unlucky minority for which something doesn't work then you are well and truly screwed. This is the reason I never use iCloud.
[UPDATE]
This is the nub of the problem:
> USE OF THE APPLE SOFTWARE AND ANY SERVICES PERFORMED BY OR ACCESSED THROUGH THE APPLE SOFTWARE IS AT YOUR SOLE RISK AND THAT THE ENTIRE RISK AS TO SATISFACTORY QUALITY, PERFORMANCE, ACCURACY AND EFFORT IS WITH YOU.
This is in the EULA. Everyone who uses iCloud nominally agrees to this. But of course no one really takes it seriously. Everyone thinks, "Ha ha, Apple is including this meaningless legalese in order to cover their asses, but they're not really going to be so unreliable that they're actually going to have to fall back on this to protect them. They couldn't possibly stay in business if they screw their customers, so they won't.
Except that they can, and, manifestly, they do. The reason they can is that there are no viable alternatives, with Apple and Google (and Microsoft before that) having driven them all from the marketplace. As long as they keep the disgruntled customer rate low enough that it doesn't make the evening news, what happens after that really doesn't matter to them at all. It's the opposite actually: the cost of fixing the marginal problems is more than the marginal revenue it would produce, so taking care of things like this is actually bad business.
None of this will change until people push back en masse and refuse to buy products with terms like this in the EULA.
[0] https://www.idownloadblog.com/2020/01/21/reuter-fbi-apple-ic...
> However, a former Apple employee said it was possible the encryption project was dropped for other reasons, such as concern that more customers would find themselves locked out of their data more often.
When I was reading all of this that was the impression I got, before it was mentioned.
How would device upgrades even work if Apple didn't have the decryption key? Not saying it is impossible, but I'm not aware of how it would work. I guess the user could use their password as entropy to decrypt? That's kind of scary.
This scenario has been my experience for pretty much every hosted solution.
I know Elasticsearch really well. Was working at a company that used the Elastic Cloud (not AWS) version since the demands were not high. There was something very wrong at the host level, which was evident by the cgroups metrics. The support engineers basically used the same endpoints that a customer also has access to to debug problems. No insights at the host at all. Could not ssh to a box. It took a long time to get escalated beyond the support drones. Elastic Cloud was not worth it.
Admittedly things should escalated and fixed but giving answers to end-users is not a priority.
I always thought it sounded like a good role for a company to have. Unfortunately such people seem to be in short supply.
Meanwhile I can stream a 4k YouTube video just fine.
Either sort it out, or give me more local storage space for a price that isn't exorbitant!
Obviously not a scalable solution for many, but I finally feel good about having access to my photos.
I only realised how much joy I get from being able to access my photos when I wasn't able to anymore. Back in ~2012, I lost several years worth when the hard-disk on my Acer laptop failed and I didn't keep any backups. Devastating. Backing up is something I learned the hard way.
Youtube has extensive global caching infra, both on the edge of Google's network and offnet within ISPs networks, to make sure videos are served from a location close to the user.
Also most of the videos people are gonna watch will be popular ones. I.e. they may already be filled in the cache. Compared to your icloud files which may only be accessed by you.
Follow-up: any recommendations for a beginner NAS? I'd love something cheap and simple; a 2-4 drive system with plug and play functionality would be awesome. I can live without streaming 4k videos or top-of-the-line features like that, though streaming 1080p would be nice.
I did a bit of research and wrote it up here:
https://photostructure.com/faq/how-do-i-safely-store-files/#...
I’m really only worried about my photos. Basically all of the photos and videos of my kid only exist on my phone and on iCloud, and even if I trusted Apple to not shut down my account, I’m one account compromise or dummy mistake away from a “delete” being synced and removing that content from all my devices forever.
I just have a script that runs nightly that runs icloud-pd to download all new photos and videos at full quality to the server in my basement. So that gets me a local copy totally disconnected from iCloud. From there, I have rclone encrypt and push up changes to Backblaze B2. Ends up costing me like $1.50/mo.
If you do this, make sure you put some monitoring in place. The 2FA on my Apple account tends to expire every few months and I need to go reauthenticate it.
If there’s one thing I hate about Apple, it’s dealing with their support. So I call in, and convince three levels of tech support that, no, I’m not a total moron and I’ve tried deleting the errant file through a variety of different methods on my laptop, iPad, iPhone, etc.
Finally they agree and I get stuck into a queue for an engineer to look at the issue. At this point it actually gets good- honest explanations (without details obviously) about what happened and what they need to do to fix it. Regardless, it took over two months from start to finish for the issue to be fixed.
I did push them for some sort of compensation since I had to upgrade my iCloud storage plan as a result of this file taking up a significant amount of my storage quota. They gave me $50 to spend in the Apple store on accessories. I felt that was fair.
I'm not remotely blaming the customer, but what a nothing-burger this submission is: Apple didn't "shut down" his iCloud account, they met an edge in their implementation that they had to remediate. Obviously customer support should be better and convey more information, but I picture a bunch of engineers frantically trying to deal with something they didn't plan for (cue replies claiming that the perfect programmers of HN would never have such an edge, how is it possible, etc) and things getting lost in translation.
I do find the story interesting in that he immediately filed a complaint with the BBB. That seems...risky. It's unfortunate that big corps can do this, but I wouldn't want to be perceived as a "problem" customer doing things like that when big Corps control your online existence.
Apple acknowledged they were not providing this person with the product the person was paying for, yet (presumably), no refund was offered.
> I do find the story interesting in that he immediately filed a complaint with the BBB. That seems...risky.
Why do people think BBB matters? I’ve had customers at my businesses threaten to go to the BBB and it affects business not one bit.
As to the BBB, personally I find it to be a worse-than-useless corrupt joke. But running to them, forcing Apple to respond, is the kind of thing that just doesn't have a lot of pull against large companies.
It's a generational thing - BBB tends to be respected by people who reached maturity before the internet became big. There was no easy way to complain or vent about business back then, so the BBB actually had clout... With businesses that actually paid to be a member... So, still not that useful, even then.
Source: oldster from back when folks naively thought a complaint to the BBB made any difference, and a former owner of a business that was a member of the BBB.
Issues happen, but it is the response that companies give you that tells you how much they value your business amd how much you should trust them with critical infrastructure.
Unless they are, in which case we should get out the megaphones and warn all businesses that those expensive phones they're buying can't be relied upon.
Being a PITA on social media tends to be the only way to reach someone authorized and competent to actually resolve issues.
Sure, try to resolve it through normal support channels first, but if you're given the run-around, you want to be a "problem customer". The squeaky wheel gets the grease.
In cases where an issue inconvenienced a customer I know we were compensating them with stuff(like upgrades or discounts).
A small company really needs to try to keep each customer. In the past I worked on even smaller company I was doing support and dev , this forced me to automate support related tasks like gathering reports with detailed information, then as dev I could get user reports and fixed. This would not have worked if the application would have been fulled of bugs but it was done in a managed language so there were few bugs and some of them were not the app but the platform.
/node_modules strikes again :)
Seriously, it seems from where I'm sitting if you want a reliable phone, one that holds its value, can be re-used and resold etc, get an iphone. I used to travel internationally and old iphones just had an INCREDIBLY long value retention period.
And AppleCare+ is not cheap.
And I dont see those on the front page of HN either .
best news reader ever