Apple Services Experiencing Outage | Hacker News Reader

Apple Services Experiencing Outage

apple.com

132 pointsrock_artist66 comments

I don’t think I’ve every heard of an Apple Services outage until today.

If the outages only happen to services that aren’t used much I understand why.

Apple is one of the largest service providers worldwide. Services are their second largest revenue stream after the iPhone.

1. About 25% of their service revenue is from charging commissions in app store and other 25% of the revenue is Google paying them for search default. Other services include things like insurance (applecare) That's not exactly same type services that most of the people would be thinking about.

2. A lot of their services have less criticality (and it's not a ding at them - it's often very explicit design choice).

3. App store having hiccups or iCloud backups being delayed it's not something that will usually gather enough attention of media.

You might be amazed to know how critical Services are to functioning Apple devices. While they mostly can run offline, there are dozens and dozens of services that Apple runs that modern ecosystems require (like certificate related stuff). Other oddball things related to iCloud, APNS and the private services like iCloud relay are all extremely critical to billions of devices. Thankfully the all mostly fail open (captive portal is particularly tricky). Not saying they are as critical or visible as, say, Google.com going down, but none the less would have a very very large and visible problem if they all did go down suddenly. Thankfully, due to Apple design philosophy, most are totally decentralized and teams are given almost complete autonomy on how services are ran, which makes them a huge confusing mess but also, kind of a feature as Apple generally expects them all to fail in odd ways and the software can generally handle it.

Same... to their credit, even during the major cloud outages I don't recall Apple services going down.

If I remember right they primarily use Google Cloud with a small mix of multi-cloud stuff for iCloud

depends on where in the world you live.

If you live in a region where they operate their own data centers, you will be running on Apple data centers. If not, you're running on a mix of Google Cloud and AWS (IIRC). They used to use Azure as well, but I think that's no longer the case.

In any case, your data is encrypted (by Apple) before being uploaded to Google or AWS, and only Apple has that key. Whatever E2EE encryption you use will be applied on top of that.

Last I checked they were phasing out their own DCs in favor of cloud-provided services. Though it's been a while since I have heard anything about it, so maybe those plans got canceled. It could have also been phasing out those DCs for only the specific services and not all services. My details on the whole thing are fuzzy at best.

As far as I know, everything iCloud and Apple Intelligence runs off of their own data centers if you happen to live "near" one, but you could still be using AWS and/or Google as well.

I live near the Danish Apple data center, and pretty much all my iCloud traffic goes there, with a small fraction (<10%) going to Stockholm, which has both AWS and Google data centers, so I assume they're using both for geographical redundancy (erasure coding)

It gets a bit more fuzzy once you start moving into Movies/Music/TV/Billing/whatever as well as their backend services for the store and monitoring.

Got it, yea my inside sources were as they relate to Siri and that was a decade ago at this point.

Even downdetector.com has nothing about Apple Services being offline: https://downdetector.com/search/?q=apple

https://downdetector.com/status/app-store/

They're in double-digit numbers for complaints. Must be a new record :)

macOS unable to open any non-Apple application (twitter.com/lapcatsoftware)

2603 points by mattsolle on Nov 12, 2020 | 1292 comments

> I am currently unable to work because macOS sends hashes of every opened executable to some server of theirs and [it's down]

https://news.ycombinator.com/item?id=25074959

https://www.theverge.com/2024/5/16/24158595/imessage-down-ou...

https://www.theverge.com/2022/4/25/23041217/apple-app-store-...

But indeed it's rare.

They have a major outage every couple of years. These don't have as much impact as some other cloud outages because they only affect Apple products, i.e. they're not hosting services for other companies.

- https://www.cnbc.com/2024/04/03/apple-investigates-app-store...

- https://www.macrumors.com/2023/02/23/app-store-apple-music-a...

- https://www.the-sun.com/tech/4944089/apple-maps-down-icloud-...

- https://www.macrumors.com/2019/05/08/itunes-and-app-stores-s...

- https://www.macrumors.com/2018/03/27/app-store-outage/

- https://www.cnbc.com/2016/06/02/apple-reporting-outages-for-...

- https://www.cnbc.com/2015/03/11/some-apple-services-sufferin...

Do they offer many services for *OS apps or are they mostly independent and only service their own first party apps?

iCloud is the backend for basically any settings/file sync for normal apps.

But it is mostly used for periodic backup/sync. So you aren't going to really notice unless it is a very long outage.

Theoretically they offer quite a few; in practice, I think nowhere or almost nowhere I've ever worked has gone for those options (except my own self-published games), instead preferring a custom solution that can be shared with web and android.

Half of the world uses Apple products, how is that not the maximum possible impact?

Apple market share is about 30%

For mobile - For desktop computers, below 10% even. Given that only 73% of world population is estimated to have internet access in total, that makes it just a small fraction of world population overall

As a portion of Internet users who regularly pay for and depend on hosted services, I bet that percentage is much, much higher.

US marketshare is higher.

No website uses Apple

A sync issue in drive would barely be noticed.

You'd probably be affected if you used Sign in with Apple, same as if you used third party sign in with Google, Facebook, Microsoft, or others and their authentication was down.

Sure, I don't use/have it though.

Apple Pay, with 65M users, can rely on external content from Apple.

https://developer.apple.com/documentation/applepayontheweb/d...

That's not a website

Half the world? Not even remotely close. Not for MacBooks or iPhones or anything else. They're a niche player.

Because the consumer market is way less demanding than businesses. Hence why B2B and B2C are categorized separately

To be perfectly honest with you, I wanted to come here to say that their services (on a good day) aren't very reliable anyway.

I routinely have email issues, file transfer issues (to icloud) and issues accessing their binary notarisation service.

The only thing that works routinely well is Apple Pay, however I think that it's refreshing a key lazily in the background and does not actually need a network connection to work. Good design at least.

So when I saw that they're having an outage, I thought. "All at once this time I guess".

I'll be really open here and say that I applied for an SRE job there out of hatred because whoever is in charge of SRE/Infrastructure Operations at Apple is doing a terrible job (or has terrible circumstances).

I've never had issues at all. I use their consumer offerings (iCloud, Apple Music, Apple TV+, Apple Arcade, etc.) and I've never personally witnessed an outage or problem. I regularly transfer stuff to iCloud from my PC running Linux, and I use the email service with a custom domain that used to live on Google until Google canceled the free tier of small business and also raised prices on the paid tier.

It’s a bit hidden, for example if Apple Music isn’t working it will act as if my device is having an issue with the app playing; it doesn’t look like its a service problem because it hangs and freezes the UI for play- and pretty often I will force my mail to refresh and it will say the imap server is unreachable.

Regardless, these kinds of things tend to be somewhat regional. I’m based in Sweden.

that's a fantastic point.

Seems both TV and Music are affected, I wonder if it's some storage service somewhere just totally degraded. I love CloudFlare for their total transparency reports about outages. I wish every other company would follow the same standard.

Been using Apple Music all day today, streaming. Have yet to encounter an issue.

Must be a partial outage then? I was using Music earlier today as well. On my phone I download most of my music so I might never notice an outage though.

Good point! I might not have noticed.

I had to try a few times to download from the app store this morning. Thought it was a dns/blocking issue on my part.

Also seeing outages relating to App Store Connect, currently unable to submit builds to TestFlight.

The ASC website isn't loading correctly either.

Apple services are realtime keys to:

- activation: wiping an iOS/iPadOS/visionOS device, or reinstalling a mac after a full disk wipe

- apns: push notifications, used for realtime notifications for all apps on iOS/iPadOS, even things like Signal that do not use Apple's messaging infrastructure

- imessage: enough said

- boot ticket signing: required for any mac to do an OS update (it's serialized to the CPU's ECID)

any one of these going down for any significant period of time is going to cause widespread global economic disruption.

And this is precisely why you need decentralized systems. It's absurd that text messages are routed through Apple's servers, or that you need Apple's permission to wipe a device.

> It's absurd that text messages are routed through Apple's servers

Well, that’s because…they’re not text messages.

If they were text messages, it's absurd that text messages are often routed through two servers even!

Aren't all text messages routed through a server? I guess it's more decentralized when it comes to telecom though.

I don’t think it’s any more decentralized than Apple if you are talking about MMS (instead of SMS). Your carrier still provides the MMS servers.

> Aren't all text messages routed through a server?

For SMS, no not a central server for all carriers. When SMS service originally launched it wasn’t even cross-carrier.

MMS acts like a mailserver at each carrier, sends a push notification and your device has like a day to download full message.

For RCS it’s supposed to be federated but carriers gave up and it’s now centralized through Google Jibe.

You need Apple’s permission to wipe a device so when you steal a device, you can’t wipe it and sell it. It’s part of Apple’s ongoing efforts to prevent stolen devices providing any value when stolen.

If this is mandatory, then it's not really your device.

Well, the alternative is that it's not your device either when someone steals it, it's now their device. The average user cares far more about that

If you can even opt-out of that, this is basically a nanny device.

Did this affect notifications/app updates? I found that podcasts from my third-party app were not showing up for me when they normally do. I would assume that when I manually refresh the list, that wouldn't rely on Apple servers, but given this outage maybe there's a linkage?

Sorry, it's my fault, I migrated my playlists over to TIDAL earlier today

I wonder about APNs and Apple Business Manager. I've heard from people seeing weird stuff happening on those products but I don't see it on the report here.

This outage was affecting developer.apple.com and pushing iOS app versions to test flight/production. I thought I was going crazy, thanks for posting this.

Curious to see the Testflight/Support apps updated just after. I’m betting they pushed an upgrade to something in their services and it was bumpy.

I'm getting a lot of 500 errors when trying to upload my app to App Store Connect. Also the website is loading slowly.

Failed to install/download newest xcode after updating Mac.

Guess this explains it.

Does Apple internally do something like COE's like Amazon?

Looking forward to the post-mortem

I'd be really surprised if Apple were forthcoming. Apple famously holds all cards close to the chest, so I dont expect anything from them. Happy to be wrong though.

Noticed this too earlier today. App Store apps not downloading.

Everything is working 100% for me

Reply on news.ycombinator.com