Ask HN: Is Apple down? | Hacker News Reader

Ask HN: Is Apple down?

253 pointscrgt106 comments

https://developer.apple.com doesn't work App Store doesn't work iMessage doesn't work. Not just me - coworkers also struggling.

Any idea what's going on?

View on HN

    nslookup
    > server a.ns.apple.com
    Default server: a.ns.apple.com
    Address: 2620:149:ae0::53#53
    Default server: a.ns.apple.com
    Address: 17.253.200.1#53
    > developer.apple.com
    Server:  a.ns.apple.com
    Address: 2620:149:ae0::53#53

    developer.apple.com canonical name = developer-cdn.apple.com.akadns.net.
    ** server can't find developer-cdn.apple.com.akadns.net: REFUSED

Ah. So Apple's own DNS servers are redirecting developer.apple.com to something on "akadns.net", which is operated by Akamai. But Apple's own DNS servers refuse to resolve that, probably because it's not in the apple.com zone.

    nslookup
    > developer-cdn.apple.com.akadns.net
    Server:  127.0.0.53
    Address: 127.0.0.53#53

    Non-authoritative answer:
    developer-cdn.apple.com.akadns.net canonical name = world-gen.g.aaplimg.com.
    world-gen.g.aaplimg.com canonical name = apple-c.g.aaplimg.com.
    apple-c.g.aaplimg.com canonical name = apple-cf.g.aaplimg.com.
    apple-cf.g.aaplimg.com canonical name = apple-lr.g.aaplimg.com.
    > server a.ns.apple.com
    Default server: a.ns.apple.com
    Address: 2620:149:ae0::53#53
    Default server: a.ns.apple.com
    Address: 17.253.200.1#53
    > developer-cdn.apple.com.akadns.net
    Server:  a.ns.apple.com
    Address: 2620:149:ae0::53#53

    ** server can't find developer-cdn.apple.com.akadns.net: REFUSED

It's clearly a botched DNS configuration. Not clear what the intent was. Did they really want to point "developer.apple.com", a web site, to "developer-cdn.apple.com.akadns.net", which is a DNS server? Or am I misreading that?

It's generally considered bad form to have all the DNS servers for "example.com" under "example.com", by the way. If you mess up "example.com", or it goes down, getting to it to fix it can be difficult.

Anyway, this looks like an attempt to outsource something to Akamai that went badly wrong.

I think something with DNSSEC:

https://puck.nether.net/pipermail/outages-discussion/2022-Ma...

I noticed a few weeks ago that developer.apple.com was failing DNSSEC and that this had been going on for a while (follow the "previous analysis" links to see earlier errors as well):

https://dnsviz.net/d/developer.apple.com/Yidc2Q/dnssec/

It doesn't seem like many people have noticed or cared, so I doubt many people use DNSSEC at all and the whole system could (and should) be scrapped one day with barely anyone noticing.

lima has an anaylsis of the issue causing trouble:

https://news.ycombinator.com/item?id=30757487

APPLE.COM isn't signed at all; this isn't a DNSSEC issue.

In the future, if you want to check if something is DNSSEC-signed (things rarely are: DNSSEC is overwhelmingly not enabled on the commercial Internet), you can just `host -t ds <domain>`.

I noticed it because developer.apple.com failed validation using systemd-resolved with DNSSEC enabled when someone posted a link on HN (but worked fine with DNSSEC disabled). It still does. The main apple site doesn't have that issue (the post I linked to gave the general, non-DNSSEC related issue this time).

I tried several local utilities and options but couldn't find a reliable way to determine if a site would resolve under systemd-resolved with DNSSEC enabled other than using systemd-resolve with DNSSEC enabled. It seemed like any time dnsviz.net shows an error the domain will not resolve, but some things it shows as warnings also cause sites to not resolve while other warnings do not. My favorite is that Verisign's DNSSEC validator's domain fails to resolve with DNSSEC enabled.

Possibly some or all of this is systemd-resolved doing the wrong thing, however the errors and warnings on dnsviz.net make me think this is not the case. www.google.com, for example, does not show any warnings or errors.

GOOGLE.COM is also not DNSSEC-signed. Seriously, almost nothing is.

Right, but my point is "not DNSSEC-signed" does not seem to be the same as "free of configuration errors that prevent resolution of the name with DNSSEC enabled".

Which configuration errors would those be? Without a DS record, there's no DNSSEC happening at the resolver, is there?

I tried looking again and found that it is systemd-resolved's error at least in the developer.apple.com case (the Verisign one is a bit different but potentially might also be a systemd-resolved issue). It seems the issue is that the servers for g.applimg.com are completely DNSSEC-unaware and querying the DS record somehow doesn't work the way DNSSEC wants it to even in the "no DNSSEC" case, however the parent zone correctly indicates that there is no DNSSEC so it should be accepted.

https://github.com/systemd/systemd/issues/9867#issuecomment-...

It sounds like systemd-resolved has had a bunch of issues like that where it fails (or previously failed) on things that would be an issue if DNSSEC was enabled but shouldn't due to DNSSEC not being used. I'll stop blaming DNSSEC.

AAPLIMG.COM isn't DNSSEC-signed either.

Yeah. Was wondering if they'd enabled it and backed off when it didn't work.

Works with other DNS servers.

  $ nslookup developer-cdn.apple.com.akadns.net a.ns.apple.com
  Server:  a.ns.apple.com
  Address: 17.253.200.1#53

  ** server can't find developer-cdn.apple.com.akadns.net: REFUSED

  $ nslookup developer-cdn.apple.com.akadns.net 1.1.1.1
  Server:  1.1.1.1
  Address: 1.1.1.1#53

  Non-authoritative answer:
  developer-cdn.apple.com.akadns.net canonical name = world-gen.g.aaplimg.com.
  Name: world-gen.g.aaplimg.com
  Address: 17.253.121.201
  Name: world-gen.g.aaplimg.com
  Address: 17.253.121.202

Now also works on their authoritative DNS servers again: https://www.nslookup.io/dns-records/apple.com#authoritative

> Did they really want to point "developer.apple.com", a web site, to "developer-cdn.apple.com.akadns.net", which is a DNS server.

It's just a CNAME, meaning go look that up. It does not indicate that developer-cdn.apple.com.akadns.net is a DNS server.

The above seems to indicate that somewhere in the chain of resolving developer-cdn.apple.com.akadns.net, a DNS server refused the query. A dig +trace should indicate which.

This looks like an Akamai DNS load balancing solution. It will route a user to an endpoint based on a bunch of statistics (think location, availability, latency, and/or load), and will often handle caching and DDOS protection as well

> Or am I misreading that.

Yes:

    developer.apple.com. 73 IN CNAME developer-cdn.apple.com.akadns.net.
    developer-cdn.apple.com.akadns.net. 73 IN CNAME world-gen.g.aaplimg.com.
    world-gen.g.aaplimg.com. 13 IN CNAME apple-c.g.aaplimg.com.
    apple-c.g.aaplimg.com. 8 IN CNAME apple-cf.g.aaplimg.com.
    apple-cf.g.aaplimg.com. 8 IN CNAME apple-lr.g.aaplimg.com.
    apple-lr.g.aaplimg.com. 14400 IN NS b.gslb.aaplimg.com.
    apple-lr.g.aaplimg.com. 14400 IN NS a.gslb.aaplimg.com.

The Akamai CNAME just points to a series of aaplimg.com CNAME (eventually ending up with apple-lr.g.aaplimg.com), which is Apple's own CDN domain. The CDN's resolvers (a.gslb.aaplimg.com and b.gslb.aaplimg.com) refused to serve A records for apple-lr.g.aaplimg.com.

They fixed that and now it's back up.

This kind of setup is typically done for flexibility reasons (geographical DNS load balancing or similar, where the Akamai DNS servers serve as the geo LB).

> It's generally considered bad form to have the all the DNS servers for "example.com" under "example.com", by the way. If you mess up "example.com", or it goes down, getting to it to fix it can be difficult.

Not necessarily - this is what glue records[1] are for. Many large companies host their authoritative DNS on the same domain, it's not a bad practice when done carefully.

[1]: https://ns1.com/blog/glue-records-and-dedicated-dns

Can we refer to this as “Doing a Facebook?”

This has nothing to do with the BGP failures that FB had earlier. This is a DNS configuration problem. It's much simpler to fix.

Yep.

Wife: My Apple Maps isn't working.

Me: Hmm, it's not working for me either. They must be having server problems. You should use Google Maps for now.

Wife: I can't download Google Maps either, the App Store doesn't seem to be working.

Maps, App Store, iMessage on macOS works

They work on iOS as well - so it seems to be a regional thing?

(Location: Germany)

I agree. Regional.

Guessing the issues are centered on North America.

Maps and iMessage are working for me in Canada, but not music.

On my side in France apple Map only partially work. Basemap are displaying correctly but query and routing function are unreachables. "Domain name not found" (translated from french). So it could be a DNS meltdown?

Usually basemap because they are heavy are served through a separate CDN.

Everything in the App Store was working for me except actually downloading apps. Seems to be (mostly) resolved now.

for me the search function also refused to work, but the start page loaded without a problem... downloading an app still seems to not work though...

Yeah I posted about that here (this just bit me) https://news.ycombinator.com/item?id=30757193 and I was flagged to oblivion.

Looks like I really need to keep a 3rd party nav app installed just in case!

Always. Here maps is a good backup solution. It allows you to download pretty much the entire world - if you have the space in your phone.

I always use Apple Maps, but once in a while if I'm in an unfamiliar city and the Apple Maps directions seem suspiciously weird, it is useful to have Google Maps app for a sanity check. (directions to a particular pier at the Seattle waterfront were insanely incorrect via Apple)

Try Organic Maps - offline-first OpenStreetMap app. It's really good!

Both Apple Maps and Google Maps work in the browser, no need for an app.

I would not say need, but the connection from the device to carplay is really nice.

For turn by turn?

Probably true, but we wanted it for driving directions via CarPlay and were in a bit of a rush. The car's built-in navigation (which we otherwise never use) ended up working fine, but the browser versions probably would have been my next attempt.

Can you use the web version of Google maps?

Yes. I have been doing that for quite some time.

Same with my Apple Maps over the course of an hour this morning.

Rough order of events:

1. Not working (could not find server)

2. Not working (request timeout)

3. Restart app

4. Working

Perhaps DNS was broken for awhile and restarting the app cleared the DNS cache and forced a fresh IP lookup?

You wouldn't think it if you went by this:

https://www.apple.com/support/systemstatus/

That is just a static github page with html. These are just green dots on a screen.

Well make sense to me to host your status page outside your main infrastructure.

A lot of system status pages are updated by humans who will verify issues before reporting them. Main reason is to avoid overly surface every minor and transitory issue to public view.

Quite easy to verify if the entire developer site is down though, non?

It's very easy, except when it's hard. Also, it's never easy.

Joking, but only somewhat. That's because the easy cases are handled by automation, etc. If you knew it could happen, you probably planned for it. Figuring out what the issue is, if there really is an issue, and the scope of the issue can take some time.

No. “Doesn’t respond for me” doesn’t imply “down for lots of people”. If you discover that foo.com doesn’t respond, it takes a while to figure out whether that’s on your system, in your network, in the city, etc.

Yes, you would set up multiple hosts across the world polling that server, but that adds complexity. Maybe, those pollers decide the site is down because of a bug in your network setup, while the rest of the world happily uses your services.

Pingdom seem to manage it. Pretty sure one of the FAANGS could to. I appreciate an obscure managed service might be a bit diffcult, but main developer site?

My response was to “Quite easy to verify if the entire developer site is down though, non?”

I never claimed it’s impossible, just that it isn’t “quite easy”, especially to check that the “entire developer site is down”. The home page may be down, with the rest being up, the home page may be up, with the rest being down, etc.

I’m sure it wasn’t when you posted 10 minutes prior, but FWIW currently listing 11 outages:

> App Store - Outage Today, 12:32 PM - ongoing Some users are affected Users may be experiencing intermittent issues with this service.

Apple Arcade - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.

Apple Music - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.

Apple TV+ - Outage Today, 12:32 PM - ongoing Some users are affected Users may be experiencing a problem with Apple TV+. We are investigating this issue.

iTunes Store - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.

Podcasts - Outage Today, 12:32 PM - ongoing Some users are affected Users are experiencing a problem with this service. We are investigating and will update the status as more information becomes available.

Radio - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.

Apple Business Manager - Outage Today, 1:14 PM - ongoing Some users are affected Users may be unable to sign in.

Apple School Manager - Outage Today, 1:14 PM - ongoing Some users are affected Users may be unable to sign in.

Device Enrollment Program - Outage Today, 1:14 PM - ongoing Some users are affected Users are experiencing a problem with this service. We are investigating this issue.

Schoolwork - Outage Today, 1:14 PM - ongoing Some users are affected This service may be slow or unavailable.

Looks like it’s been updated. Currently showing 11 services down, some of which have been down for over an hour.

At the bottom of that status page, it says: Looking for developer system status? Find it here: https://developer.apple.com/system-status/

The link is currently not working...

Looks like that link is working now...and shows a page of all GREEN :-)

18 red as of 2:50PM EDT

Here's a lot of crowd-sourced anecdata points:

- "Multiple Apple services are down such as: (Will be updating this list)"

https://old.reddit.com/r/apple/comments/tjg8tz/megathread_ap... ("[Megathread] Apple Outages")

I chose the perfect time to restore a repaired iPhone, don’t seem to be able to fully login to iCloud, it’s hanging on the login screen…

Edit: It’s also refusing to download any apps, doesn’t even show the progress circle. Just a download icon next to the app name on the Home Screen and errors out when you click it.

Edit: Login and app downloads now working as of 6.00GMT

It's times like this that force us to remind ourselves how reliant we are on critical services like these. On one hand, we can celebrate (Internet snow-day!) but on the other we are forced to shop around for alternatives too.

I often wondered how medieval the world would become if there was a huge sun flare ejection that breached the magnetic field and destroyed a bunch of data-centers. Think of the mess we'd be in!

I’m sure it’s happens more than I’m aware but i have to say that i can’t recall an App Store outage since i got back in the platform 3-4 years ago. Not bad!

I picked a terrific time to lose my temper and do a `rm -rf /Library/Developer/CommandLineTools ; xcode-select --install` /facepalm

Looks like their DNS servers are responsive, but refuse to serve records:

    $ dig -t NS developer.apple.com
    [...]
    apple-lr.g.aaplimg.com.     14400   IN      NS      b.gslb.aaplimg.com.
    apple-lr.g.aaplimg.com.     14400   IN      NS      a.gslb.aaplimg.com.

    $ dig @a.gslb.aaplimg.com developer.apple.com
    [...]
    ;; ->>HEADER<<- opcode: QUERY, status: REFUSED
    ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
    ;; WARNING: recursion requested but not available
    ;; WARNING: recursion requested but not available

Most likely a configuration mistake that'll be undone as soon as they figured out how to re-deploy their DNS servers while DNS is down.

Unlikely to be BGP shenanigans as some people on Twitter claim. My network has direct peerings to Apple's AS714.

Definitely. Downdetector shows a bunch of reports too (e.g. https://downdetector.com/status/apple-music/). I noticed issues with Music and News, seems like a ton of their services are down

Downdetector has predicted about 50 of the last 3 outages, and linking to them here just makes the self-fueling cycle even worse.

I think they’re about as useful as any anecdotal data out there. Unusually high numbers of reports when you’re seeing issues yourself is about as good as it gets until a status page is updated (which it thankfully has been finally).

They're fine for knowing that something is going on, but not great for knowing exactly what the cause is.

For example, when Facebook's services went down in October, people were reporting that AT&T and other cell carriers were down because they couldn't open the apps. As far as I know there wasn't an outage with any of the carriers that day.

Yes. Even developer.apple.com won't load at all for me. Who wants to take bets on DNS as the culprit?

MacRumors says Apple is down.

https://www.macrumors.com/2022/03/21/icloud-and-apple-servic...

Big outage... is it some stupid DNS issue again?

It could be DNS. I had to disable DNSSEC for the `.apple.com` zone to even work.

My app update was rejected because my Upgrade screen was unable to fetch prices from their servers and instead showed an infinite spinner.

Would an infinite spinner also show up if the server was up but the connection was problematic? If yes, this would be about not handling network errors, which sounds like a decent rejection reason to me.

My Apple Music stopped working mid song and is being weird now. Everything seems to be working fine for my wife. Weirdly spotty.

iCloud Private Relay is shown as affected as well. This is an interesting case when it comes to failure behavior. From security perspective, you want your connection to stop working instead of falling back to insecure. Is this the case? Can anyone confirm?

It fell back to insecure for me, for about 30 seconds (maybe longer before I noticed) I couldn’t connect to the Internet from my iPhone, then I got a notification saying private relay was unavailable and I was able to connect again.

A few minutes later it gave me another notification saying private relay was working again.

iCloud Private Relay is not designed to be a full-fledged VPN anyway. HTTPS traffic in apps (other than browsers) bypasses it AFAIK.

They seem to have been having a bit of a lie-down, today. I can't submit TestFlight builds, but now, it is taking longer, before the server throws a nutty, so I guess the fix is on its way.

MobleMe is still down.

https://www.mac.com/

https://www.me.com/

Yeah, I'm seeing anecdotal reports of a bunch of services out

App Store Connect was down for me but appears to be up again now.

I haven’t been able to cancel subscriptions lately. I filed a refund request and complaint to Apple, maybe it didn’t get through because of this?

It's a partial outage for me. I was just able to send an iMessage, but directions on Maps are not working. I live in central Texas.

I've been struggling with a DNS downtime at Mediatemple all day. Is there a possible more global DNS issue?

Yes, got a notification that Apple private relay is unavailable

And another notification that it’s back online 40min later

Yeah had to close private reley because websites didn’t load.

I had abnormal trouble pulling video I uploaded to iCloud yesterday. Something is up.

I noticed a blip in iMessage earlier, but it sorted itself out before too long.

the domain name developer.apple.com resolves through a series of CNAMEs to Apple's CDN (applimg.com), which if it was down would explain other things like iMessage also being unavailable.

Yeah, for me the CNAME chain ends with apple-lr.g.aaplimg.com, which doesn't resolve to anything

Some reports that there were DNSSEC validation issues w/ proxy.safebrowsing.apple which CNAMEs to aaplimg.com.

AAPL is down too, today.

For me in Germany: iMessage up App Store up Developer site down

It’s coinciding with an AWS outage. Probably not unrelated.

https://www.the-sun.com/tech/4944089/apple-maps-down-icloud-...

Can't upload an ipa to App Store Connect for an hour.

> Any idea what's going on?

Must be gravity. (Sorry, I had to.)

Had a few issues with the App Store with OS 12.3

Me too.

Must be regionalised. Nothing wrong here.

Down in French Polynesia.

iMessage texts are working fine for me but an image I sent to a friend is stuck. Music is also down for me.

Down for me via CloudFlare WARP

It's always DNS.

gotECC?

https://earthsky.org/sun/sun-activity-solar-flares-cme-week-...

LPDDR5 in the SoC doesn’t.

If I had just dropped $2K-$12K on a media-centric computer with the intent of running encrypted backups, spreadsheets, databases and other inappropriate tasks for non-ECC memory (looking at you Leo), I’d downvote too!

It is always DNS ;-D

Down for me.

fedora is down.

maybe their dns expired?

Reply on news.ycombinator.com