Outlook.com DNS records disappeared | Hacker News Reader

Outlook.com DNS records disappeared

outlook.com

132 pointssmcleod92 comments

:-( something went wrong Sorry, we can't get that information right now. Please try again later. If the problem continues, contact your helpdesk---dig outlook.com; <<>> DiG 9.8.3-P1 <<>> outlook.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 62938 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0;; QUESTION SECTION: ;outlook.com. IN A;; Query time: 33 msec ;; SERVER: 10.51.20.100#53(10.51.20.100) ;; WHEN: Fri Nov 22 09:37:02 2013 ;; MSG SIZE rcvd: 29

Open articleView on HN

I especially loved their status page during the outage:

  current health
	CRM	No issues
	Exchange	No issues
	Identity Service	No issues
	Lync	No issues
	Office 365 Portal	No issues
	Office Subscription	No issues
	Rights Management Service	No issues
	SharePoint	No issues

aka "works fine from my machine"

That's an executive console.

Edit: It's back up now.

Yup, it's pretty broken, though it is still up some places.

Outlook.com:

  Grapevine TX, United States (bigguy.gte.net) - Down
  Sacramento CA, United States (CalWeb) - Down	
  Providence RI, United States (Verizon) - Down
  Pasadena CA, United States (Mindspring) - Down
  Mountain View CA, United States (Google) - Up
  Vancouver BC, Canada (Radiant) - Down
  Recife, Brazil (Hotlink Internet) - Down
  London, United Kingdom (Legend Comm) - Down
  Lille, France (Nordnet) - Down
  Merzig Saarland, Germany (Probe Networks) - Up
  Milan, Italy (BT Italy) - Down
  Ankara, Turkey (TTNET) - Down
  St. Petersburg, Russia (Uni of Tech & Design) - Up
  Karachi, Pakistan (Supernet) - Down
  Delhi, India (Tikona Infinet) - Down	
  Bangkok, Thailand (TOT) - Down	
  Petaling Jaya, Malaysia (Clear-Comm) - Down
  Beijing, China (China Unicom) - Down
  Sydney NSW, Australia (Exetel) - Down
  Collingwood VIC, Australia (Pacific Internet) - Down
  Auckland, New Zealand (ICONZ)  - Down

https://www.whatsmydns.net/#A/outlook.com

Did something change yet? I'm in Vancouver and have no problem going to outlook.com... that's what is down, right?

outlook.com, xbox.com, microsoft.com, live.com, ...

Pretty much all sites are down for me (Germany).

It depends on what your local DNS has cached. Also, I’m finding that 1 out of 5 of their DNS servers is still responding from my location (not too far from Vancouver).

Based on what I have heard, the issue was caused by a bad active directory GPO that caused firewall rules to refuse DNS requests. Neither outlook/xbox/etc would have had any control over it.

I heard that the issue has since been solved internally from 4.30 PST, though DNS changes often take time to propagate externally, taking TTL settings and ISP caches into consideration.

They did release a new version of their Active Directory service in Azure yesterday. Wonder if its related.

Yeah, dropping all your NS records from your domain will do that;

  $ dig NS outlook.com

  ; <<>> DiG 9.9.3-rpz2+rl.13214.22-P2-Debian-1:9.9.3.dfsg.P2-4 <<>> NS outlook.com
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 8538
  ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0,   ADDITIONAL: 1

  ;; OPT PSEUDOSECTION:
  ; EDNS: version: 0, flags:; udp: 4096
  ;; QUESTION SECTION:
  ;outlook.com.            IN    NS

  ;; Query time: 28 msec
  ;; SERVER: 10.51.20.100#53(10.51.20.100)
  ;; WHEN: Fri Nov 22 09:58:48 EST 2013
  ;; MSG SIZE  rcvd: 40

SERVFAIL doesn’t mean they dropped the records (that would be NXDOMAIN). It does mean their nameservers aren’t responding.

...which can be verified (as I did earlier in another thread) by querying the nameservers directly.

https://news.ycombinator.com/item?id=6778272

Apparently it was due to Azure dying: http://www.theregister.co.uk/2013/11/21/azure_blips_offline_...

No, backwards. Azure outages were caused by DNS problems. http://gigaom.com/2013/11/21/another-bumpy-day-for-windows-a...

https://twitter.com/scottgu/statuses/403670409406468096

Do they never learn? I am suddenly remebering distant flashbacks to 2001 when microsoft wiped out hotmail for 24hrs with a massive DNS screw-up:

CNET: Microsoft blames technicians for massive outage (2001) http://news.cnet.com/2100-1001-251427.html

Wired: How, Why Microsoft Went Down (2001) http://www.wired.com/science/discoveries/news/2001/01/41412?...

Can you imagine in those days, everyone was on hotmail, gmail hadn't even been invented yet.

I'm curious about the cause, but it's funny to see "Do they never learn?" + links from 12 years ago.

And that's why I run my own mail server nowadays. Privacy, control over data, and at least I'm responsible for my own downtime without impacting more than a few people. Decentralisation for the win, in my opinion.

Also credits to my ISP for allowing hosting of my own server, running native IPv6 and of course opening port 25. Xs4all is a really cool ISP to have in the Netherlands :)

Xs4all is a really cool ISP to have in the Netherlands :)

I heard they had gone downhill recent years, resembling more of the regular commercial ISPs, but good to see they still have something going for them.

We definitely need more ISPs like this.

Who's scroogled now?

http://www.microsoftstore.com/store/msusa/en_US/cat/Scroogle...

This is so embarrassing.

this has nothing to do with the scroogled campaing though...

Correct, your data being inaccessible isn't the same as being Scroogled (per MS)

You could argue that it's MORE secure if even you can't access it. But I wouldn't argue that.

Maybe if Microsoft wasn't wasting its time trying to spread slurs about competitors, it could actually spend that time creating interesting software, building viable new products, and making sure it doesn't f*ck up its own DNS records. Or maybe someone's making that point for them ...

You're really reaching now.

Yes, I'm quite sure the marketing team in charge of the Scroogled campaign has the technical chops to prevent a DNS outage.

Judging by their revenues, someone must find their software interesting, or at least useful.

They just haven't finished burning through the world's biggest lead.

Ah, so they had a huge lead and they are losing it, I see.

Oh wait:

http://www.microsoft.com/en-us/news/press/2013/oct13/10-24fy...

REDMOND, Wash. — October 24, 2013 — Microsoft Corp. today announced revenue of $18.53 billion for the quarter ended September 30, 2013. Gross margin, operating income, net income, and diluted earnings per share for the quarter were $13.42 billion, $6.33 billion, $5.24 billion, and $0.62 per share.

I chuckle at how a lot of anti-MS zealots are absolutely mystified at how non-anti-MS simply can possibly be so delusional.

Wow. That was a reach.

Yes, ALL the developers (judging by your comment, who are no doubt better coders than you) at Microsoft are working on the new marketing campaign. Jesus christ, viable new products, perhaps they should just start making beer delivery apps on the rails stack, that'll prove themselves right? That 4chan thread the other day was exactly about guys like you.

Fair point. My original comment was poorly worded; I obviously didn't mean to imply any of the things you're claiming here. I merely (over)reacted out of extreme frustration with a marketing campaign that I can barely believe someone at Microsoft sanctioned. Whatever any of us think of Microsoft (I'm obviously not their biggest fan), you have to admit that this is a low, LOW move, and something that probably never would have happened were Bill Gates still in charge.

Hubris -> Atë! (not that any of you youngsters would understand that reference)

Wow, didn't expect they were selling merchandise. This is a new low even for Microsoft. Microsoft were never considered to be a company which puts ethics over money, and this just shows that they are jealous. They would do worse if they were in Google's place.

But selling merchandise? Come on, really! Is there at least one person in the world who could use these items and make it seem like they are making a fair point?

Jealous? To me it shows aggressive brand management. I've never really seen Microsoft as a company above ribbing on competitors. Why is it even a bad thing? Apple did it against PC for like a decade.

Apple had a point, that they were cooler. They were saying, "hey, we are different". How does what Microsoft's saying about Google not apply to it? MS doesn't care about your privacy any more than Google. With the Snowden revelations and all, they have no case to make.

I disagree. Microsoft is a company who makes most of their money selling software, and hardware and services play a small part. They do almost no advertising.

Google is a company who derives about 99% of its revenue directly from its advertising business, of which utilizing user data from free services is the crux of the data-based approach that drives their success.

It seems like false equivalence to me. Same reason why I trust Apple way more than Google with data. Because Apple makes all their money on the hardware, not on the future use of your data for advertising purposes.

Most aggressive marketing/brand management generally doesn't involve selling products saying "[competitor] is bad and evil!!!"

Commercials sometimes, sure, but not wearable merchandise. I think it makes Microsoft seem quite petty, especially since they're trying to coin a whole new word out of it ("Scroogled").

Because Google never has outages.

Totally off-topic: Based on your nick, I may know you. Live in Northern Illinois by chance? With a short stint at a place near Oak Brook?

Looks bigger than this. It's ns*.msft.net.

A friend of mine working at Microsoft mentioned this afternoon the Redmond campus had a full network outage.

I was unable to resolve any hostnames from my Azure machines from 4:20-4:40 Central time. I'm assuming this is all related.

This about sums up how I feel about Azure: http://i.imgur.com/ESakkWk.png

Wow. Your screenshot is giant.

Cmd+shift+4 on a retina MBP. Not sure why it does that...

It's because the metadata saying to render the image @2x is lost when imgur serves the PNG back to our browsers.

I got a 404 error on my signup for Azure instead of the "completed order" landing page. Then I got about a dozen phone calls asking me how my experience went with the service stuck in some half setup state.

Google's DNS cache will drop them in 250 seconds...

It looks like a lot of their other services are down too.

Edit: Also, the Xbox One is launching tonight so if their Live servers are down...it won't be pretty.

Isn't it launching in Europe in just a few minutes?

Ah, maybe it had something to do with the Xbox One launch. Updates, under pressure, mix and serve.

Xbox Live is down. Just posted on Reddit: http://www.reddit.com/r/xbox/comments/1r6axx/anyone_else_not...

What a bad time for this to happen for Microsoft. It's almost too badly timed for it to be a coincidence, if I were an attacker with access to Microsoft's DNS' this would be the perfect time to launch an attack.

Will negative caching of DNS records prevent a quick recovery? If so, for how long will the negative records be cached?

Damn the guys at the NSA are trying to get on xbox live.

This probably explains an error we were getting on a Node.js app running on Azure websites. Simply creating a new AWS.S3 instance was throwing this: Error: getaddrinfo ENOTFOUND at errnoException (dns.js:37:11) at Object.onanswer [as oncomplete] (dns.js:124:16)

It'll be interesting to find out the root cause, assuming it ever comes out. Malice or stupidity? Did someone do an oops within the Outlook team (or some other team), or are they under attack?

My guess is someone messed up when making a change to their DNS

DNS problems are at least partly affecting the microsoft.com homepage as well.

My guess is they changed DNS so they could keep the service up while cutting off an attacker's route.

I've heard from people know that an AD GPO was pushed causing firewalls to refuse DNS requests, which would fit in line with what everyone was/is seeing.

unreal, i was editing a document, when it bombed on me, guess i should been editing locally!

Oh man, this is epic.

Pro tip for anyone trying to check if a DNS record is working: do an NS lookup in cmd (Windows) in interactive mode. Set the server to the IP of a root DNS zone network (you can find these on Google) then enter that. Now do an nslookup for the domain you want.

Actually no. The first step would be to go directly and query the nameservers set in the whois record (not the registrars which could be wrong but the registry).

For xbox.com that is:

Name Server: NS1.MSFT.NET Name Server: NS2.MSFT.NET Name Server: NS3.MSFT.NET Name Server: NS4.MSFT.NET Name Server: NS5.MSFT.NET

While looking at this, I saw some other temporary DNS fails as well, including HN, Google News, and Twitter. But all came back quickly, except for Outlook/Microsoft domains. (I'm in Minnesota)

I have my domain at outlook.com and my MX record points to edcb3f68479b498ee412acc8524def.pamx1.hotmail.com and I am getting no return for that. There is a big outage somewhere

Not a good day for this to happen. Xbox Live is impacted as well.

It's working just fine for me. I use Outlook.com mostly as one of several back-up email accounts. I tested it by sending a message to my Gmail account, and all is well.

Yeps, a lot of Microsoft services are down. I'm in europe and can't reach outlook.com, microsoft.com, windowsazure.com and probably a whole bunch of other services

I have no problem accessing Outlook.com, Office 365, Skydrive, or any other Microsoft website. The problem many be regional...or it may be with your ISP.

There's a lot more than one person complaining on twitter. I've seen reports from New Zealand, Kansas, my linode in Dallas, Oregon, etc.

Yup. I'm in Iceland and every microsoft service is unreachable, except my regional version of microsoft.com

Australia too.

Yep - http://mail.office365.com/ is down for me in New York.

' Grapevine TX, United States (bigguy.gte.net) - Down' So we definitely didn't hear it down the grapevine?

It reminds me of the following quote - don't attribute to malice that you can attribute to incompetence.

Technet DNS entries was gone for mediacom DNS servers about an hour ago in Northwest FL.

I was having some issues connecting to technet earlier, perhaps related?

Down when accessing from Central FL (SouthEast US).

It's up in North FL

status page https://status.live.com/history

Yep, Xbox Live is down here in the UK.

Back up already :)

CodePlex is down too? (in AUS)

oh no. Where will I go now to find information on gentleman enlargement pills?

Can confirm Sydney is down :(

Still up in Toronto

Down in Melbourne

Back Up

Up in Melbourne Australia (on AARNet).

Down in MN.

is up in Pune, India

fixed in japan

Reply on news.ycombinator.com