Facebook Down, Like Buttons Vanish, Internet Implodes | Hacker News Reader

Facebook Down, Like Buttons Vanish, Internet Implodes

techcrunch.com

162 pointsaresant105 comments

Open articleView on HN

They closed shop after DHH's scathing criticism.

DHH's fantasies realized.

More like zuckerberg making a statement to detractors, how important they are.

They knew the gig was up and decided to cash in before everyone realized they really weren't worth $33B.

that and the movie.

enough is enough!

Well somebody has screwed it up on their end. Only the www record is not resolving, a lot of the others are, for eg:

http://developers.facebook.com/

  $ nslookup facebook.com ns1.facebook.com
  Server:	ns1.facebook.com
  Address:	204.74.66.132#53
  
  Name:	facebook.com
  Address: 69.63.189.11
  Name:	facebook.com
  Address: 69.63.181.12
  Name:	facebook.com
  Address: 69.63.189.16
  
  $ nslookup www.facebook.com ns1.facebook.com
  Server:	ns1.facebook.com
  Address:	204.74.66.132#53
  
  Non-authoritative answer:
  *** Can't find www.facebook.com: No answer

(YMMV based on local cache etc.)

Hilariously touch.facebook.com still seems to work.

I imagine that would be a relatively easy fix, more likely they took the record down to debug internally without getting flooded with requests.

It looks like you were right - they dropped that record to stop the requests coming in

It's hard to overstate the damage this is causing for merchants that prominently display FB like on their pages - lots of big ugly blank spaces / errors out there.

Realize this is temporary but shows why if you install any external vendor tool as a part of your site make contingency plans that switch on errors.

Or people who rely on Facebook for shared login...that must be down as well.

Good point. We recently had a client set their "signup" to FB login by default and I cringed.

Great, simple services but the macro effect that this is going to have on commerce today is crazy.

You're telling me--of course this has to happen the day before client review of a major Facebook Connect site whose only authentication is via Facebook Accounts. We had discussed potentially having two types of accounts (Facebook- and email-based), but having two parallel authentication methods just seemed clunky. I guess this is the price we pay for trying to keep things simple...

This must really be hurting Quora.

Luckily, they added Twitter authentication as well. But still, now a large portion of their site cannot login because of Facebook.

Seeing as Facebook login lets you easily collect email addresses these days you could always have a page that says, facebook is down click here to get an email confirmation link to login or something similar.

Well frankly, If you rely on some other business for such critical functionality, you deserve to lose.

Unfortunately its difficult to avoid relying on other businesses for crucial systems.

Example: Payment Processing, try building a business, especially an online one, without depending on Paypal, or at least Mastercard/Visa.

Sure you can reduce it by say not having FB Connect as the only log-on option but the simple fact is that you will almost always have to depend on other businesses, the best you can do is diversify and hope for the best.

Exactly...people made this same argument a few years back when Amazon S3 went down for a few hours. Just have a few different options, and the chance that they will all be down at the same time is pretty slim.

If you have a few options, then you don't rely on a single company, do you?

Yes and No. If you have a few options than your entire business won't come grinding to a halt but if half of your customers choose to use FB Connect as their login option than they likely don't care that FB's down, or that people with email logins can still access your site. To 50% of your user-base your site has failed and so in that sense I would say that you still rely on FB for your business.

for like buttons you should iframe them (same for anything from a third-party service)

facebook uptime has been good overall, considering they doubled in size in 12 months to 550m users

Even iframed, you end up with an ugly "service unavailable" blot of text.

Do better systems exist? A few hours downtime in several years is still better than most brick-and-mortar stores, I think.

cauliflower farms everywhere are withering for lack of water.

The first 10 times I read this comment, I thought it was a pun on facebook's color scheme, until I realized that it's cornflower blue, not cauliflower blue.

It's cornflour, not flower

Huh? It is cornflower:

http://en.wikipedia.org/wiki/Centaurea_cyanus http://en.wikipedia.org/wiki/Cornflower_blue

Yes, because cornflour blue would be, well, white.

Please don't downvote me. I am not a native speaker. But I still don't get it. Someone care to explain ;-)

FarmVille.

As a non facebook user, I didn't get it either... :)

I use Facebook (sometimes), played FarmVille and Farm Town up until this January, and I STILL didn't get it right off. Guess pulling out of those games was a good idea after all. :)

My co-worker just told me his mother is level 137 at Farmville :) Apparently she spends 6 hours a day playing it...

This affects me in no direct way. I officially have a good life.

Same here, I already have most of Facebook's domains in my hosts file. I haven't seen a Like button in ages.

And that, my friends, is why its risky to build your business on top of FB Graph API.

In terms of reliability, FB is not twitter, but it certainly isn't google either...

I came here to post this same thing, but I just started thinking about it. When was the last time FB went down like this and took a bunch of core functionality down with it? Has it ever happened? And how long are they down...30 mins? Is that 30 mins of downtime every few years really more damaging than that the (potential) value that integration with the graph API or FB Connect can bring? Yes, you look like an idiot to a few users, but maybe you have 100x as many users as you'd have otherwise. I think the real reason not to build your business on the FB API (or Twitter or whoever) is strategic, not because of potential downtime.

They had a BGP-related screw-up a couple of weeks ago as well.

I fully agree with you. If your app is heavily social it would be ridiculous not to leverage the worlds largest social network, even with very occasional down-time.

You're right, I should have phrased it "And that, my friends, is one of the many reasons its risky to build your entire business on top of FB Graph API."

Just went through a nightmare building a product entirely on top of fb graph. FB fails, a lot. And not just in terms of downtime, in terms of inconsistently implemented technology, poor documentation, lack of transparency, inconsistent policies etc.

Not saying don't use FB ever, just that it's risky.

Oh, don't get me wrong. I won't do FB apps for clients anymore, despite the high demand and great pay. It's just way too stressful to have the clients always angry at me because of the platform's shortcomings.

Amen! I tell ya, I have had so much stress over the last few months as the direct result of this. Well put.

If you have 100x as many users, you will look like an idiot to 100x as many users too.

Their SSL site still continues to simply not work:

  www.facebook.com uses an invalid security certificate.

  The certificate is only valid for the following names:
  a248.e.akamai.net , *.akamaihd.net

This is the biggest problem with the SSL PKI. It is partially responsible for the lackluster deployment of SSL. We need to switch to a TOFU model supplemented by a more nimble version of the current PKI for first connect acceptance.

You would not believe how often I run into problems just trying to use SSL on major sites. If you can't trust a large organizations like facebook and akamai to mess up SSL support during problems, you simply will never get it deployed well on all the rest of the sites.

SSL is simply a poor user experience. Twitter's SSL site often goes down more often than the main site too. SSL users get a second class experience and that is the most damming thing I can think of for a technology where deployment is so critical.

Excellent points, but please be sure to simply never split infinitives. ;)

I shall continue to boldly split all the infinitives I please. We speak English, despite the best efforts of medieval Latin grammarians.

Everyone should calm down and realize that Facebook being down for a few hours is not going to kill anyone's business. It may cause trouble and disruption, but not anything more intense than a daily commute in Los Angeles.

That being said, this makes me question Facebook's ability to deliver on the "Social Graph" promise. If they truly want to build a completely social internet, this kind of downtime can't happen.

> "If they truly want to build a completely social internet, this kind of downtime can't happen."

Why not?

Even commerce sites such as Amazon or Ebay have downtime. Studies demonstrate that unless you're selling a perfectly commoditized generic product, obtainable instantly from multiple sources, downtime doesn't materially impact revenue.

Users seem to manage to "queue up" their intended actions and do them when a site is back. And it's not like Facebook's users can gravitate en masse to anywhere else.

The founding principle of the internet is not "5 nines" uptime for all components, but resiliency. Switch off your mail server for a few hours; you'll still get all your mail when you turn it back on.

"Kids these days" build apps and APIs (and sites that rely on APIs) that are much more brittle, because today's developers are spoiled into thinking communication conduits are reliable. They're not.

The fabric fails, so your site, your apps, your protocols, should fail gracefully -- and recover gracefully later.

I like how their API response time chart intentionally doesn't have a Y scale. But clearly it is busted:

http://www.facebook.com/developers/chart.php?type=at_total_t...

Oh dear, DNS error on that link.

As somebody pointed out, the www record was removed probably to stop the barrage of requests coming in.

As an aside I notice that 'Facebook' is not a trending topic on Twitter. Seems the term must be blacklisted in some way along with all the swears and other undesirable terms.

Which seems a bit underhand by Twitter.

Edit: I notice that 'Facebook isnt working' has appeared now. I've blogged about this issue here: http://blog.dansingerman.com/post/1174729229/twitter-mostly-... (if anyone cares)

during one of the twitter hacks the blacklist application was accessed and facebook was on the list (as was google and a bunch of other brand names)

I'm curious about this and Googling failed me, do you have a link?

I took a screenshot on my other machine - I will upload it when I am on there next

There were hundreds of items on the list

(Edit: here is this link for now: http://techcrunch.com/2009/07/15/another-security-tip-for-tw...

I am sure that I saved the entire page of blacklisted words)

That screenshot shows which of the current (more or less I guess) topics need to be blacklisted. Not which words are always blacklisted.

Wow. So twitter censors disparaging comments about companies? Wow.

More likely that terms that are continuously mentioned in huge numbers (and therefore don't count as "trending") are omitted.

There are more principled ways to do that than a blacklist, though, like looking at current frequency compared to typical frequency. That way even something commonly mentioned could still genuinely be trending if it gets mentioned much more than usual on a particular day.

(It's possible there are scale-related reasons that make this infeasible, though.)

Just not tracking the frequency of the most common words that will never be in the trending topics is probably a big performance win. I'm sure there is an additional blacklist that includes stuff like "and" and "the".

You know you've made it when your brand name is a stopword.

No, it just prevents comment from trending. Not just disparaging comments, all comments, it seems.

Well looks like someone is having fun...

http://twitpic.com/1yqrwc

I see "Facebook down again" trending and earlier saw "Facebook down".

Trending topics: Mono Jojoy , Facebook Isnt Working , Bishop Eddie , Internal Server ,

Before, two of the trending topics were "Facebooks" and "isnt working".

In researching web performance DNS configurations issues are common problem. It is usually related to www.foo.com and foo.com being configured differently. This seems to be at a high level the issue with facebook.com. if you go to "facebook.com" the DNS will resolve correctly, but it will redirect you to www.facebook.com. I guess they could stop the redirection and fix the problem immediately, or wait till the sorry.ak.facebook.com.edgesuite.net is fixed.

=====www.facebook.com

new-host-3:dpu2 robertbuffone$ dig -trace www.facebook.com ;; Warning, ignoring invalid type race

; <<>> DiG 9.6.0-APPLE-P2 <<>> -trace www.facebook.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19626 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION: ;www.facebook.com. IN A

;; ANSWER SECTION: www.facebook.com. 0 IN CNAME sorry.ak.facebook.com.edgesuite.net. sorry.ak.facebook.com.edgesuite.net. 0 IN CNAME a1030.g.akamai.net. a1030.g.akamai.net. 4 IN A 63.84.59.59 a1030.g.akamai.net. 4 IN A 63.84.59.10

;; Query time: 10 msec ;; SERVER: 198.6.1.142#53(198.6.1.142) ;; WHEN: Thu Sep 23 16:57:29 2010 ;; MSG SIZE rcvd: 144

========facebook.com

new-host-3:dpu2 robertbuffone$ dig -trace facebook.com ;; Warning, ignoring invalid type race

; <<>> DiG 9.6.0-APPLE-P2 <<>> -trace facebook.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19984 ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION: ;facebook.com. IN A

;; ANSWER SECTION: facebook.com. 352 IN A 69.63.181.12 facebook.com. 352 IN A 69.63.189.16 facebook.com. 352 IN A 69.63.189.11

;; Query time: 13 msec ;; SERVER: 198.6.1.142#53(198.6.1.142) ;; WHEN: Thu Sep 23 16:57:44 2010 ;; MSG SIZE rcvd: 78

When I heard FB was down I went over to twitter to see how many tweets this was generating, I was surprised to see the first refresh of "XXXX more tweets since you started searching."

I decided to start keeping track and I made a graph of the difference between refreshes. You can see my graph here. It seems like in the course of just a few minutes (I was probably conducting my little experiment for 10 min?) you can see the frenzy gaining momentum then it seems to die down until only hundreds of people are talking about it per 15 second refresh.

A little bit about the graph, each tick mark on the bottom represents a refresh (15 seconds or so I think). The Y axis is the difference between refreshes.

Graph: http://drp.ly/e3oH

It's a funny coincidence that this comes right after the recent facebook valuation news.

http://en.wikipedia.org/wiki/Evil_eye

Current Status: API Latency Issues We are currently experiencing latency issues with the API, and we are actively investigating. We will provide an update when either the issue is resolved or we have an ETA for resolution.

DNS issues are the cause for so many outages by a myriad of prestigious web shops. It's incredible how often overlooked something so pivotal to your infrastructure goes without being properly monitored. I have seen entire businesses suffer hours of downtime because their monitoring systems would query an authoritative name server from RFC1918 IP space - but not the same from an externally visible address. An NS not responding to local queries can be just as detrimental internally as they are externally because of how web services are architected, obviously. In their defense, Facebook rarely incurs outages of this capacity, and I can bet it won't happen again and someone is getting seriously reamed. They can and will happen to the best of them, no matter how many MIT PhDs you have designing your systems.

maybe its the 1 billion like buttons they serve a day that tipped them over the edge. ;)

perhaps the Stuxnet virus was built to target facebook? ;)

You don't get to 500 million friends without making a few enemies

Wait, doesn't Facebook already have 500 million enemies?

Don't place 3rd party javascript on your pages if you don't really need it.

Facebook is our biggest traffic source after Google. I'm inclined to forgive them for occasional outages.

I did say:

> if you don't really need it.

In your case, you obviously do.

I don't think we're uncommon. Most sites that I see plastered with Facebook embeds are publications, and I'm sure they all see results similar to ours. Facebook is RSS for the masses.

PPC? CPM?

Not ads. We post links to people who like us on Facebook, and we also get traffic from people who share our articles with their friends.

The significance of the "like" button missing from other sites is lost on me.

I've been using the Application Boundaries Enforcer feature of no-script in Firefox to block facebook content when I'm not visiting a facebook site directly.

  Site .facebook.com .fbcdn.net
  Accept from .facebook.com .fbcdn.net
  Deny

This is motivated by privacy concerns of Facebook's ability to track which partner sites I visit.

First I just had an empty feed, then I got the feed back but posting links and comments timed out. Then it worked for a minute, and now I get the DNS error. Seems like this affects several of Facebook's applications.

I experienced a facebook outage yesterday afternoon (-03:00 UTC) although the DNS resolved the facebook I could not ping their IP addresses. As you can read in facebook's twitter timeline.

At least Hacker News still works. All is well.

Is this a coordinated blackhat attack with DHH's PR attack intended to benefit some investor somewhere? (joking)

It's odd, I've been hearing about this all day but I have had no problems at all contacting facebook.

IT Dept. around the world will now get washed with calls, on how the internets dont work.

the BOFH will be making a list of people wasting time on facebook for later blackmail.

joking aside, this actually had this effect, as so many pages actually resolve some components to facebook.com, the slower response was felt on a wide variety of sites.

Aaaaaand, they're back. Down for about 15 mins (from Tucson, AZ).

heh... ironically facebook's error rate is at an all time low!!

http://developers.facebook.com/live_status

The one day I choose to update my FBConnect code!

Who's next to go down? downrightnow.com?

downforeveryoneorjustme was throwing errors when I tried it: http://downforeveryoneorjustme.com/facebook.com

But I can still harvest my cows ...

Thank goodness for Ping ...

Nice to see the big boys experiencing mundane issues as well now and then...

I demand a refund.

I [Like] this.

In A.D. 2010

War was Beginning.

Internet: What happen?

MarkZ: Somebody set up us the bomb.

FB: We get signal

MarkZ: What?!

FB: Main screen turn on.

MarkZ: It's you!!

DDH: How are you gentlemen !!

DDH: All your Information are belong to us.

DDH: You are on the way to destruction.

Rarely use it so I didn't notice...

I'm sure somebody out there told you so

I thought I had a crappy router, I didn't realise it was that crappy!

Reply on news.ycombinator.com