Cloudflare DNS problem | Hacker News Reader

Cloudflare DNS problem

support.cloudflare.com

60 pointssinak35 comments

Dyn also appears to be having a real, real bad day. Glue records vanished like a fart in the wind.

  $ dig @8.8.8.8 ns2.p29.dynect.net

  ; <<>> DiG 9.7.3-P3-RedHat-9.7.3-8.P3.el6 <<>> @8.8.8.8 ns2.p29.dynect.net
  ; (1 server found)
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 2690
  ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

  ;; QUESTION SECTION:
  ;ns2.p29.dynect.net.		IN	A

  ;; AUTHORITY SECTION:
  dynect.net.		600	IN	SOA	ns0.dynamicnetworkservices.net. hostmaster.dyndns.com. 2015070600 10800 1800 604800 1800

  ;; Query time: 2 msec
  ;; SERVER: 8.8.8.8#53(8.8.8.8)
  ;; WHEN: Mon Jul  6 21:42:48 2015
  ;; MSG SIZE  rcvd: 120

Does CloudFlare use Dyn or are these unrelated issues?

The affected zones are those cnamed from CloudFlare to Dyn.

Any example hostname thats configured on Dyn and is not resolving?

Sorry about this: just posted a blog post. Appears to be a Dyn issue, but affects a lot of our customers (including support.cloudflare.com which is Zendesk)

https://blog.cloudflare.com/the-internet-is-a-cooperative-sy...

This appears to not be a problem with CloudFlare, but a problem with Dyn: https://www.dynstatus.com/. Of course, Dyn and CloudFlare have mutual customers and so if a Dyn has a problem it's possible for a mutual customer to have a problem.

Looks like your support site might need a little.. support http://i.imgur.com/X76QdaC.png

You might want to try putting it behind a CDN to cope with the load

We use Zendesk. Zendesk uses Dyn. Boom!

Strange, we are using CNAME apex resolution with cloudflare to point to Heroku apps, so I'm not aware that Dyn is used anywhere in that chain, but resolution is still down for those apps.

edit: resolved now.

    $ dig +short NS heroku.com    
    ns1.p19.dynect.net.
    ns4.p19.dynect.net.
    ns3.p19.dynect.net.
    ns2.p19.dynect.net.

Thanks!

More details here on Cloudflare's status page here: http://www.cloudflarestatus.com/incidents/m8hf8q8vs81d

Sorry -- our ops and engineering teams are looking into this and will update/resolve as quickly as possible.

https://www.cloudflarestatus.com/incidents/m8hf8q8vs81d

Good luck, this is really rough. I encountered a 101 on a random site before I saw this and was a bit baffled.

I love how even https://support.cloudflare.com/ is down by this.

Makes me feel good as an impacted user. I know that the problem won't be ignored!

They do at least use a different DNS provider for their status page: http://www.cloudflarestatus.com

I do love how everything is operational...

I see the big blurb about DNS at the top. But it still feels like the table is misleading. When they're having a major DNS outage and every site is "operational" it feels misleading.

Although if this was Amazon's AWS console we wouldn't even have the blurb at the top...

It's frustrating for me, too. Because my side project https://statusgator.io pulls in this status data and sends alerts to subscribers when services go down.

But since CloudFlare has not indicated any downtime, everything still appears up.

ZenDesk uses Dyn. Hence, it was down. Sorry about that.

If you're on Heroku, one immediate way around this problem is to guide your users towards your heroku domain. I've been doing this for clients contacting me who absolutely need access right this moment.

E.g. https://myappname.herokuapp.com

That's a nice tip.

Can I ask a question while we wait for Cloudflare to fix this... Heroku looks nice. But also quite expensive after you add on dyno, Postgre/Redis, and "addons." What about Heroku do you like/appreciate that justifies the slightly higher costs? Or do you just like that all of the devops-style stuff is handled by them (as opposed to AWS/Azure/Linode, etc where you'd have to do it yourself).

I am genuinely asking, I have no reason to dislike Heroku, and I know it is quite popular. Just trying to understand what the "value add" is over other cloud services (or why they're dissimilar).

Please don't feel obliged to answer, but if you have the time it would be appreciated/interesting.

No problem, here's my two (relatively unorganized) two cents:

With Heroku (frankly also thanks to Ruby on Rails) I can be a sys-admin, developer, database administrator and QA all at once; far easier than before. For the price of $100k I have seen developers build successful products that used to cost $1M+ to develop.

On Heroku you definitely pay for convenience and ease of use, knowing that you get a benefit in doing so by being able to focus on the development of your app, the part that matters. The costs grow if you get traffic, but through careful management (such as packing multiple processes into one dyno), you can reduce costs. Even if many find Heroku's prices high, they are still much lower than what you see in enterprise land, or even on AWS if you aren't careful.

Heroku's documentation is also incredibly good, and this helps you a lot to move fast. Almost by accident, simply by following Heroku documentation I have found that developers end up following good cloud server practices almost by default (i.e. because Heroku forces you to build your app to follow certain conventions, conventions needed for making your app horizontally scaleable; such as delegating clock processes to a separate process, and not allowing dynos to manipulate or rely on unique file memory specific only to them).

If I architectured or configured my own servers it would have taken 10x as long, exposed me to far more security issues, and I would have probably screwed it up any way. The lesson from this is that I think as developers we need far less freedom to configure than we think we need.

So in conclusion, if you're a small to medium-sized app that does not have complex server or bandwidth needs (i.e. you are not Imgur or Pied Piper), then I highly recommend it. I'm sure many Oracle salesman will be unhappy with this opinion, but take my word for it that the vast majority of applications do not need complex, overengineered server systems to be successful. (After all as a startup, you should not be wasting months planning on how you will scale to support 10 million concurrent users. I've seen that hubris distract failing startups before.)

> With Heroku (frankly also thanks to Ruby on Rails) I can be a sys-admin, developer, database administrator and QA all at once; far easier than before. For the price of $100k I have seen developers build successful products that used to cost $1M+ to develop.

Unfortunately, Heroku is down constantly. Don't take my word for it; google "Heroku down" and see how often it occurs.

Microsites? Sure. Hobby sites? Sure. Business applications that generate revenue? Good luck.

Heroku is just a layer on top of AWS. I know a ton of successful business applications who rely on Heroku + AWS. Obviously after a certain size you will want to customize your hardware more, but in the early stages Heroku can take you a very far way.

Ello is a recent app that had to scale to millions of users extremely quickly and did on Heroku. Read their technical write up: http://mikepackdev.com/blog_posts/40-5-early-lessons-from-ra...

Ello had issues, and Heroku would be considered amateur for what they were doing, but if you read it Heroku was not the cause of their performance issues -- their application code was. IMO most business applications run into problems due to performance issues within their application code, not because of server downtime.

Regarding how common downtime on Heroku is... yes, Heroku has lots of minor incidents impairing its toolkit and some services, but these are not issues that go notices. Two years in I still haven't had significant downtime from Heroku itself (more than 5 minutes) during business hours where it matters. Cloudflare, and other DNS providers, have frankly given me more trouble.

Thank you for the reply. I appreciate the insight.

Lately my number one bloody cause of downtime has been DNS lookup failures. I switched to Cloudflare hoping to avoid this. If I can't trust Cloudflare, then who the heck can I trust?

Also, is there any way to actually resolve this issue quickly while Cloudflare deals with their fuckup? I don't like being at the mercy of their support team while my site has literal downtime.

We should have it fixed very shortly; 99.9% on the cause.

I use DNSimple and they have a feature that lets you sync to another provider.

I have not used it but, as I understand it, the feature helps you setup slave servers which respond to Notify messages from DNSimple. That way, if they go down, your slave provider will keep things up.

You could always run your own DNS server. Of course, you'd then lose the other services Cloudflare offer like basic DDoS protection, but that's the trade off you'd have to consider.

I've never had any trouble with Amazon Route53.

That's because Route53 is basically bullet proof, such that their SLA guarantees 100% availability. I have switched everything I touch to Route53 and have had zero issues.

In this case you would have been unaffected if you used CloudFlare for DNS, rather than using Dyn for DNS and CNAME from CloudFlare to it. Using CloudFlare for DNS is our preferred configuration.

> Also, is there any way to actually resolve this issue quickly while Cloudflare deals with their fuckup?

Learn basic troubleshooting so you can determine which vendor is the problem (ie not Cloudflare in this case).

We had the same question and concluded that the only reasonable way to say more than "Well, we hope this doesn't happen again" was using 2 different authoritative DNS providers (both anycasted). Details: http://blog.papertrailapp.com/dns-outage-on-monday-december-...

It's not a question of trust; at some point, any single provider will get DDoSed or have an operational bug.

The only question is whether you're willing to accept that. If you are, then try to pick one great provider.

If you're not, and are willing to spend a little time to avoid this problem, try to pick two great providers. Since then, DNSimple has made primary/secondary providers even easier: https://support.dnsimple.com/articles/secondary-dns/

Basically, if you don't want this problem, don't pick one provider. It's not bulletproof, but it's a lot closer.

Reply on news.ycombinator.com