Netlify CDN Outage | Hacker News Reader

Netlify CDN Outage

netlifystatus.com

70 pointsajsharp24 comments

Netlify is describing the outage as being the result of some kind of attack:

"We’re currently suffering an attack that’s affecting our Free and Self-Serve CDN. Enterprise is currently unaffected. We are working on mitigating the effects and will update with more information as it becomes available."

I understand the growing pains Netlify must be going through. They have created a great product, and are becoming very popular. I'd love to see the Enterprise version (or any paid version) segregated out from the free sites so we can get more reliable uptime.

The build pipeline was down for several hours a few days ago, now a CDN outage makes things much worse, as it's taken down all our sites.

I'm sure they will resolve things quickly, in the meantime our clients will start asking about other failover options.

According to their overall status page, they already do run two separate CDNs for normal and enterprise customers: https://www.netlifystatus.com/

Ostensibly they do seem to have that.

> Our non-enterprise CDN is failing a small number of page loads, usually with SSL errors; Enterprise CDN is not affected.

But looks like there's room for improvement, because the enterprise CDN ended up failing as well.

> We are still working to mitigate the effects of these problems and do see some page load failures on the enterprise CDN as well.

It'd be really cool to see a writeup of what is going on at Netlify.

They are tweeting about it, and I'd expect a post mortem as they have been transparent in the past.

So I wasn't going crazy haha - just today we migrated our site to Netlify from Heroku and the second I wanted to demo it my boss the COO, it was failing and not loading.

It wasn't just netlify though. I also was having trouble loading random websites like aptible, youtube, imgur.

This affects us too. We were able to migrate to zeit now in ~30 mins.

Will you migrate back to Netlify when Zeit will suffer a similar downtime?

We're definitely staying on Netlify after this outage is done. We pay for some of their enterprise features, e.g. audit logs. We just need our sites up right now and zeit was a fast, temporary alternative

Do your DNS records have a short TTL to enable such a quick switch from one provider to another?

Fortunately (or non-optimally), our DNS TTLs were 300s (Route 53 default) so our sites were not fully accessible off Zeit for ~5 mins. This wasn't too bad given that our sites were completely inaccessible from Netlify.

One day when we're bigger, we'll extend those TTLs

Short TTLs are bad, but I confess using them in some small scale projects in anticipation of this kind of worst case scenario :)

My site is back up. (1:13 EST)

EDIT: intermittent issues now. Sometimes nothing loads, sometimes everything, sometimes SSL issue, sometimes HTML but no CSS or JS load. (1:22 EST)

Their CDN hostname, `cdn.netlify.com`, resolves to a DigitalOcean node, so it seems like they are rolling their own instead of delegating to a third party.

Yup, Stream's blog is down as well. Thought it was WPEngine at first, but turns out it's netlify.

Huh, loaded HN because it seems YouTube videos are down for me. I wonder if it's related.

Another comment claimed that Netlify rents servers from other providers, so this could be the case.

FWIW Youtube is fine for me right now.

insane coincidence but literally the minute i started my first project ever on netlify just to play with it, i deployed it and nothing was loading; i kept googling and thinking it was my fault and then it clicked to check their status page and alas.

So it was YOU who took down the network then

netlify.com appears to be affected as well.

Curious if this is a software issue or a host issue. I believe netlify rents a lot of its infrastructure from other providers, such as Digital Ocean.

At any sort of scale (which I have to imagine netlify is running at), this will be a software issue since hosts are designed to fail.

Serving static content is crazy easy for computers, my guess is a bad configuration got pushed out.

Here I saw some mentions that it may be related to recent google cloud problems https://news.ycombinator.com/item?id=21418979

Bad publicity is still publicity, right?

Reply on news.ycombinator.com