Netflix was experiencing issues | Hacker News Reader

Netflix was experiencing issues

help.netflix.com

37 pointsjameshk27 comments

Really looking forward to read about what happened. I know they have great resiliency measures in place, I'd like to know if they worked, in what measure, how they failed or how they weren't enough. Or if there was anything they could have done.

Great resilience measures? Like depending on AWS that's seemingly going down like one full day per year?

I guess what I'm trying to say is that one the one hand i see netflix having lots out outages, on the other hand being celebrated as some sort of thought leader on resilicency/high availability. Good tech/open source marketing goes a long way in certain crowds, it seems.

It's not like doing lots of stateless transactions/distributing read-only content is a really hard problem by now. Akamai has done it well for ages. Yet Netflix while being celebrated primarily for their tech marketing in this area is struggling.

Yes, those types of measures.

https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey

Great resilience measure, in spite of their choice to use AWS.

Any infrastructure will involve downtime, be it your own hardware or some cloud provider. Probabilities will vary, but they're always there. Despite their hosting solution, Netflix is doing (or showing) extra effort in dealing with the reality of failures. I'm interested in that bit.

Sure it might be a good idea to diversify their cloud providers, although I'm not sure this might be involved here, since AWS seems to be doing fine at this time.

According to this audit EC2 had just 2.14 total hours downtime last year, the least of any major provider by region.

https://cloudharmony.com/status-1year-group-by-regions

This is HN.. You shouldn't use dodgy language like "seemingly" to avoid doing homework and citing sources.

Then there's S3, and xyz, etc.

S3 had 2.81 hours downtime, according to the report I just linked to. Homework...

xyz? S3 is on the same link. Even if they are additive -- assume the S3 downtime doesn't overlap with EC2 -- they are still at less than a day, half day, quarter day...

What does S3 mean? Seems you guys are working for troubleshooting.

S3 is an AWS product: http://aws.amazon.com/s3/

johansch is trying to paint AWS -- the culmination of its products (EC2, S3, "xyz", etc) -- as something that is down for day(s) a year. We don't (have to) work for these providers to know the real truth and shut down such FUD.

During the evening hours, Netflix represents over 35% of all internet traffic in the US. That is an insane demand to be able to service, but they've done extremely well in my opinion. Much of their tech is open-sourced leading to much better practices industry-wide and they're transparent when things go wrong.

Here's an example of their preemptive work to deal with the loss of AWS/ELB regions:

http://techblog.netflix.com/2013/12/active-active-for-multi-...

An an example of their post-mortem after deploying Node.js with a bad handler:

http://techblog.netflix.com/2014/11/nodejs-in-flames.html

Is there any other service with a library well over a petabyte in size that delivers multiple terabits/second worldwide to millions of customers with better reliability? Maybe YouTube with the backing of a company worth $300B more than Netflix?

Again you are confusing openness with availability. I do agree they are open. I don't really see the connection between their openness and their performance though.

I think my last paragraph was pertinent to your question, is there anyone providing the volume of traffic to millions of worldwide customers comparable to Netflix who is more reliable?

Podcast where Adrian Cockcroft goes into the philosophy and technology behind scaling at Netflix: http://www.se-radio.net/2014/12/episode-216-adrian-cockcroft...

Worked fine from my iPhone over LTE but not Wii over wifi.

And apparently it started right before my son's designated Sesame Street time. Boo.

> And apparently it started right before my son's designated Sesame Street time.

Something to be said for local copies of programs that really matter.

Something something offline viewing?

AWS status is all green so this is a first. Monkey gone wrong ?

Rise of the simian army? I suspected netflix was hiding the "good" AI somewhere (because they haven't been using it on my recommendations).

Failed to load resource: the server responded with a status of 503 (Service Unavailable: Back-end server is at capacity) https://support.netflix.com/en/rest/views/popularbytopcat.js...

From their Twitter: https://twitter.com/Netflixhelps/status/562760983790485504

We're aware some members are experiencing technical difficulties on the web site in all regions. We hope to resolve this asap.

I'm watching House of Cards right now, but it took a few tries for it to start and not throw an error.

Planet of the Apes? Eager for the postmortem.

i have been getting the issues warning and a limited movie selection on and off for days now.

Failed to load resource: the server responded with a status of 503 (Service Unavailable: Back-end server is at capacity)

Chaos monkey strikes?

Reply on news.ycombinator.com