AWS outage brings Netflix down for some devices on Christmas Eve

I know they have some degradation of service in their recommendations but I can't speak for the rest of the service. A coworker was telling me that each recommendation on your main netflix screen is provided by a different service. So lets say its suppose to be feeding me scifi recommendations. If the scifi service is down, then it will just remove it from the main screen and I will get a different recommendation category until its back up.

Yeah I can't wait till people wise up and don't just see "the cloud" as one AZ in us-east-1.

(It wouldn't have helped in this case, but its a general annoyance I have.)

Why should they care, Netflix is the biggest competitor to Amazon Instant Video.

They do care, but they're also developing software pretty quickly and they break stuff just like everyone else. Only, when they break, people tend to notice, and those people tend to spread word around pretty quickly.

When I worked at a hosting company, we'd have at least 5 shared hosting servers (with ~75-200 websites on each) go down each day, and yet we had a reputation for good uptime, because comparatively, we DID have good uptime.

I think the problem is that most people think about uptime wrong. Uptime is a compromise; a trade-off. You can pay for more uptime with diminishing returns after about 98%. Running one dedicated server colo'd with decent fault recovery systems in a decent datacenter will probably get you 100% uptime most of the time, until it doesn't, thus ~98%.

If you're going to attempt to get beyond power failures, you'll need a second server (or "instance") somewhere. If you need the capacity anyway, this might not double your costs, but you may still have unused headroom. You could buy flexible computing power by using shared hosting ("the cloud") or whatever, but it's the same problem.

Once you get into a state where you have a global business, customer demands, supplier issues, vendor lock-in, etc, it becomes a numbers game. You can hire (more|better|famous) devs and possibly get more uptime. You can test more (and slow down feature releases) to get better uptime. You can break stuff (and pay for the recovery + lost face + downtime) to decrease downtime. Everything is a trade-off, and right now it makes sense to chase about "four nines" of uptime -- 99.99%.

Four nines is 4.32 minutes per month -- four minutes and nineteen seconds AT MOST once a month. This is very possible for many large services and while it does have cost overhead, it's manageable.

TL;DR don't go chasing waterfalls (100% uptime.) It's not practical. It may be possible, but it depends on how long. 100% uptime for 10 years would be a pretty damn lofty goal. In my opinion, it's much more important to recover quickly and gracefully with awesome communication with your customers than it is to be up 100% of the time. 100% uptime goes unnoticed, but consistently great customer service does not.

This works well until it isn't cost-effective to do at Netflix scale. Doubling your spending for once-a-year events doesn't sit well with investors.

This is only reasonable to a point. You're essentially doubling your costs because things don't exactly work the 'same' between zones. It's not like you're putting a bunch of servers in the same platform exactly and spreading them out around the planet.... so you can't have four web-servers instead of four and send traffic two all four (with two sets at another AZ) without considering and also having a separate DB setup in that AZ, etc.... so costs quickly mount (not to mention it's a much more complex platform suddenly).

> I wonder what percentage of the population will have never owned a DVD player in the next generation.

Probably the same amount of people that have never owned a record player or, for the younger people, a CD player.

> If netflix owned their own hardware and could reach out and touch it, would this have happened?

Yes, outages happen whether you have the hardware in your own hands or have it hosted by a cloud provider.

I recently attended a talk by netflix on building fault tolerant systems. From what I could tell, there wasn't an architecture, everything could talk to everything else, with a thin Api layer on top. The way they achieve fault tolerance is by not having an ops team, instead they have a resiliency team and monitor everything. This team uses various monkeys to try to break the application in different ways.

They then put all developers on call, and force them to write code that can recover from faults by trying different methods to break it.

tl;dr netflix systems are chaotic, because chaotic systems tend to tolerate failures.

Not all devices support all streaming protocols and formats. I.e. their desktop player uses the proprietary SMPTE 421M codec (microsoft) which is not available (or efficiently implemented) on platforms where silverlight is not available (read: most of them).

Since many of these other platforms also use proprietary formats they end up having to maintain a range of different proprietary streaming servers, many of which presumably need to interface with their respective mothership platform services (XBox live, PSN, Roku).

Oh, and recently they have begun building their own CDN[1] which may add further to the diversity.

[1] https://signup.netflix.com/openconnect

Not especially. What you do get is:

- SSL for "free"

- If you use Route 53, it's a single DNS lookup (no CNAME hop)

- If you're using a VPC you can use an ELB to face the public Internet.

- (Slightly simplier) auto-scaling.

- Redundancy across AZ's

Many/All of these things you could achieve yourself, but if you were using EC2 resources you'd probably find it reasonably expensive.

Roku too