Hacker News Reader

Comment by dnate | Hacker News Reader

I hope they release a post mortem about this. Would be very interesting.

I was just about to reply with the exact same message.

Netflix has spent A LOT of time and money on their infrastructure, expounding their engineering views, proclaiming best practices, and how they use redundancy/sharding/insert best practices and major buzzwords in order to combat the chance of this happening.

I'm a huge fan of Netflix and their approaches to infrastructure, and I'm almost certain that they'll have some very interesting conclusions from this.

Can't test for everything, but seeing this is the most significant outage in ages, I'd say they're doing a damned good job.

I mean, after all, the internet's true natural predator still lurks, its yellow CAT steel present across all of the nation...

I speak of course, of Backhoes.

(1): https://www.wired.com/2006/01/the-backhoe-a-real-cyberthreat... (2): https://it.slashdot.org/story/06/01/19/1643215/the-backhoe-t... (3): https://www.urbandictionary.com/define.php?term=Fiber%20Seek...

I heard a story from the construction company (not int US) where my brother is working: on a construction site, they had to dig in an area with fiber. So they called the telco, aking them to disconnect the cables so they could start digging. But the telco was late and a few days later, fed up by the delay (and associated costs), the site manager said "start digging, we'll deal with the fallout". So the backhoe start digging and cut the cables. The telco were quick to send a guy after that...

I knew a guy who was the head of networking for a university. His biggest rant was that the fine structure for cutting a line made it cheaper to cross their fingers than to wait for someone to mark the exact location of the cables so they could dig.

Or also known has backhoe fade

Nothing is bullet proof. Even resilient systems like Amazon’s S3 are subject to human error, it’s not like Netflix being down is the end of the world.

true. netflix is probably the world’s least critical system at global scale.

consider the increase in child birth rate by this outage. lives will be forever changed :-)

Ad networks would be the least critical. Netflix at least provides content some people actually want. I have yet to hear someone say "I want to see ads".

Certainly not, considering how heavily businesses rely on ad networks as a source of revenue (both by selling ad space but even more so by buying ad space and advertising their products)

Do their employees and CEOs actually say "I want to see ads"? Nope.

I fail to see how any employees/CEOs personal ad viewing preference would be relevant here.

Employees, CEOs and businesses in general like generating cash and/or getting paid. If an ad network being down hinders their ability to do so, then that seems pretty critical to me.

Do they want to stay in business and keep their jobs? Yes.

Many people won't be happy if their salary would be delayed because ad network was down and website didn't generate enough income to pay expenses.

They are welcome to jump ship to companies that sell an actual product and don't profit off wasting everyone else's time with their ads.

Sorry but I really have no sympathy for people that happily waste my time by showing me irrelevant content, trying to sell me garbage or flood my mailbox at every opportunity (the only reason my mailbox is clean is because I am allergic to any web form that asks for my email).

Just curious, do you use any google products?

I only use Search, and even then I'm trying to cut down as much as possible by using DuckDuckGo.

I don't have anything against Google per-se, by the way. I'd be happy to pay for their services should they allow me to do so.

But you at least recognize that ad supported businesses have solved some fairly interesting technical challenges like: - email spam - search - fast dns - secure browsing - giant linux forks

And that we would be unlikely to have such things in a pay for everything model?

> Ad networks would be the least critical. Netflix at least provides content some people actually want. I have yet to hear someone say "I want to see ads".

That's actually a commonly-heard phrase in the US around Superbowl time.

Well, when the alternative is watching American football...

> watching American football

Are you referring to the 17 minutes per hour of actual gameplay, the 21 minutes of rambling commentary/sports celebrity gossip, or the 22 minutes of ad spots?

Upon recent years of dissapointment, I no longer watch superbowl ads.

I think there was a massive marketing campaign to get people to pay attention to superbowl ads claiming they are funny.

They've just turned into ads in recent years.

And if the curiosity gets the better of you, numerous people upload them all to YouTube, often bundled together in one long clip (unfortunately too often with meager attempts at witty commentary).

I thought the amazon alexa one this year was really funny: https://www.youtube.com/watch?v=J6-8DQALGt4

True but you know people will act like it is.

The worst outage my company ever had was due to bad cost cutting decisions.

My company ended up having an equipment failure that took out both production servers and out internal support system at the same time. Lots of pale faces that day. What has two thumbs and bureaucratic bullshit that put runbooks and deployment tools on production hardware? This guy.

My bet is whatever cause this problem, someone got a raise last year for doing it.

You make it sound as if good engineering practices somehow should guarantee no outages ever. Best practices are risk minimization, not risk elimination.

Reply on news.ycombinator.com