Ask HN: YouTube down? | Hacker News Reader

Ask HN: YouTube down?

575 pointsLiweiZ468 comments

View on HN

Watch "Let's de-Google-ify the Internet" on PeerTube

https://peer.mathdacloud.ovh/videos/watch/c193a2f6-8b24-48c3...

https://github.com/Chocobozzz/PeerTube

https://instances.joinpeertube.org/instances

https://d.tube/

Opened it up, saw a foot fetish video on the front page.

It's like old Youtube then.

Or like random network shares on the University dorm network.

Not until it learns your specific for fetish them shows your kids on NewPipe.

Everything on there looks like a scam or those 5-minute lifehack compilations where you lose a finger if you tried them yourself.

> Clicks on link.

> 0 peers.

> Refreshes YouTube.

> 500.

> Realize the internet is a distraction from what is really fulfilling and meaningful in life.

> YouTube's back up.

> Watch "Micheal Jackson's 'Bad' but Every Time he says 'bad' the Music Speeds Up".

Only getting like 1 peer per video and <150kb download.

That de-Google-ify video I linked has 26 peers currently.

I had the same problem, but it was because one of my privacy extensions had disabled WebRTC.

When YouTube comes back up again, I can recommend Tom Scott with "Single Point of Failure: The (Fictional) Day Google Forgot To Check Passwords"

https://www.youtube.com/watch?v=y4GB_NDU43Q

It's back up.

it's black!

Yes! This video was what caused me to reconsider my use of Google services seriously.

I was wondering how to react if this happened to me while watching the video. If, like me, you use Google with your email on your own domain (eg. gsuite), you could, in an emergency, change your MX records to something non-google.

Worth noting that "In an emergency" also relies on your MX records' TTL being set to a low value

And the DNS not being managed by Google and the domain not being bought through Google.

Wow, thanks for the recommendation. Harrowing.

damn!!!

4 peers with about 320KB download

Download finished, but I only uploaded 6MB total.

Too bad because I've got a symmetric up and down.

Oh wow-- just went to the "Trending" page and it's got a bunch of thumbnails of stuff like Stallman speech videos next to thumbnails of pornography. The Stallman/Porn ratio is about 2:9. (Well, to be fair there's also Peertube related videos.)

So in keeping with the history of the internet, porn seems to be the main driver here. But then their UI solution to keep newcomers from being offended is to blur out the porn thumbnails. So either:

a) the newcomer-- like me-- isn't after porn, and the majority of the screen real estate is wasted, or

b) the newcomer or return visitor is there for porn, the thumbnails they want to view are blurred out, and Stallman's face is eating up valuable porn thumbnail real estate.

Modest UX suggestion: follow the history of nearly the entire internet and separate out your porn from your everything else.

Medium sized regional AS here: I am looking at a fairly large drop in IX traffic charts for our ports that face the IX, updated every 60s, which directly corresponds in time with the beginning of the Youtube outage. (We are not big enough to have a direct, dedicated peering session with the Google/Youtube AS).

At any given time of day 4pm-11pm a huge percent of our traffic is Youtube (or netflix, or amazon video, or hulu, or similar).

Is any of this info publicly available anywhere?

there are a number of IXes that publish aggregate traffic charts, updated reasonably frequently, a lot fewer that publish traffic charts for individual ports or peers. If you want granular data you have to know somebody who runs the core BGP stuff for a reasonably sized ISP, and you can ask them for redacted copies of charts.

A lot of them have stuff like total traffic over time under the statistics link.

https://bgp.he.net/report/exchanges

https://ams-ix.net/technical/statistics

Doesn't really show much of a change right now, nor does the aggregate chart for the SIX in Seattle.

The largest downstream/eyeball networks that take Youtube traffic do it via PNIs with them so you won't see that in an IX fabric traffic chart. For example the Google/Youtube AS exchanges traffic with Comcast directly by their own dedicated ports, no IX involved.

Indeed it's not showing the Youtube issue for the reason you mentioned. Still relevant to the question above :)

Is this not indicative of the drop of traffic in Australia?

https://metrics.ix.asn.au/d/000000041/ix-peer?orgId=2&var-Cu...

IX charts are public in Chile by law. Everything looks well except the Telefonica (TIWS) international trunk [1]. It started dropping all traffic at the same time.

[1] http://pit.grupogtd.com/default.php?id=1

Que bacan weon! Estoy en viña ahorita. Sabes por que Movistar es muy lento com LTE aca?

Are you on prepaid? Virtually all Chilean prepaid carriers are deathly slow, especially outside of Santiago. I had barely any service in Algarrobo on Claro, but postpaid plans ran just fine, especially Claro, even in crazy places like Farellones :)

For what it's worth, I hated every second of my life in vina and that stupid town to the north of it and also Valparaiso. God, bring me back to La Parva or Embalse El Yeso. Some of the most incredible, untouched mystical land there!

I’m currently roaming on a post paid US plan. When I arrived in Chile I had 3G on Claro. Then it switched me to LTE on Movistar. 3G on Claro was more reliable.

Interesting to know about pre vs post paid. Will keep it in mind for longer trips when I normally buy a local SIM.

We just spent the last week in Torres del Paine backpacking around ... what an amazing place! Going to spend a couple days in viña and then headed to Conce to see some old friends. I’ve not been to La Parva but will add it to the list!

Enjoy, my friend! If you have a car, the mountains to the East of Santiago are absolutely awesome. Everything is safe and cool there, nothing worse than Californian tier driving. There's even free wifi at La Parva, and a ski lift to I think 12,000 feet in elevation if you dont want to walk haha.

Enjoy the city! I just had really bad experiences there. It's probably not the norm but I just hate petty thieves and the grime when I'm paying foreigner prices. I was based in Las Condes for 3 months and really loved it.

I think I still have my little $7 claro sim card lying around here somewhere :)))

siempre pillas un chileno en los comentarios

TIWS is back online. I'd presume they are draining most traffic to prevent overload according to the public SRE practices. I theorize it should take a few minutes to get everyone back in.

Just one I found quickly, the UK results are t interesting but check out LINX NoVA https://portal.linx.net/

I just see a tiny blip (or 2) on Toronto’s IX, but that’s aggregate, not port by port:

https://www.torix.ca/traffic-statistics/

Here's for Portugal, we need around 85gbps at peak.

https://gigapix.pt/pt/informacao-tecnica/estatisticas-de-tra...

What is an AS

https://www.google.com/search?client=ubuntu&channel=fs&q=bgp...

It saddens me that the link you posted had to leak the OS you're using... (the fact that it happens)

Or maybe I'm using an extension like this, and am pretending to be a Ubuntu+Firefox user so that they get their commission money from Google --> Mozilla, indirectly funding them.

https://addons.mozilla.org/en-US/firefox/addon/uaswitcher/

Sounds like cheating.

I think clients are allowed to send whatever user agent they want - it's essentially a preference for how they would like the website rendered for them.

Also I believe it's technically a violation of the HTTP spec to serve different GET requests to different user agents.

Arent those two statements contradicting?

If it would be a violation to change things based on user agent, but the purpose of the user agent to specify how you want things changed, then...

if it was purely about different rendering of the exact same content, then wouldn’t it be the browsers responsibility (and thus UA is unecessary)

I have to imagine browsers are allowed to extend the spec, and UAs are either in theory or in practice a way to communicate those extensions; but because site operators were only specifically designating content for the major browser(s), any compliant but unlisted browser would never get that content.. and suddenly everyone was calling themselves netscape

> I believe it's technically a violation of the HTTP spec to serve different GET requests to different user agents.

Is this actually true?

For what it's worth, Google serves a completely different webpage to IE 5 (try it, they still support IE 5/6/etc) as opposed to a modern browser.

That's just presentation and layout though. Am I really not allowed to send different document content to different UAs?

my nerd-profiling radar had you as a network engineer = it's probably ubuntu ;-)

I only wish I could currently watch a video link to that instead...

I guess it goes to show how much we take connectivity and uptime for granted these days.

From the wikipedia article: https://en.wikipedia.org/wiki/Autonomous_system_(Internet)

    ...an autonomous system *(AS)* is a collection of
    connected Internet Protocol *(IP)* routing prefixes
    under the control of one or more network operators on
    behalf of a single administrative entity or domain that
    presents a common, clearly defined routing policy to the
    Internet.

Autonomous System

IX?

Internet Exchange.

I'm guessing it stands for Internet eXchange?

AS? IX? Not everyone knows your insider lingo.

AS = https://en.wikipedia.org/wiki/Autonomous_system_(Internet)

IX = https://en.wikipedia.org/wiki/Internet_exchange_point

But seriously, reading how the internet works is pretty amazing.

See: How I feel every time someone talks about using some six week old framework to program in some 3 month old extension to some 6 month old language (except, this lingo has been around for decades).

This is hacker news, I think someone can be forgiven for using hacker lingo.

It's ISP / Network Engineer / Bit-shifter Lingo.

AS - Autonomous System (Number, sometimes called ASN). What big networks like ISPs use to identify themselves and route to each other, usually via BGP.

It's a fancy way of saying the parent runs a "mid-sized" network big enough to have an AS, likely a mid-sized regional ISP, or part of a larger ISP.

IX -- Internet Exchange, similar to IBX or DC. Essentially the building where there are routers and switches that connect the networks to each other.

Basically parent is saying that they're an ISP and their systems in the IX that route to Youtube saw a dip in traffic, implying a network event of some sort on the Youtube side.

Do they encrypt the server side error stack trace?

500 Internal Server Error Sorry, something went wrong.

A team of highly trained monkeys has been dispatched to deal with this situation.

If you see them, send them this information as text (screenshots frighten them): APkpgMWbWQ3LvimPoFynDB0W8VeUJ9ECiPcDCm8L0Qiku1I2TbAShWp- taKn-AzOGigwq0sU4oe9mbWb2Bwv4BK37C5xOAL7qm11fHn4L0swqhLk wbcnyKH2HM3AQNf-ucVsolyigJTNKA2SSNUMVZnDPmfsFH7ecKkpQmNi VGWhtXypv0zJyz9d_mpkgMoONtIrPUA4imxK-gNnE-_WQWQZNJm0CTae slJVC-TYgnvOZ9AYp6nodeUNpCoGspWaJVXn_ZSxy-71oGdlkCqWs6AY 2wmIEKe8eeAMqwkTHZNHkbAaH-fxWE_WDPuG-q7AFbOz8jZCFD06MYgf obFUSaH6B7PUdBFwVvjEaTD34J8PVhZTIJziRK-9-wSHOI6Vwf1lTuFe X0m52abRMW1VJaZB3taHK09kFT8Lv546OPhsL0Bn70UIs2durkAAYe4Z ...

Yes, they do. Pretty clever idea, I think.

Yes I agree, encrypting the stack trace enable them to share sensitive server side error codes to the client which in return can be easily reported back without having to explain anything or try to decode cryptic custom error messages.

This is fantastic!

Also probably a unique hash too

I actually thought it was a bit jarring. Mostly because I don't know what personal information it might be collecting and having me send them.

Well, it's information collected at the server side. So if they really wanted they could just store the data and give you a code, but that would require more storage on their end.

(sorry for the sarcasm but...)

Oh no! YouTube needs to store a few kb of text! How will they have enough space?

Well, it obviously only happens when something goes wrong. Seems valuable to keep the complexity and external dependencies of your error handing to a minimum.

Maybe its + a placebo since already stored in storage system + ability to see an outage maybe dropped in the filters cause it was assumed its not user-facing and so assumed lesser severity + assurance incase storage failed due to server-storage network unreachable

I too would like to know more about this such as is it encrypted by any know method (ex: pgp)? It's rather brilliant if it's an encrypted stack trace or similar. I imagine it might even contain info about who the user is as well (if logged in or tracked via cookie) to help them debug it more.

Thats prob base64 and your IP might be in the decoded string.

Messing around with it for a minute it seems it's a modified base64 (+ and / are replaced by - and _) with a space every 56 characters. I couldn't get any permutation where a-z, A-Z, 0-9, and -_/_- were consistent so it's either encrypted or one of 1,268,869,321,858,841,641,034,33,389,335,161,480,802,865,516,174,545,192,198,801,894,375,214,704,230,400,000,000,000,000 permutations of the encoding alphabet (or I messed up the decoding).

The modified base64 is probably just URL encoding https://en.wikipedia.org/wiki/Base64#URL_applications

Why does this look like a problem for https://sentry.io

I'm imagining millions of kids in restaurants crying out because their "iPads aren't working"

My son is having "a moment"

Mine too. He’s so confused and mournful.

Mine is sleeping, I was ready to enjoy the new justforfunc episode... I'm the one that is crying...

....

I fear something terrible has happened.

As if a million minds were screaming in terror from the sudden free thinking time.

I have just come to realise that there is no point in life without purpose.

Edit: It's back!

My wife says: "they probably ran out of space on their server"

Oh she knows

Probably because the logs directory filled up...

Insiders everywhere.

Ahhh, not enough monitoring and/or non-existent "disk space usage" alerting? Been there.

Just send her this (obligatory)

https://www.stilldrinking.org/programming-sucks

I have never seen Youtube not work. Ever. This is a special moment. Glad you are all here to share it with me :)

It actually makes me appreciate how reliable youtube has always been. I never even consider a youtube link might not work perfectly when I click one.

Internet sites go down all the time, though usually not for long or en masse.

Makes a good case for self-hosting videos as a backup

Or a good case for keeping one or two physical paper books on a bookshelf, and maybe a candle and lighter.

I don't think yt being down is a signal for the end of times

Google sites are probably the most reliable, I don't think I've ever seen google / gmail / yt going down.

I think a Google search outage would be a spectacular train wreck.

Imagine if Google's search was compromised in such a way as to be has bad as the Sony PSN network hack of ..2011? Imagine not having Google search for several weeks. Bing/DuckDuckGo/Yahoo would struggle just to keep up and people would immediately realize their dependence on the big G.

> I think a Google search outage would be a spectacular train wreck.

I'd argue that a youtube outage is worse. If google search is out, you can just use duckduckgo.

If you want to watch a video on youtube, you're SOL.

This is speaking from experience with blacklisting both google.com and youtube.com in my hosts file. I was doing fine with blacklisting google, but couldn't handle blacklisting youtube.

Try it yourself if you don't believe me.

Youtube is mostly idle entertainment though so an hour of no youtube is not the end of the world but people rely on search engines to get stuff done.

I have seen gmail go down many times years ago but I haven't used the web interface for years so I don't know how often it happens nowadays...

It depends how you define "going down." Here's an old screenshot of mine of when Google Search failed spectacularly by marking every single link as malicious and therefore not taking the user to that site when you click the link [1] [2]. Perhaps that might qualify as going down, although clearly it didn't go down at the network level.

It remained broken (in that way) for quite some time (maybe longer than an hour.) But, happening in 2009, I can't accurately recall.

Here is another scenario when Google Search failed to successfully serve my query [3].

And an early time when YouTube went down [4].

[1] https://i.imgur.com/X05BCpI.png [2] https://i.imgur.com/CaDh1Y6.png [3] https://i.imgur.com/VZI4Yew.png [4] https://i.imgur.com/bPTp4bg.png

I am proud to say that in 2008 my country knocked this puny website off the interwebs in a bid to censor it within the country. They accidentally hijacked the global IP addresses[0]

I am also proud to say that my country left the domain for their original Internet Exchange Point[1] unregistered. Guess who scooped it up?

PS. My country is Pakistan.

[0]: https://www.cnet.com/news/how-pakistan-knocked-youtube-offli...

[1]: https://pie.net.pk

Great work sir!

> Guess who scooped it up?

This is the best domain squatting I've seen.

I have a new hero.

I was going to link to the video of a Dyn employee (Todd whatshisname) doing a parody of American Pie about this -- "The Day The Youtube Died" -- but the only link I have to the video is on Youtube. :/

Here are the lyrics [0] instead, you'll just have to sing it to yourself.

[0]: https://dyn.com/blog/the-day-the-youtube-died-1/

Well everyone here seems mighty proud of censorship. Its too bad when Google themselves want to do it in China everyone loses their shit!

I guess the sarcasm in my original post wasn't obvious enough. But, yes، that was sarcasm۔

I'm also suing the Pakistani Federal Government to hand over a list of censored internet resources as part of a FOIA request. The hearing is tomorrow [0], wish me luck!

[0]: https://link.medium.com/RkevqMKY4Q

In the early days it went down all the time. Same with all of their competitors. The YouTube engineers had a Python+Mysql+lighttpd stack and even had a fork of lighttp for faster performance. They spent every day eliminating bottlenecks until it was stable and then Google bought them.

The original talk about their stack was on Google Videos back in the day. Now sure if it got moved to the YouBoobs.

I don't remember the exact timeline of YouTube's growth, but I think during the time period when it was the most popular site for pirated TV shows, it was really stable and has been ever since.

I totally forgot google video was a thing.

Heh so did i! That was a weird feeling...

It's still Python+MySQL for what I can tell. But also has parts in Go (see i.e. their work on vitess to scale MySQL access https://vitess.io/ )

They now run their python code using a transpiler into Go with a Go runtime and Go libraries: https://github.com/google/grumpy

Reminds me about Facebook, who transpiled PHP to C++. Before creating their HHVM interpreter and their hack language.

This is not true. Grumpy exists, but it is mostly unmaintained now, and certainly isn't capable of transpiling all of youtube.

I agree, but a special moment because these days there are people like you that never had to see "big" sites crashing or dial-up speeds.

Going on over an hour now. Someone goofed big time.

Never seen Youtube go down on their end. But I have seen it stop working on dozens of workstations at a news site simultaneously when they dropped ie7 support.

Magically this enabled the outsourced infrastructure provider to roll out a new default browser quite a bit faster than they previously claimed they can...

As someone who ran a channel with a few thousand subscribers for a year or two, I’ve seen my fair share of 500 error pages on YouTube. Granted, it was always isolated and may recover after a few refreshes (IIRC), but still.

Interestingly, my youtube-dl archiving script is still chugging along with only a handful of elevated retry rates. So whatever the problem is, it's probably not at the API or CDN layers.

Both m.youtube.com & the mobile app are down.

Along with their live TV product, tv.youtube.com

The static contents are still available so it must be problem in their application. Strange that is down world wide, not a region. But, then Google likes to make things as a whole, ie. Spanner database.

isn't it also strange that the shadow of their ui components are still loading?

Not really, this is done client-side.

They're generic templates to make you think the page is loading faster than it is. They don't represent the content that eventually (normally) shows up.

I was able to finish watching a video on my phone, and it definitely hadn't buffered all of it yet, so some servers must still be able to handle requests for videos.

YT generates resource links with ~couple of hours expiry dates, once generated page/app will keep reusing that link.

SMTube (aka http://tonvid.com/) failed[0] due to "(500) Backend Error"...

[0] http://i.imgur.com/PRNMtwY.png

What are you archiving?

Ha! My time to shine.. I’ve got somewhere around 80TB of YouTube content on my NAS including thousands of videos that aren’t available anymore on YouTube or anywhere else online. The joy of hoarding data..

80TB is a lot just to hoard data... I don't think I even like so many vids.

Well.. I’ve got to have a selection for every occasion!

But it’s not like I’ve watched every video I have, often I just download entire channels if I deem the content to be worthy for archiving.

I bet we'll have tons of makeup channels then, right ?

I’m a bearded, bald guy..

Not really my thing ;)

I hope you've got plans for what happens to that content when you're not around any more. With how ephemeral a lot of content is on YouTube there's sure to be valuable videos on people's drives.

I've had a lot of luck with finding YouTube videos that disappeared via archive.org

Glad you mentioned that!

I just recently checked my collection to see which videos aren’t available anymore online. I should probably cross check the results with Archive to see if they have them available!

I would love to have videos verified as no longer available.

As soon as I figure out how to check them I’ll be sure to send them over to you!

Unfortunately YouTube also deletes the title when a video is being removed. I regularly have a couple of "deleted video" entries in my favorites list and I don't have the slightest idea what these videos were about.

I actually do have plans for that in place!

I’ve got two identical servers, one will go to The Internet Archive and the other one should go to some German archive but I haven’t decided which one, still looking around.

I have a script in place that deletes all private data off of those servers before and instructed two friends and some family members on how to proceed.

Of course I don’t just hoard YouTube videos.. There’s a wide variety of data on there.

About lost content; I just recently started looking up all the YouTube videos I own to see how many aren’t available anymore on YouTube (A LOT). Still trying to decide if I should re-up them with some burner accounts or leave it be..

Talk to Jason Scott at archive.org. Those taken down videos could be uploaded now for safekeeping

I just have to figure out how to check which ones already are on Archive. But I’ll absolutely do that!

https://github.com/bibanon/tubeup

I know a kevingrahl from gradeschool in leipzig. Interesting comment, but thats the only thing I have to say.

Are you a ginger by any chance? ;)

Nope but that could be someone from my extended family.

So, sharing is shining. Where can we see your warez?

Send me some drives!

Long shot here, do you have any videos from The Portland Art Museum channel? https://www.youtube.com/channel/UCXneDVRq1lLde3oREF_cFyA # they deleted (or made private) one of the talks by Richard Mosse and i'd like to see it again as it was a great talk IIRC.

On topic - I've also made a habit of downloading any videos/channels that are super interesting but I feel may also be ephemeral. Some of that ephemerality is often due to copyright claims, performances of classical music by amateurs and such.

Your ISP must not have a cap ... or a very reasonable one.

I don’t even have regular WiFi.. I’m mostly using my phones LTE via hotspot because it’s much faster than anything I can get (in a big city that is) here. I don’t have a cap for that.

Every cell provider (in the US) has a cap (or cap and throttle system), especially for hotspots - I'm perplexed.

T-Mobile (my provider) has a plan with unlimited LTE data and when someone in the EU advertises something as unlimited is really has to be unlimited. I managed to download as much as 2TB within 24 hours with no complains.

I pay 80€ per month (~$92) which I think is fair. I get 25GB roaming data per month to use within the EU.

But most mobile plans do throttle your speed to something unusable after you’ve used up your included high speed volume here too. This can range anywhere from a couple MB up to 50GB/month afaik.

This is going to be one good postmortem.

Mistakes were made... anyways. here's Wonderwall.

I sure hope they live stream it.

On Twitch of course

You're joking, but I like the idea of watching a live postmortem with Twitch chat.

Me too, but I imagine a tsunami of 4head spam.

No postmortem yet?

Getting errors too. First time ever tho so props to the YouTube team for being pretty damn good most of the time.

Are we going to start a betting pool as to why?

I put $20 on DDoS, but I'm hedging $5 that someone spilled their drink on something important and quickly got out of there before anyone noticed.

The janitor needed that socket to plug the vacuum

Experimental Roomba built up a static charge and fried something important.

Wouldn't DDoS be incredibly hard to pull off given YouTubes vast resources? I mean they generate so much traffic all the time and serve so many requests a second I don't know if botnets of that size exist. Anyone here more knowledgeable?

Would it involve hitting whoever/whatever handles their DNS like that one attack about a year ago?

FYI: I legit am retarded to such things and my networking knowledge is...meh. So please correct me.

DNS is resolving because people can get to the site. Because of the distributed nature of YouTube's resources (at DNS as well as other layers), it's very unlikely to be DDoS related. Static content loads, but video content doesn't. Seems like an application error, or a problem with media streaming infrastructure.

Yeah i feel like a DDoS that big would be afecting other google services right now...

Maybe it's a brain teaser - if you can answer how to DDoS Youtube, they'll call you up for an interview.

I could tell you a few ways, but would never work for them out of principle. They can find their own bugs.

I suppose if you had something new and incredibly powerful, this would be a great way to show it off.

Still more likely that YT just made an oopsie.

A few hundred AI learning instances were started to train on youtube videos to "understand" humanity. They have become self aware and started to consume double amount of CPU/GPU/Disk/RAM/Network resource every few seconds - a skynet/super nova moment.

NSA was installing a wire tap and sliced some fiber.

That would not cause a global outage. YouTube has caches in all the major ISPs.

maybe nsa was installing a whole bunch of wire taps?

With the help of aliens since they have a better vantage point of where all the cables are... because, you know, the Earth is fla-... nope. Can't do it. I can't go that far. I can only say "so much" stupid. That's too far.

I bet it’s a core network issue since it went down globally at the same time.. they deploy new binaries slowly so that shouldn’t cause it. Maybe some config was changed without testing?

A. You're not being fun.

2. Good point.

III. One thing I did notice. I had some other videos in other tabs. They still loaded through. I could skip around and it streamed in video like nothing was wrong. When I refresh, bust.

Quatre: Five......

Agreed. I'd suspect it'll be something similar to Google's 2016 GCP outage:

https://status.cloud.google.com/incident/compute/16007

(I'm thinking something like router firmware or BGP).

i think that if this is the case, it's an fast and easy fix, but they haven't solved yet so i think this is not the issue

Is it down globally? I've has some EU people on Mastodon say it's still up for them.

I'm betting it has to do with video indexation.

I was watching a long video just fine, then I searched for something else and clicked on one result. Error message.

> I put $20 on DDoS

I'd take you on that, except I have no anonymized payment account. Otherwise, I'd bet $100 that it's not.

Only Google has the scale to DDoS Google.

Hey! Conspiracy time!

Maybe it's an inner splinter rogue group within Google that's been secretly hating on a Youtube team and decided to get the ultimate payback for like... not inviting them to a lan party...or something...aliens ate all the cheesey poofs.

It's crab people for sure.

I'm not saying it's aliens, but...

It's the sunspot thing. They took down the Hubble, that other telescope, that random lab, and now the frog people Alex Jones warned us about are coming to inject you with the gay and ship you on a raft made out of Michael Moore to Cuba.

All to conceal the real truth about what happened on 9/11 and how it connects to the plot to obfuscate what really happened at Pearl Harbor and how the Australian mafia actually had JFK assassinated to cover up the truth about Marilyn Monroe, Tupac, and Charles Proteus Steinmetz. A clever scheme, no doubt orchestrated by the evil consortium of George Westinghouse, Thomas Edison and Guglielmo Marconi. Hopefully Elon Musk, Robert Downey Jr. and Nikola Tesla will be able to bring the truth to light!

You forgot George Soros

Conspiracy theory time.

The DDoS came from inside the Google servers!

Now see, that I'd believe.

Maybe. Netflix is on the same order of magnitude. China has many websites whose volume is not much smaller than google.

Fat fingered a script would be my guess. Not the first time that caused a 'big4' site to go down.

DNS, it is always DNS.

My vote: Bad configuration got pushed.

I thought they had all sorts of crazy canary deployment processes that will automatically stop/rollback the deployment if the failure metrics start increasing?

Ran out of canaries?

Well, could be an unforeseeable-wack-crazy-bug. Can't plan for every possible failure.

That's very true, there's always a bug somewhere lurking, even if you're Google.

There was a GCP Global network outage maybe a year or two ago, where their canary of a canary (Yo dawg, I heard you like bugs, so I I put a but inside of your bug) caused a cascading network failure.

It was a bad config push -- brownouts fighting with brown outs. Consistency issues. I bet this is happening.

I'll take your $20 bet and raise you to $100. Not a DDoS.

$20 says it was Greg. Again!

My bet is rm -f instead of cp -R.

Somebody probably accidentally powered down the Web:

https://home.cern/about/updates/2012/12/month-1991-web-sprea...

My bet is data loss. Some engineer deleted MySQL table accidentally and they found out their backups are not working.

Joe, the only YouTube support person in the office this evening, accidentally hit the pause button on the YouTube service in his admin console before stepping out for a long bathroom break. Hey, it could happen...

The Saudis dont like all the attention?

Metadata querying I imagine. The search was working, buffered videos and main site were loading, just not displaying videos. (so not network)

I bet on Google Global Cache problem.

YouTube was down for me through my residential ISP (which no doubt has internal caching via GGC) but through VPN was fine.

I was setting up a new network and thought it was my settings! I've been driving myself mad for the last 45 minutes thinking that I had fudged up somewhere! Thank goodness for this post

It was working fine for me until the minute I hit Upload on a video. I was entirely convinced I'd had my account auto-banned or something, and was very confused/frustrated. Couldn't find any mention of it anywhere at that point, then this thread popped up.

It's pretty telling that YouTube is so reliable, and Google is so horrible to their users, that my first instinct was that I got banned for no reason by some bot, rather than the site was broken.

This is what happens when one is the user be not the customer

First thing I see on their Twitter page are fake verified Youtube accounts perpetuating some Crypto scam: https://i.imgur.com/PkEbjH5.png

Typical scam. Problem is verified accounts can change names so scammers steal details for random verified accounts and then use them to pretend to be high profile accounts.

From Ops to Ops, much respect YouTube/Goog and a tremendous pat on the back to this even being such a big deal.

I'll happily, but very-boredly, wait until you're done. :)

Kudos to the Ops team indeed, can't wait to read the PM report.

New Boston Dynamics dancing robot video launched today is probably to blame. :)

(and if you haven't seen it now you have something to look forward to when Youtube comes back)

Ghost of the dancing baby.

Or you can view it now https://twitter.com/YouTube/status/1052213813108453377

Woke up randomly at 3am (utc+2). Opened HN. saw this ask HN

Think I felt a great disturbance on the net and woke up, as if millions of voices suddenly cried out in terror and were suddenly presented with a black screen on their YouTubes. I fear something terrible has happened

This exact same thing happened with me. I woke up at unusual time.

Its working now though.

Very glad to see it's not just me. I was about to think they had a radical new redesign again with everything AJAX'd and decided to make it totally unusable in anything but the very very latest DRM-encumbered browser(s).

Well...this is a good excuse for me to setup a PeerTube mirror for my YouTube channel.

Let's speculate, how much money would you guys guess is being lost every minute of downtime?

A metric @!#$ton? :D

> Estimates for YouTube's annual revenue, nearly all of which still comes from ads, vary a fair amount. But many of the estimates are now above $10 billion. At different points, Bank of America and Mizuho forecast that YouTube would post 2017 revenue of $13 billion and $12 billion, respectively. And in February, Baird's Colin Sebastian estimated YouTube is doing around $15 billion in annual sales. [0]

I think that works out to a bit more than $28k/minute.

[0] https://www.thestreet.com/investing/youtube-might-be-worth-o...

Unrelated, but on a recent network outage (https://status.cloud.google.com/incident/cloud-networking/18...):

  The incident occurred while Google's network operations team was replacing
  the routers that link us-central1-c to Google's backbone that connects to
  the public internet. Google engineers paused the router replacement process
  after determining that additional cabling would be required to complete the
  process and decided to start a rollback operation. The rollout and rollback
  operations utilized a version of workflow that was only compatible with the
  newer routers. Specifically, rollback was not supported on the older routers.

And the postmortem action item is:

  Fix the automated workflows for router replacements to ensure the correct
  version of workflows are utilized for both types of routers.

The action items should have been "1) make this work for these two routers, and 2) make sure no platforms ever get left out again".

This shouldn't have happened, because it should be standard practice to test both upgrade and rollback on all your gear. Network gear vendors do this as standard practice before they ship new gear with upgrade instructions. Google can throw together end-to-end automated tests of upgrades/rollbacks and refuse to perform maintenance until tests pass.

The bigger postmortem question should be, why was the change allowed at all if the platform didn't support rollback? Additional action item: "3) don't allow changes if the platforms don't support and have successful rollback tests".

Now, did they need to test rollback? Maybe they don't mind portions of CloudSQL, Spanner, Storage, BigTable, and AppEngine being down for 41 minutes in one zone. But if they're not even testing rollback for BGP changes, what else aren't they testing?

...Also, lol, they realized in the middle of an upgrade that they didn't have enough network cable? Maybe add an extra action item: "4) count how much network cable you have before you start replacing core routers"

Spanner and Storage at least are region-wide, so a single AZ going down shouldn’t affect customers on those products.

Some analysts predict YouTube to bring in $15 billion in ad revenue in 2018. If so, they are losing around $28,000 for every minute they are down.

I think you meant $280,000?

Nah, $28k - https://duckduckgo.com/?q=15%2C000%2C000%2C000+divided+by+52...

https://www.wolframalpha.com/input/?i=$15+billion+%2F+(minut...

-1 on my mental math, +1 on duckduckgo's UI for arithmetic.

Google's is more impressive: https://www.google.com/search?q=15+billion+divided+by+minute...

Or even better: https://www.google.com/search?q=%2415+billion+per+year+in+do...

nah, I think 28,000 is right

Could be more (people stop watching and go outside) or less (people go back later to watch what they would've watched anyway, and free advertising for everyone talking about it) depending on several things.

Yup, this is all I get when I load it: https://www.dropbox.com/s/thbiecwx04on7d5/Screenshot%202018-...

Interestingly, this article [1] mentions a 2008 YouTube outage caused by a BGP misconfig during an attempt to ban YouTube in Pakistan. Included is a screenshot similar to what we've seeing with this outage. Possibly related?

[1] https://blog.cdemi.io/beginners-guide-to-understanding-bgp/

Can Netflix handle the extra load? Haha.

And Twitch.

I spent the evening binging on Agents of SHIELD, so I assume yes. ;)

A lot of different sites are down right now. I wonder what's going on.

what are some others?

PornHub is down for me.

it just got serious

Not for me

Quora is down for me.

Same to me.

Quota still seems to be up for me, South West England.

Seems fine for me.

Ironically, outage.report, a website for reporting downed web services is also down.

So signs point to it not just being youtube, a number of websites seem to be having issues. May be a provider is down?

Maybe outage.report is down because so many people are hitting it after YouTube going down...

discourse is down for me.

GCS issues maybe?

GCP [1] and G Suite [2] appear to be green.

[1] - https://status.cloud.google.com/

[2] - https://www.google.com/appsstatus

Fortunately, I still get to see a banner advertising their streaming music service, or to watch the World Series through Youtube.

Nice to see their ad-service platform isn't impacted. /s

It's back! I've been refreshing every second - That must've fixed it.

We take a special Joy when things break, I wonder if there is a word for that?

Shadenfreude?

It's close, that's pleasure in someone else's misery? maybe it is the same thing after all? But I get this feeling when the power goes out too.

Schadenfreude?

A made-up word and a nice video, now that YT is back up: https://www.youtube.com/watch?v=wqovm-Io2D0

Wow! Yeah that feeling of the big storm

videolectures.net is still up, so if anybody is bored...

Here's a good one:

http://videolectures.net/mackay_course_01/

David Mackay's videos and books (freely available) are all great. He was an excellent teacher, and doer.

Yep, that's what I'm seeing. I also came directly to HN to see if others were, and this is the top thread. Well done HN!

I wonder how much each minute of downtime is worth in revenue numbers?

Still not up here in Metro Boston. I eagerly await the (I assume) super interesting explanation of what went wrong tomorrow.

Back now in Boston.

Child getting crazy with @youtube down. LoL, they don't understand that services can get offline. #youtubeisdown

I'm trying to explain it to a very distraught 2.5 year old.

Tell them YouTube is sleeping. My toddler explains every sort of absence as whatever it is (inanimate or otherwise) sleeping.

Youtube-dl is also not working surprisingly. It's crazy how I still assumed it was my network and not youtube. Speaks to how much I depend and rely on youtube being available as a staple in my life. For me the error just says webpage is unavailable. I am in NJ so east coast has a large outage probably.

Melbourne, Australia here reporting a home page which loads with blank placeholder content. 2018-10-17 12:52.

Edit: one minute later, content and images load but videos do not.

Edit: back to placeholder home page.

Ha I thought my Chrome had some hanging process that resulted in js engine error. So I switched to Safari and same error...

I tried my phone, laptop and remote server and nothing worked. But I still thought it was a problem with the video until I read it on Mastodon.

twitter noticed: https://twitter.com/hashtag/youtubeisdown

https://googleads.g.doubleclick.net/pagead/id returns a very invalid json to me, starts with: )]}'

That's just the common magic XSSI prefix for JSON.

Yeah I'm getting the following error:

500 Internal Server Error Sorry, something went wrong.

A team of highly trained monkeys has been dispatched to deal with this situation.

If you see them, send them this information as text (screenshots frighten them):

I get the same for individual videos.

https://oopsiewhoopsie.com/ (Warning loud / NSFW)

https://twitter.com/TeamYouTube/status/1052373937839980544

@TeamYouTube's Twitter stream now fully overloaded with "apologize" and "we a working" tweets...

Look like nobody will sleep now in YouTube HQ.

How many kids will be born in 9 month?

There's a bunch of engineers in some conference room right now probably going "oh shit...." while slowly accepting the impending doom of a late night at work.

I had a couple youtube tabs open from this morning, the videos still play and it's not cached, the videos are hours long. But any new videos aren't working.

Ya I was listening to a podcast on youtube that was over an hour and no issues until I clicked home to see something else afterwards.

Wow, thought my internet was just being weird. Guess not.

I just went to post this - it does appear so:

http://downrightnow.com/youtube

Newly accessed videos started failing. Tabs/videos with partially played videos that were paused, continued to work.

I went and made dinner.

Looks like outage report itself is down too.

https://outage.report/youtube

I get an almost empty Youtube interface, Youtube Music is also down, Google play music is working a little bit slower than usual. I'm in MX.

YoutubeTV down too. Big footprint outage.

Any idea how much money they are going to lose?

Tried to decode text as base64, but nothing interesting. Either it's not base64 or the original is in some binary format.

It’s an encrypted stack trace, to allow G employees to debug reports but not allow everyone to see into their systems.

Yep, me too. Was just trying to figure out if it was just me and reloaded the page here to check my connection.

Hmm the YouTube app on my TV timed out I just switched to Netflix didn't figure it was a global issue

If YouTube can go down, I feel less bad about my crappy code derping out in production once in a while...

I have tested the status of youtube to confirm if it is still down and it is T.T

https://snippet.uilicious.com/test/public/RuGwPjbwFX1QU5EVhv...

Looks like they're up again. :) Was that like a full hour of downtime. I hope they publish a postmortem. Like others have said, for me, Youtube has always been there. It is a odd feeling to have experienced [albeit shortly] internet sans Youtube [again].

It's back up judging by Google Trends https://trends.google.com/trends/explore?date=now%204-H&q=yo...

Some specific pages are giving me internal server errors, e.g. https://www.youtube.com/user/MLB?pbjreload=10

How about this one? https://www.youtube.com/user/test

They're back online https://twitter.com/TeamYouTube/status/1052393799815589889

Well I was trying to load a video on YouTube a second ago and searched worked but the video said "error tap to retry". I opened HN to verify my internet and saw this. So it's broken for some of us ;)

Wow, so i was trying to watch a video on the train at 4:45pm pst and the video was all jacked and loading weird. I gave up and went on to something else... so that must have been an early symptom of the outage.

This seems to be GCP related. I know the status page is all green currently... but we have a lot of services using GCP products and some are failing and showing increased latency. Pubsub failures as well.

YouTube is not built on GCP.

Why would you assume this previously known tidbit to be true for the rest of Youtube's life?

Now we all get track how our user traffic increases when YouTube is down...

Google search very slow / down for me. Anyone seeing the same?

I noticed this and my first thought was that my phone needed to reboot. Come on, Youtube doesn't go down... Then I went to a different site and realized it was an outage...

Imagine how many pages serving content, product demos, tutorials, etc are affected by this yt outage.

Good case for hosting your own content/diversifying.

Don't worry. A team of monkeys is working on it

Yes.

https://twitter.com/hashtag/youtubeisdown

American Express is also down!

https://www.americanexpress.com/

A coincidence, I am willing to bet.

Working here.

clear your cookies and try again... they have a cookie bug that overflows and breaks their whole site all the time.

YouTube TV is throwing errors on the "Home" page, but the "Live" page is working fine. Some app-side issue, I guess.

I wonder if there's any link to the Brazilian elections, the candidates are using YouTube heavily and it's prime time there.

Bolsonaro has not taken over the world ... yet. 17

Still down for me. Nothing is working. Loads and then the interface seems to blank itself out. App just generates playing ID errors.

It just started working for me again, in Sweden.

Same, in the US

Getting 503's on the Android app on my end.

What a good time to learn about Peertube:

https://joinpeertube.org

https://tv.youtube.com/ is also down

I also can't access Google analytics

Works for me still.

Suddenly I have nothing to do at home..

Not me, I watch netflix, read HN while playing PWI and tinking with my Amiga CNet BBS running inside WinUAE.

Check the latest G+ updates?

oh, wait...

Right now the front page loaded, but no video played and a refresh went back to just placeholders.

The UI is generated client-side

I meant it loaded completely, thumbnails and descriptions. It came back shortly after my comment and has responded as fast as ever since.

Has YouTube published their outage postmortem?

Not working here in India as well.

I'm curious what happens to Netflix and Amazon Prime Video during moments like this.

Down for me as well in Colombia.

Ah, Colombia. I was there about 20 years ago and it's still my favourite country by far. <3

Hacker News is up for me.

In all seriousness, YouTube music and videos are down for me in California.

Thank god my daughter fell asleep before this happened. It'd be a long night.

Getting "An error occurred. Please try again later" playback errors here

It seems like the interface is working, but the actual content is failing to load.

Same

At first, the whole interface was failing to load. There was a 500 error message and a monkey with a wrench.

Embedded videos also embed the monkey on play.

Looks like it's back now.

Yep, I actually got my subscription list again and could continue the video that stopped playing when it went down.

Yep, down for me here in NYC

Doesn't work in New York, search functionality works but no video load

Direct videos are up again, but it looks like other site features are wonky

Not working here in Vietnam

Youtube is love...Youtube is life(or so we thought). R.I.P 2005 - 2018.

thought it was just my connection ... indonesia can be like that sometimes and believe it or not i came on here to check if anyone else complaining ... seems like its true and not just my isp

Here in bangladesh too... YouTube not working ... very surprising ...

Probably a good time to torrent what with the intertubes unburdened.

What I would give to read the postmortem and $$$ cost on this one.

Unfortunate timing, right about when San Bruno leaves for the day.

I wonder if this is a Vitess issue. It’s pretty unusual if so.

Down and out in London.

Works fine for me, but I'm on a regional fiber provider.

In Brazil, im middle of jungle (Amazonas) Youtube Down!!!!

Yep down for me too, 500 Internal Server Error. (Montana)

YouTube app does not load any videos in Romania either.

And it's back!

Not for me.

nope.

Look at all that work that’s going to get done now.

I get a 503 network error on the mobile app from western canada right now. NewPipe will complete searches but gives a content not found error on trying to load a video.

Roger that. “Invalid response received.”

Gmail isn't working well either.

down in switzerland, phone and laptop

Yeah - having some problems myself.

https://outage.report/youtube

Maybe related to the #libssl CVE ?

I have YouTube TV... No TV for me.

my aws-based builds just went from 5.5m to > 15m about when this started. aws issue?

Youtube is owned by Google (since 2008). Would be highly surprised if they hosted anything on aws

A company like google would never rely on just a single company like google!

All greens according to https://status.aws.amazon.com/

everybody just started working

back up and with a new youtube music streaming service ad I haven't seen before.

Any idea or guess on the cause?

Yeap is down for me as well :(

Yep. It's down. "pbjreload=10". Seems like an old issue related to cookies. But who knows.

...I remember where I was...

Sign of the end times folks.

Frogs, Locust, Hail, and YouTube going down.

Cloudfare site also affected

Not working in Thailand too

Youtube down in Netherlands

Issue with Google AMP CDN?

YouTube not working for me

Think of the children!

Down in India as well.

Still down east coast

Not working in Poland

Down in New Zealand also

I blame Shane Dawson

Same as South,Korea.

probably an unhandled undefined error in the JS code.

And it's back!

Down in Canada too.

Not working for me.

major outage worldwide i'd say. im from brazil

Not working in NJ

All fixed now...

down in switzerland, on phone and laptopn

It's back!

Nope, not yet.

2:40 UTC/10:40pm EDT: up for me.

Just checked, not for me https://screenshots.firefox.com/xbO5zc86pso83oH8/www.youtube...

Up for me now!

Down in boston

yes, i am from Algeria North africa

YouTube is back. Should be all back to normal.

Facebook definitely down. Youtube still fine.

just tried and its working now

not working in Hungary also

Down in MD

Inaccessible from Europe, 3am UTC

It just came back online for me.

Just came back up for me...

down in Los Angeles

Not working in Ireland

It's back up.

is it black?

Down for me.

Did Google just decided to sunset YouTube?

Hardly anyone uses it. By Google standards, I mean.

They probably have some similar yet slightly worse product that they've decided to introduce instead. Such is the life of a google app.

The Google One experience? Make sense

We shall call it, Google Video.

Family member said “did they take it down?”

Maybe, and we’re not sure who “they” is yet.

They've decided to release the SeeYa and HearYa apps which let you watch video content and audio content separately so they can provide you with the most tuned experience. You can't use those in G-Suite though so still use YouTube there for now. Also YouTube Premium is immediately cancelled despite being popular and profit making.

We should all develop a song to remember and never forget this day.

A long long time ago, I can still remember how those videos used to make me smile... (Take it)

And I knew if I had my pick, that I could make those people click, and maybe they'd be happy for a while

or "And I knew if I was slick, ..."

subscribers want me to deliver another video with facts and figures

But October made me quiver, with every UDP packet I'd deliver- bad JS on the door-step, the DOM couldn't take one more step

Said bye bye and pew pew die the drives in the levy but the drive got fried the good old boys are drinking whiskey and rye singing this the day that YouTube died

"Where were you when YouTube died."

"On HN, of course."

"And why were you not offline?" (continuing song lyrics i think we're making)

We should set it to "American Pie" and make sure to start off mentioning how we were all still mourning the loss of Paul Allen when this happened.

Let's make sure to make this the next stanza ;)

Where were you when YouTube died?
On HN, of course.
Putting off dealing with my divorce.

And wishing Paul Allen were still alive.

I think it goes more like

Where were you when YouTube died

On HN of course

and why were you not offline

putting off dealing with my divorce

I wish paul allen was still alive

..?

(has more of a feel to it, alternating rhymes.)

warning Reddit style comment ahead

Woah. This thread just hit me in a - ridiculously - profound way.

The day the Internet died.

“Putting off dealing with my divorce.”

listening to a podcast on soundcloud, funny enough.

Just add "disable_polymer=1" to every URL

https://www.youtube.com/watch?v=kJQP7kiw5Fk&disable_polymer=...

I use a TamperMonkey script that does that for me, still down.

Looks like it.

New NSA/google patch gone wrong?

Devastating for streamers. Google must compensate

Sure, Google should give all streamers a 500% refund on their YouTube fees, which comes to, let me see, zero dollars.

i picked a bad day to quit facebook live

I get a mostly empty page with a lot of pro-SJW stuff, which is the opposite of what I normally get. Not enough Clapton.

Reply on news.ycombinator.com