Hacker News Reader

Hacker News

Comment by lpmitchell | Hacker News Reader

Full thread

lpmitchell

This seems to be impacting a number of huge sites, including the UK government website[0].

[0] https://www.gov.uk/

https://m.media-amazon.com/

https://pages.github.com/

https://www.paypal.com/

https://stackoverflow.com/

https://nytimes.com/

Edit:

Fastly's incident report status page: https://status.fastly.com/incidents/vpk0ssybt3bj

View on HN

Fastly Engineer 1: Seems like a common error message. Can you check stackoverflow to see if there's an easy fix?

Fastly Engineer 2: I have some very bad news...

Haha! That sounds highly plausible!

Stack Overflow is down, can someone tell me how to declare a static multidimensional array in C++?

Google and DDG surface SO results cached within their own page. Here’s the copied answer:

int main() { int arr[100][200][100]; // allocate on the stack

    return 0;
}

Oh man, how do we keep a pocket copy of SO? All of our jobs depend on it.

no they don't, but if yours does you can download a complete datadump of SO from them.

You can use kiwix (https://www.kiwix.org/en/).

It is an open-source software that allows you to keep and read offline static versions of websites in a specialized archive format (zim-files)

It was originally designed to allow you to read wikipedia offline, but there are also dumps of stackoverflow available on the relevant page : https://wiki.kiwix.org/wiki/Content_in_all_languages

Here, just pin the underlying IPFS object, or use this one hosted by cloudflare: https://ipfs-sec.stackexchange.cloudflare-ipfs.com/

You can download the database dump from https://archive.org/details/stackexchange.

https://kapeli.com/dash also has the ability to download offline archives of SO. Its interface is very good.

Well, with SO, at least you can search on Google and view the version cached by Google just fine.

With Reddit however, these days almost all comments are locked behind “view entire discussion” or “continue this thread”. In fact, just now I searched for something for which the most relevant discussion was on Reddit; Reddit was down so I opened the cached version, and was literally greeted by five “continue this thread”s and nothing else. What a joke.

Try “site:old.Reddit.com”

That's not going to work.

  $ curl https://old.reddit.com/robots.txt
  User-Agent: *
  Disallow: /

Also, even if search engines are allowed, old.reddit.com pages are not canonical (<link rel="canonical"> points to the www.reddit.com version, which is actually reasonable behavior), so pages there would not be crawled as often or at all.

Yeah it's crazy how bad user-hostile reddit.com has become. Fortunately old.reddit.com is still available, but for how long? If only Javascript did not exist, it would be impossible for UX people to come up with something that bad.

> only Javascript did not exist, it would be impossible for UX people to come up with something that bad.

Arrange the html so that the list of comments is at the end (via css). Keep the http connection open, have the show more button send some of request, and when you receive that request send the rest of the page over the original http connection.

As usual, solve people problems via people, not tech.

Well technically everything is possible. But Javascript was precisely designed to encourage this kind of patterns.

> As usual, solve people problems via people, not tech.

So true..

How would you make the button send a request without js and without navigating to another page?

Maybe css to load an image on :active or is there some better way?

Here are two robust techniques that I haven’t seen actually employed in production for maybe fifteen years:

① A submit button or link targeting an iframe which is visually hidden. (Or even don’t hide it. If only seamless iframes had happened, or any other way of auto-resizing an iframe: relevant spec issues are https://github.com/whatwg/html/issues/555 and https://github.com/w3c/csswg-drafts/issues/1771.)

② A submit button or link to a URL that returns status 204 No Content.

(CSS image loading in any form is not as robust because some clients will have images disabled. background-image is probably (unverified claim!) less robust than pseudoelement content as accessibility modes (like high contrast) are more likely to strip background images, though I’m not sure if they are skipped outright or load and aren’t shown. :active is neither robust nor correct: it doesn’t respond to keyboard activation, and it’s triggered on mouse down rather than mouse up. Little tip here for a thing that people often get wrong: mouse things activate on mouseup, keyboard things on keydown.)

Mhhh, iframes all the way down. Could make a nice experiment.

Yep:

.button:active { background-image: url('/some-reference-thats-actually-a-tracker'); }

“Continue this thread” links don’t depend on JavaScript at all.

“View entire discussion” couldn’t be implemented perfectly with <details> in its present form, but you can get quite close to it with a couple of different approaches.

I think the infinite scrolling of subreddits is about the only thing that would really be lost by shedding JavaScript. Even inline replies can be implemented quite successfully with <details> if you really want.

When it goes away you can try teddit.net

Why wait? Teddit has been a great substitute for reading in a mobile browser, and making an iOS shortcut for transforming Reddit links was pretty straightforward.

Yeah I’m going to stop using the platform when they get rid of this . Not interested

Why wait? You are wasting your life away.

And commenting on HN is any more productive?

Impossible? Man, it's crazy how fast people forget things like good old fashioned <form> GETs and POSTs. It would obviously be a full page refresh, but other than that the same awful UX could still be implemented.

Reddit's attempts at dark patterns are embarrassing from all perspectives. If you use dark patterns it's a laughably abysmal implementation. If you abhor dark patterns, it's a frustration.

It's just enough to annoy you but not enough to make everyone leave the platform

Not _yet_, the same was said about Digg once.

The mobile version is literally unusable. Half the subs show an error and you can't load most comments.

I think it's because there haven't been any interesting alternatives. I know if I ever see one I'll probably switch in a femtosecond.

On the same day that old.reddit.com stops working I'll leave.

Preach!

That and third party access to their API.

Sync is so much better than the official app it's not even funny.

They've actually done a masterful job of finding this balance. I've been on reddit for 15 years and would have quit if they didn't leave the old interface available.

I honestly thought Reddit would die when they introduced Reddit awards, it seemed like such an obvious cash grab. You can't underestimate the amount of community momentum that the site has though.

Its one of the three major sites I use. Yeah...

Eh, as far as funding methods go, letting people throw away money is not the worst one.

I wanted to suggest site:old.reddit.com since I use that version with automatic redirect, but this:

https://old.reddit.com/robots.txt

is very different from this:

https://reddit.com/robots.txt

I guess there is a market for search engine (maybe accessed through tor) which does not care about robots.txt, DMCAs, right to be forgotten etc. Bootstrapping it should not be that hard since it can also provide better results for some queries since nobody is fighting about the position until it's widely known.

I'm not sure how far are we from being able to do full text internet search. Or rather even quote search, preferably some fuzziness options. That would be cool, Google's quotation marks were really neat back when they were working.

> I guess there is a market for search engine (maybe accessed through tor) which does not care about robots.txt, DMCAs, right to be forgotten etc. Bootstrapping it should not be that hard since it can also provide better results for some queries since nobody is fighting about the position until it's widely known.

That’s not going to happen before Cloudflare is dethroned. See this recent thread for some perspective: https://news.ycombinator.com/item?id=27153603

And even if there’s no Cloudflare, large sites that people want to search will always find ways to block bad bots.

The only thing I can think of that might work is using crowd-sourced data, with all the problems that come with crowdsourcing.

Sadness.

There is a solution for all this mess and I'm blocking HN and a few different domains until I implement at least the first step after which I can share it here.

Wonder what the story is behind these two...

    User-Agent: bender
    Disallow: /my_shiny_metal_ass
    
    User-Agent: Gort
    Disallow: /earth

Easter eggs

It's neckbeard humor.

I'll have you know our beards were neatly trimmed when we added those.

I know you, and I find that hard to believe ;)

It might be related to the time few years ago when Google added exclusions for user agent t1300 in regard to its founders. Gort seems to be a robot from old scifi and bender might be something similar.

Bender is from Futurama, Gort is old classic scifi: https://en.wikipedia.org/wiki/Gort_(The_Day_the_Earth_Stood_...

I reod some people use false slugs in the robots.txt as a honey pot of sorts. IPs that actually read the robots.txt, ignore the disallow, and still access the uri are outright banned.

Ha, that would have been a really smart idea! Sadly we didn't think of that at the time. But we had other honey pot URLs.

Then when a flamewar breaks out you just have to get your adversary to click a link to get them IP banned.

That's the good old Easter eggs, perhaps a memory from when Reddit was a nice place. They stop appearing and are replaced by dark patterns once sites jump the shark.

Just some fun humor we added for other nerds who read robots.txt files.

try editing your hosts file to redirect reddit to old.reddit

/etc/hosts

reddit.com old.reddit.com

www.reddit.com old.reddit.com

np.reddit.com old.reddit.com

I am archiving subreddits on Github in plain-text org-mode. If you have some subreddit in mind, open an issue, and I'll create an archive repo for it.

- https://github.com/NightMachinary/r_HPfanfiction

- https://github.com/NightMachinary/r_rational

Haha! That explains why the internet was down for a while!

But https://news.ycombinator.com/ is UP! :) Prepare those HN servers for massive influx in 3...2..1..

I am already here

While we're here.. I am a bit surprised to see how many sites use Fastly. As a dev I've always been happy with Cloudflare.

Me too, but in a way I'm even happier knowing that not everyone does and something else popular exists too.

Now imagine how many sites would go down if it was CF

Do more rely on Cloudflare? Because this felt like it was more than half the internet, certainly more than half the biggest sites.

I think so, Fastly seems to have a few huge enterprise clients while Cloudflare seems more balanced (and larger)

I think that Fastly starts at $50/month, no free tier. So that would preclude small or not-profit-motivated sites from using it.

No need to imagine! Just search HN for "cloudflare outage" and you'll see that it happened several times over the last few years

Is this a call for competition? I regard Cloudflare as state-of-the-art in terms of security and ease-of-use. I certainly hope their knowledge replicates across other organizations. As of now they're still building highly impactful tools that are easy to use and that noone else quite provides. I don't really expect another organization to match them given the strength of their current leadership. I think they've built in a head start for awhile.

They have also been responsible for one the worst security incidents ever:

https://news.ycombinator.com/item?id=13718752

Only discovered we should not forget,due to the good graces of google project zero.

A certain those of skepticism towards any technical offer out there would be advised.

I like Cloudflare's post mortems, and I like how they fight back against patent trolls. For me as a dev they are #1.

Do you have experience with the competitors?

I prefer tech that I can use both at work and on hobby projects at home.

To that end I've only used cloudflare and netlify. The others have too much friction to try out. I expect I would get experience on the job if necessary.

> Cloudflare as state-of-the-art in terms of security and ease-of-use

Depends whose security. I value my security dearly and that's why i use the Tor Browser. Cloudflare has decided i cannot browse any of their websites if i care about my security (they filter out tor users and archiving bots agressively) so i'm not using any cloudflare-powered website. Is it good for security that we prevent people from using security-oriented tooling, and let a single multinational corporation decide who gets to enter a website or not? In my book creating a SPOF is already bad practice, but having them filter out entrances is even worse.

Also, are all of these CDNs and other cloud providers are solving the right problems?

If you want your service to be resilient against DDOS attacks, you don't need such huge infrastructure. I've seen WP site operators move to Cloudflare because they had no caching in place, let alone a static site.

If you want better connectivity in remote places where our optic fiber overlords haven't invested yet, P2P technology has much better guarantees than a CDN (content-addressing, no SPOF). IPFS/dat/Freenet/Bittorrent... even multicast can be used for spreading content far and wide.

Why do sysadmins want/use CDNs? Can't we find better solutions? Solutions that are more respectful to spiders and privacy-minding folks with NoScript and/or Tor Browser?

Speaking for myself here, I don't see how people can use the web without javascript. As for Tor, you're routing other people's traffic while they route yours, so I can understand how such connections would be blocked given that blocking IPs is still a method for mitigating security issues, and you can't determine the IP of a Tor browser.

> I don't see how people can use the web without javascript.

Its pretty easy: browse marked up documents, not applications. If some developer conflates the first for the second, move on.

> As for Tor, you're routing other people's traffic while they route yours

Using Tor doesn't imply that your machine is also a Tor exit node.

interesting thought ... a new type of 'to big to fail' ?

No more than a transatlantic cable...

Where is Akamai in this comparison?

Fair point. Maybe Fastly is more akin to Akamai given it seems to be more enterprise-y. By market cap, Cloudflare is 26 billion, Akamai is 18, and Fastly is 6.

Fastly's free offering gives you "$50 worth of traffic" whereas Cloudflare has a perpetually free option. And for Akamai you have to apply for a free trial.

This is market cap, but if you look at amount of traffic you have Akamai estimated at 15-30%, CF at 10%.

So if it would go down, it would cripple vast amount of internet.

Akamai is balls deep in video streaming, which is probably the most bandwidth/traffic intense thing for a CDN to dabble with. My guess is that CF has much more diverse traffic. Hence the fallout from an interruption would be quite different.

Not quite, Akamai is more large corp centric (they don't serve average Joe) besides that they do also security. If it went down you would get all of sudden e.g. a lot of DDOS possible.

Google's Firebase platform uses Fastly so that's a significant chunk of the web.

New error now, hopefully fix in progress.

Fastly error: unknown domain: www.fastly.com.

Details: cache-syd10161-SYD

>The issue has been identified and a fix is being implemented.

According to the status page.

That doesn't take away their embarrassment. It's mean how many websites rely on fastly. Twitter hasn't been loading emojis in a while, and I believe it's for the same reason.

Might not be the case anymore, but a few years back, Hackernews was just running on a single server.

fairly sure it still is.

Amusingly, the Stackoverflow 503 page has a typo:

  Error 503 Service Unavailable
  Service Unavailable
  
  Guru *Mediation*:
  Details: cache-lon4236-LON 1623146049 854282175
  
  Varnish cache server

Looks like an error (typo) with Varnish .. it's the same on multiple sites.

Maybe a good way to work out which versions are being used.

Same 503 error message on the actual http://fastly.com/ website

It's interesting because Varnish gets it write in their docs: https://varnish-cache.org/docs/trunk/users-guide/troubleshoo...

> It's interesting because Varnish gets it write in their docs

Sorry I couldn't help myself...it's too funny to misspell "right" in a thread about spelling mistakes

;)

Mruphy's Law in action

Isn't it Muphry's Law? Or was that another example :)

https://en.wikipedia.org/wiki/Muphry%27s_law

this is devolving quickly into an r/something

https://news.ycombinator.com/item?id=27444585

I don't see the typo? definitely a fastly problem though..

Fastly error: unknown domain: numpy.org.

Details: cache-pdk17841-PDK

The typo is "Mediation" instead of "Meditation", I think.

Mediation vs Meditation I think.

I think Varnish uses mediation intentionally though, it was this way 7 years ago when I last used Varnish.

Does not seem like: https://github.com/varnishcache/varnish-cache/blob/606977bbf...

According to the Wikipedia page, it is intentional:

https://en.wikipedia.org/wiki/Guru_Meditation

Or did one of you already edit the Wikipedia page to reflect this discussion on hn?

It appears the mention of "Mediation" was made very recently, likely in response to what's currently going on.

https://en.wikipedia.org/w/index.php?title=Guru_Meditation&d...

And it seems to be incorrect, since this "spelling variation" is only used by Fastly and not part of Varnish?...

I assume "Guru Mediation" is supposed to be "Guru Meditation", a reference to the way AmigaOS used to describe system failures.

Mediation is a real word though?

That is true, but the typo is because they presumably meant to reference this: https://en.wikipedia.org/wiki/Guru_Meditation

That wiki references the Varnish version.

I think it's an intentional alteration of the original, given the context.

That edit was added after today's fastly outage began.

Even so, mediation makes sense in this context given fastly's business model.

It could be a typo or an attempt to be clever.

...and naturally someone has already updated the page today to include (and highlight) its use and mispelling on Varnish.

Someone (I can't unfortunately due to IP block) needs to change that. The part about the spelling is false, apparently [1] it's an intentional change by Fastly so that they can tell if it's their own Varnish or a customer's Varnish that is throwing an error.

[1] https://news.ycombinator.com/item?id=27433139

I assumed it was an intentional pun.

Fastly modified the Varnish error to ensure that it is known if the error is returned by Fastly's Varnish or by the origin's Varnish should the customer run their own Varnish on the origin.

We use Fastly (and our site is down too) but I asked them about this a couple of years ago. It is deliberate. They said it was so they can tell if it is their Varnish service or the customer's Varnish service that is down

This comment is correct. I made that change ages ago. Amused it's still there.

This seems like it's intentional given the context.

also https://www.reddit.com (at least in Netherlands)

edit: 12:05 up again for me, no images or custom fonts loading though ... and down again 1 minute later

edit: 13:01 reliably up again for me

Down in UK

Down in US. Also Imgur, which is closely related

Down in india

Same here in Germany: imgur and reddit are down, plus a bunch of other sites.

Same in France

Up for me but no images mate.

Down in SA

> potential impact to performance

So it is a "performance" issue when all pages give a 503.

Does the 503 page load fast(ly) or slowly?

I wonder why Amazon is not using Cloudfront for their own website.

Yeah, this is what makes me feel this is more an AWS thing

It sure looks like a AWS error, even Amazon.com is mostly down.

The m.media-amazon.com domain (and a few other CDN'd domains that they use) are running through Fastly:

    nslookup m.media-amazon.com
    
    Name:  media.amazon.map.fastly.net

It is very interesting that they are not using CloudFront!

Why?

Good to have 3rd party redundancy, time to fail over to something else now I'd think though.

They already have:

  $ host m.media-amazon.com
  m.media-amazon.com is an alias for c.media-amazon.com.
  c.media-amazon.com has address 99.86.119.84

(which is a Cloudfront IP)

Yep they did exactly this and are now running on cloudfront

really, m.media-amazon.com seems to have a very short TTL (showing 37 seconds right now) and has been weighted to cloudfront now.

Amazon is also known to use Akamai. Sure, Amazon relies heavily on AWS, but why should it surprise anyone that a retail website obsessed with instant loading of pages decides to use non-AWS CDNs if the performance is better.

Even if CloudFront became the default, I'm certain amazon.com would keep contracts with fastly and akamai just so they can weight traffic away from CloudFront in an outage.

AWS is reporting no issues across the regions:

https://status.aws.amazon.com

AWS is almost never reporting issues on this page.

AWS don't report outages until it's undoubtedly them.

looks like amazon.com started using fastly in May 2020 (https://www.streamingmediablog.com/2020/05/fastly-amazon-hom...) so it's not an AWS thing

Fastly deploy their own hardware, (That's one of their selling points) I don't think they rely much on AWS, maybe just for network interconnection?

Fastly doesn't run on AWS.

Fastly gives them the edge performance they need without having to build it themselves. They have been a customer for a while I think.

But they have competing products through AWS.

I imagine it works well for the whole business that they allow product teams to use the best cloud tools for the job rather than requiring them to use AWS for everything. If AWS is forced to compete even for Amazon.com's custom, that should make the whole company more resilient to long term technical stagnation.

AWS Route53 and Cloudfront are direct competitors to Fastly.

That's how good Fastly is. Outside of this issue it's a great service.

Cloudfront, by Amazon's own admission, specialises in high bandwidth delivery (ie huge videos). Fastly has consistently better performance as a small object cache, which makes it the choice for web assets

https://www.streamingmediablog.com/2020/05/fastly-amazon-hom...

I wonder why amazon.co.uk uses Fastly and not CloudFront?

Must be more than fastly. Heroku is also down.

That's also because of fastly, I've got this response from the Heroku dashboard:

Fastly error: unknown domain: dashboard.heroku.com.

That doesn't appear to be the case does it? Amazon sites are all working fine, at least for me

Their CSS and JS were down for a few minutes. I was able to login to Amazon but the entire site was in Times New Roman, but was fixed a few minutes later

I imagine they use a few different CDNs for things like this.

Good thing we use Cloudfront and Cloudflare where I work.

> Statuspage Automation updated third-party component Spreedly Core from Operational to Major Outage.

> Statuspage Automation updated third-party component Filestack API from Operational to Degraded Performance.

Oh, right. :-D

Don't get me wrong, I love the proliferation of APIs and easily-integrated services over the past 20 years. We're all one interdependent family, for better and for worse.

CSS/Javascript at https://github.com/ appears to be down as well making GitHub quite unusable.

GitHub Pages appears to be down too, taking an awful lot of sites offline

Github is working fine for me in Canada but others aren't. Tried without browser cache too and it works okay.

EDIT: Most sites seem fixed now here in Canada. Tested stackoverflow, reddit, GitHub, PayPal, gov.UK and all worked fine.

Yeah, things are mostly back now. Including my personal GitHub Pages site :)

Yikes seeing just a "connection failure" on Paypal is something else.

edit: PayPal looks be back up at least in US East but when I turn off my VPN and access from Asia I get "Fastly error: unknown domain: www.paypal.com."

Now I'm seeing a 503

> Monitoring The issue has been identified and a fix has been applied. Customers may experience increased origin load as global services return. Posted 4 minutes ago. Jun 08, 2021 - 10:57 UTC

Looks to be working again my end.

Interestingly, Twitter only has its emoji SVGs down.

And this is (one reason) why using images instead of actual emojis is such a stupid idea. Why, Twitter, WHY?

Branding! (Fun fact: Hacker News strips emoji.)

How does it strip it? Test:

Edit: You are right. It got rid of the emoji after Test.

err, to make representations platform independent?

That sounds antithetical to the purpose of emojis.

Same reason why Gmail uses their own emojis rather than the system ones — (as said above) branding. When you send a tweet, Twitter wants it to look identical across all devices. The classic native UI vs cross-platform UI debate in a nutshell.

Cool, so instead of actually serving text, they could also just serve up little SVGs for each letter. Because god forbid the recipient chooses a different font than Gmail!

Which is why Slack style :tofu_on_fire: emoji notation is genius

You're pretty much defining webfonts!

Indeed. Another abomination.

That's not a minor UI issue.

Twitter is a media between people. Removing emoji representation differences on user devices is a way to hopefully reduce misunderstandings between users.

Vendors don’t even agree on whether the :gun: is a revolver or an automatic or space ray guns or even water guns, btw it’s an 1911 in original DoCoMo emojis

1: https://blog.emojipedia.org/content/images/2018/04/microsoft...

Sure, that's a benefit of emojis being semantic. If you want 'SFW' emojis, you can get them. Converting them to images makes that impossible. And uses vastly more bandwidth, makes them impossible to copy+paste, probably has accessibility issues, etc.

https://deb.debian.org is down too which borked my installation.

https://hackage.haskell.org/ down as well

The mirrors still work though, and cabal will just fall back to those

https://www.bbc.com/news/technology-57399628

"A number of leading media websites are currently not working, including the Guardian, Financial Times, Independent and the New York Times."

Not that the BBC are gloating that they're still up

The BBC.com site was down for about 10-15 minutes.

also https://www.theguardian.com/

What's far worse than half of the internet being down was that Hacker News also had problems. If I waited long enough on a comments page I got an error message. I don't quite understand what happened there. The communication between my system and HN must have been working otherwise I would never have gotten an error message, so it must have been some internal HN problem. But since HN should only need its own internal "database" to generate comment pages, I don't understand why it should be impacted by the Fastly problems.

I could not tell from the fastly status page. What caused the fault? Could anyone point to any past stories which may be of similar nature other than DDos?

Still a bit vague: https://twitter.com/fastly/status/1402221348659814411

Bitwarden is also down (the Web Vault, not the website).

My self hosted bitwarden is fine, as are all my self hosted sites.

I will never understand the meaning of putting CDN behind CDN.

Yo dawg, I heard you like CDNs...

What makes sense in the world is what puts bacon on the table and not what actually make sense.

Seems to affect Target ( https://www.target.com/ ) and Reddit ( https://www.reddit.com/ ) as well.

https://docs.python.org/ https://www.last.fm/

PayPal seems to be working for me at the moment. Rest of the sites are 503s.

Centralising everything™ and the whole internet goes down because of that.

and yet you're able to leave this comment.

Because HN and those who use less or those who use backup services are smart and those who are caught now have to panic and wait.

Probably going to short the hell out of $FSLY.

Over one issue that highlights they have an abundance of top level customers? Interesting strategy when it's already at a low.

One issue that should have been mitigated at least by Fastly; worse if the client has to do it.

They proudly stated this from their own website to their customers:

> "Fastly’s network has built-in redundancies and automatic failover routing to ensure optimal performance and uptime."

If that isn't one huge lie, I don't know what is.

Please don't call it a lie. It means that they knowingly presented something they knew to be false as the truth. So far I have seen no evidence to support that.

It is definitely a lie, but it's the same lie sold by all cloud offerings. Can you name a single cloud/CDN operator without downtimes?

It's normal to have downtimes but they are usually scheduled and quick (think <10 minutes per month for rebooting and/or hardware parts replacement). I'm pretty sure most non-profit hosts like disroot.org or globenet.org have similar or better 9's than all these fancy cloud services.

It can have all these things and still fail, suggesting otherwise would be fairly naive.

if by "everything" you mean one thing, and by "centralize" you mean not centralized, then sure.

How is having a large chunk of the internet using the same CDN provider not "centralizing"? It's not a hard monopoly obviously but still it meets the definition of centralization.

how is private companies choosing to use a common supplier in a competitive market centralization? monopolies are not centralization either. you need to read a better book.

it's centralization because they all use the same provider. Why do you care about incentives here? The result is the same, just like capitalism and free market tend to monopolies in the long run.

How is a market competitive when there's a quasi-monopoly on infrastructure? When public money is used to irrigate the same corporations with huge $$$, while non-profit network operators are left to rot?

Pantheon was also affected: https://status.pantheon.io/incidents/n9sngt0q0mct

For what its worth, I'm having these problems also with cnn.com, reddit and many others, however when I switch away from WiFi to use my cell provider network, they work fine.

Paypal back, off fastly

Why no other sites bypass the CDN and go directly?

If you aren't prepared to do CDN changes on a whim when something like this happens, it's often better to wait for the problem to be resolved instead of making things worse for yourself due to misconfigurations, revealing your origin IPs, etc.

Can always improve the process for the next outage.

For sure, similar to other industries all changes come after big troubles like this. But would be interesting to heard about how them (paypal) deal with that

Also it takes time for DNS changes to propagate(some people hates this word but actually)

You need big infra and Crack teams of ops people, which paypal can't afford not to have.

Is their anything these big sites could do in this situation, or must they choose between running and maintaining all of their own infra or relying on a single CDN?

If you have absolutely vanilla CDN requirements, you can run multiple CDNs and fail-over or load balance between them using DNS.

Quite a few Fastly customers have more than vanilla requirements though, and may have a lot of business logic performed within the CDN itself. That Fastly is "just Varnish" and you can perform powerful traffic manipulation is one of it's main selling points.

I suppose it’s still a bad experience for the user if some % of attempts to connect fail or if some % of scripts/styles/images fail to load. So I think that means dns information about failures needs to somehow be propagated quickly. Not sure how well that works in practice.

Use two CDNs and DNS providers for redundancy. Gets expensive, but at scale, probably doesn't make a huge difference. More complexity for the site operators to manage, however.

Spotify is behaving strangely as well https://www.spotify.com/

Quora and reddit too

All of these work from here in Grenoble, France...

Interesting. Here in the Netherlands they don't.

Not for me

depends where in France, most people i know here are affected as well

yeah grenoble, updated.

Same here in central Poland (Łódź area), no problem with any of linked websites.

edit: My whole Twitter timeline is full of posts saying "Twitter outage? what outage?". Same on Reddit and Twitch chat, feels like for a short time I was invited into some exclusive circle lmao. StackOverflow and other StackExchange sites also work so I can look stuff up for you.

Germany here (n=1), everything works except reddit and ft.com

Not in East Germany :D

What about banana.ch?

Same from Germany, all of these seem up except for reddit and ft. Maybe we got lucky with our edge node...

Not from Paris, France.

That's the problem with these black-box cloud offerings, that you can never know what will work (or not) and from where. You get semi-random, pseudo-localized outages that are not accounted for in all the 9's of availability.

With a standard TCP/UDP session, it mostly just works or doesn't and you can get a proper traceroute to know what's up. With these fancy CDNs, there's a whole new can of worms to deal with and from a client's perspective you have no clue what's happening because it's all taking place in their private network space where we have no "looking glass".

Fuck the cloud, i want real Internet.

"Gets blasted by a DDOS and is no longer on the internet"

https://repo.maven.apache.org/ is down too

https://www.theverge.com/ seems to be down too

https://getbootstrap.com/ as well

Is the fact of looking at those links is like looking at a road accident with insistence instead of just passing by?

https://docs.npmjs.com down as well

https://www.reddit.com as well

Terraform having issues and rubygems down too

That explains the spotty container build failures over the last half hour. Good thing I decided to procrastinate instead of debugging the issue!

also https://www.speedtest.net/

Seems to be every site that runs varnish...

Fastly largely runs on Varnish, it seems: https://www.fastly.com/blog/benefits-using-varnish

>At the core of Fastly is Varnish, an open source web accelerator that’s designed for high-performance content delivery. Varnish is the key to being able to accelerate dynamic content, APIs, and logic at the edge.

I think Fastly is the one having problems (they happen to use varnish but I haven't seen anything which says varnish is the root cause) - so all sites using it are down.

Firebase hosting has been affected as well

also https://www.ft.com/

SSO and github are back online now

nature.com

You would think that the UK GOVERNMENT would have their own private CDN or something...

Why?

twitch also, lots of other minor ish websites

Searchable offline backup of stack anyone?

www.gov.uk & bbc are back

elastic.co down as well

developer.spotify.com

reddit down aswell

It's OK though, because large swathes of this discussion seem to have turned HN into reddit, at least temporarily. Normal service will no doubt resume in due course.

twitch.tv Too.

etsy.com too

> [0] https://www.gov.uk/

Just checked, thank god the NHS vaccine site is still available - vaccines just got rolled out for under 30s today.

Edit: I didn’t mean anything negative here! Just slightly shocked that as the UK is opening up under 30 vaccinations, the US is struggling to find any more willing takers. It’s really probably a sign that there’s fewer anti-vaxxers in the UK more than anything. And that universal healthcare is more efficient at distribution than an inherently for profit system. I don’t know, but I just didn’t realize it was so different in the UK

I think this may be because we've had much higher uptake as far as I know, so getting down the age ranges has been slower (by which I mean, yes, maybe the US has made it available to all adults, but how many (as a proportion) have taken it up)

This is awesome. And also exposes how broken the US is with its anti-vaccination trend

I have seen the argument made that one of the reasons for high vaccine confidence in the UK is as a result of Andrew Wakefield's MMR fraud, which was perhaps debunked more effectively in the UK than the US.

https://www.youtube.com/watch?v=8BIcAZxFfrc

UK is ahead of the US https://ourworldindata.org/covid-vaccinations

For one dose. For full vaccination, the US is (slightly) ahead according to that same site.

I think we can agree it's certainly not "far behind"

This is by design though, the gap between the two doses is higher here.

Fascinating. So those rates are including only ages 30+, which means that once it’s unrestricted the UK should have a very high vaccination rate while ~15-25% of the US will still remain unvaccinated entirely by choice. Wow. So you’re absolutely right, the UK is in reality far far ahead and the US is completely broken as far as public health is concerned because of willing ignorance.

You know which one’s worst? Japan... still reservation based and for 65 and up only!

Both the UK and US are doing well.

https://ig.ft.com/coronavirus-vaccine-tracker for reference.

What's important is important to share vaccines with all nations, and non-nations.

US and UK have very similar vaccination rates despite the US being open to more age ranges. This indicates that a higher percentage of eligible people have gotten the vaccine in the UK, and the US has somewhat hit a wall in terms of vaccinations (though there is the concern that the rates will slow down in the UK also).

I must admit, it has been strange seeing my US peers getting the vaccine months before I can in the UK, but I guess I take comfort knowing that both countries are still doing pretty well!

and Imgix

Click the new tab. Lots of posts about sites being down. All flagged.

Yes, because they're all just repeats of this one.

Reply on news.ycombinator.com