Cloudflare API, dashboard, tunnels down
cloudflarestatus.com
cloudflarestatus.com
Cloudflare at least accurately reflects the incident on their status page.
Often dashboards will show everything green despite the service being down.
In these moments communication is critical and Cloudflare does it well especially as we all know when it comes to the post mortem. They are very transparent about these issues and incidents.
It's also nuts that this downtime according to creation of the post -> resolved in comments, was only 27 minutes...
I've literally been on standby to get an answer what happened with a huge cloud provider for 3 weeks now and today they got back with static assets that weren't reachable ( ico's ???)
They don't even know what happened since it's not logged in their logging, since it's using middleware and because of the sudden weekend downtime, they middleware wasn't triggered...
After a detailed description that we found that it was related to a app service plan, they came back with 404's on static assets...
Only now it got escalated..
Fuck the cloud, outside of cloudflare.
The only time we had a downtime with cloudflare, they gave a detailed description and it was related to our cloud provider and not Cloudflare . Then their was a infra update and it also didn't get logged... Because of that we thought it was the wrong party...
18 hours down, redeployment fixed it, nothing logged on their end. Luckily one service wasn't redeployed ( barely used), so we can prove there was an issue.
Ho boy...
I'm wondering how many more weeks it will take...
I was affected by the tunnel outage. It was down for closer to 3 hours, not 27 minutes. (Their status page does accurately reflect this time)
I checked the author post. Normally it's pretty quick to post here as soon as something is "noticed".
Hah. Cloudflare's rise to fame was them proudly announcing how their servers were running but the users misconfigured their site, by "defacing" the users' non-resonse.
Cloudflare detects downtime of the underlying cloud/host.
You'll literally see a cloudflare error page and think it would be related to them, while it's actually the underlying provider.
Looks like there were 2 outages[0], one in June and one in July of 2021.
[0] https://www.reuters.com/technology/websites-airlines-banks-t...
Your regular, run-of-the-mill outages aren't reported in Reuters... but are likely much more common than you'd believe.
There is an option to use ngrok instead, but I have never seen such a setup for "local" development.
As much as I like cloudflare and easy to use stuff, lets be a little bit mindful of the blast radius of highly centralized services which are even more complex to operate due to the scale of the deployment and problem scope.
luckily this time its just administrative interfaces and supplemental services that assist in development and that hopefully aren't used in production.
People will defend cloudflare here with the old saying “Operating at scale is hard” but simultaneously holding the opinion that operating services themselves is equally hard.
I guess the argument is that outsourcing is easier, but then you are very much out of control.
“the internet” in its practical entirety going down becomes more possible every passing day.
Grab lunch, come back, and it'll have fixed itself. Nobody blames you because it's just another provider outage -- whether it's Cloudflare or AWS or Microsoft or Gmail doesn't really matter, it's a convenient outsourcing of not just network admin but also blame.
The alternative is, what, doing it all in-house with a bunch of people on pager duty? Few companies can afford that. Diversifying across multiple cloud providers? Sure, if multi-nines uptime is really critical for your business.
For most small-med businesses, it's more valuable to be able to rapidly build on top of existing, professionally-maintained infrastructure in order to deliver some unique business value on top of it, than to recreate that basic network infra themselves. If you're not a FAANG, going down for a few hours a year isn't really a big deal. Especially when you can just blame it on an overall "internet outage at some big provider" rather than some internal fuck-up.
(Edit: Not to be snarky, but really, so what if that happens? Same thing happens if there's a power outage in your neighborhood. Is it a big deal?)
Could not find data how this is distributed..
Sorry man, our Stripe terminals aren't working, we can't even accept cash... Um, come back later.
Edit: Just to clarify, obviously this can't be both common and rare... what I meant is that different neighborhoods will frequently go down, but there's usually some neighborhood within a 10-min drive that still has power, or some food truck has a cellular connection off their iPad PoS that still works, or you happen to have cash in your wallet or whatever or know the guy and they'll just put it on your tab for next time, whatever. I live in a semi-rural community in Oregon and we have a lot of small businesses, a lot of wildfires, and not a whole lot of infrastructure. We just kinda make do when things happen.
A couple weeks ago, half the town lost power for the whole day, and the other half of the town just got some increased business and traffic. Life was back to normal the next day.
However the internet doesn't require people to "operate at scale". That's the beauty. It's a mash of different (relatively) small autonomous systems, with many hosts. If any host goes down, or any network goes down, the system ignores it, or reroutes it.
The internet itself does not, yes, but the internet as a system, filled to the brim with bad actors ranging from 14 year old skiddies with Metasploit over hardened Russian ransomware groups to virtually all nations' secret services and militaries does.
As soon as you start a server with a globally routed IP address, it will get hammered from all kinds of scanners. Anything on port 80/443 will get bombarded with Wordpress, Drupal and log4j attempts, anything on port 22 will get hounded by people attempting credential stuffing, and anything on port 25/587 by spammers and anyone trying to exploit Exchange/Exim vulnerabilities. It's a nightmare to keep up with that crap alone, and you're one forgotten update away from getting your server 0wned by someone using Shodan and whatnot.
And then, once your site, whatever you operate - I've seen as an admin DDoS extortion attacks happen against small sites for dog walker services, pizza parlors or random bloggers as well as large car dealerships - finds itself some opponent for whatever reason, it gets nasty. Be it for political reasons (people don't like what the blogger writes), for the lulz (people don't like a Twitch streamer so they DDoS their sponsors), or for financial extortion, doesn't matter in a time where you can order such attacks for less than the price of your average McDonalds visit. There is no, utterly no, alternative to the giants (Cloudflare, AWS CloudFront, Akamai).
It's sad that it has come to that point, but unless we get the most obnoxious sources of such attacks - enemy nation states (China, Russia) and compromised domestic IoT devices/servers - off of the Internet, there is really no viable alternative.
Yet my servers are fine without such services.
If someone chooses to host their service relying on a single unreliable point of failure that's on them. Plenty more hosts on the web.
dig SOA news.ycombinator.com
; <<>> DiG 9.18.16-1-Debian <<>> SOA news.ycombinator.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10818 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;news.ycombinator.com. IN SOA
;; AUTHORITY SECTION: ycombinator.com. 900 IN SOA ns-225.awsdns-28.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
;; Query time: 39 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP) ;; WHEN: Wed Oct 25 10:52
It's also one of the least reliable and performant public forums I've ever used. It goes down or slows down several times a week for me. Wish they'd just move it to the cloud instead of occasionally upgrading the bare metal...
The centralization around AWS and GCP and Azure is more concerning to me than the centralization around Cloudflare. It’s not obvious to me why lots of people using Cloudflare is any worse than lots of people using AWS.
If we go even deeper, the entire internet runs on a limited number of undersea cables connecting continents. That in particular has always seemed like the most vulnerable aspect of the internet, considering a targeted attack on < 100 locations could meaningfully cut off internet access for millions (billions?) of people for an extended period of time.
How does this work when most edge providers manage your DNS to point at the available servers they have?
Also, if you "HAVE" to be online, then you're the type of target that's at big risk of DDOS which complicates this even further.
What if its more than a few hours because theres a natural monopoly and a dearth of capable people to run systems independently - thus no strong external incentive for rapid response outside of punitive SLAs?
How much soft power do you think such an entity would have on the world stage.
People get some off-screen time? Life goes on.
> What if its more than a few hours because theres a natural monopoly?[...snip...] How much soft power do you think such an entity would have on the world stage.
They're just commodity infra providers. The Facebook and Twitters of the world have way more power and influence. This isn't even a hypothetical. The big cloud providers have been around for what, a decade or so now? How many times has the world collapsed with every Cloudflare or AWS downtime? It's just not as big a deal as you paint it to be. These providers going down is like the digital equivalent of a highway crash -- a small inconvenience for a few hours, then forgotten about in a day or two.
Okay, so obviously a lot of "important" stuff happens via the Internet. You could come up with some financial impact if the Internet went down for some time.
Harder to measure: A lot of people would be inconvenienced or suffer. Some number of people would die indirectly, e.g. due to emergencies/lack of communication.
People are inconvenienced and suffer and die all of the time. It's part of life.
I'm not advocating for more death and suffering, but I am saying that, as another comment said, "life goes on".
---
Put another way, if the Internet disappeared for a week, month, year, what would happen? It'd suck and we'd adjust, but we (humanity as a whole) would be fine.
Those are much more volatile (compute heavy and not enough compute to meet peaks), harder to setup/maintain and benefit greatly from sharing infrastructure for hardware utilization. Everything is moving in that direction.
Really the best innovation in the last 10 years has been the penchant for putting things on development budgets instead which loosened the purse strings considerably.
I wish I had sufficient data to back this up but for context; I was part of a small infrastructure team running a little over 1% of all web traffic at one point in a pair of colos (with DDoS protection) for less than £5MM on infra costs.
Now I can easily be in the 6 digits with just a CRUD app
It's readily evitable, just choose not to write the generic, repetitive thing you already know is generic and repetitive.
I actually disagree; the centralization is a symptom of severe, systemic, unfixable flaws in the design of the internet itself. At least we didn't let AOL define internet.
I tend to think centralization is more about people's tendencies than it is internet design. As I see it, centralization on the internet happens as the emergent behavior of all of the elements in the system and the forces that exist. In other words, it's the natural result of incentives.
People use centralized services because
- prices (economies of scale)
- network effects ("I can't certain people without WhatsApp")
- social pressures ("People look at me weird when I say I don't have an Instagram")
- visibility ("There are other CRMs than SalesForce?")
- making choices easy ("I don't want to choose between all these options. I'll pick the most familiar")
- safety ("Nobody got fired for choosing IBM")
- convenience ("It's a pain to self-host")
- security ("Google probably protects data better than I can")
- etc
Level 3 Communications / Lumen; AT&T; Verizon Enterprise.
With those three, you've knocked 2.35 million km, or 1.46 million miles, of fiber optic offline. That's over half of the US backbone in those three companies. Comcast "only" has 0.15 million miles for the sake of comparison. The internet would crash and burn immediately to speeds not seen since the dial-up era. There's nothing we can do about that. Cloudflare and AWS centralizing everything is actually, statistically, less risky than backbone attacks. If you succeeded in knocking down all of the big seven at once; you've got about 80-90% of the backbone offline plus all of the online cloud providers with direct connections to them; at which point the internet would cease to function in any practical sense.
2. Port scanning attacks
3. Geolocation attacks (you shouldn't run a server from your home, when your IP Address could be used to find you; be it from a hack or an abuse of the legal process)
4. The US internet relying on a backbone mostly operated by 4 companies (meaning a direct connection is a massive perk)
For as much as we moan about Amazon and Cloudflare centralizing things; believe me, just taking out Level 3 / Lumen would be almost an act of war and would affect every ISP and cloud provider.
If you took out Lumen, Verizon Enterprise Solutions, and AT&T, just those 3 companies; that's 2.35 million kilometers of fiber going dark. For reference, Comcast has "only" 150,000 km.
"I can't be the only person who literally doesn't give a F?"