Absolute lack of faith in cloudflare rn.
This is amateur hour stuff.
It's especially egregious that these are new services that were rolled out without HA.
Tbh. As far as I can see, their data plane worked at the edge.
Cloudflare released a lot of new products and the ones that affected were: streams, new image upload and logpush.
Their control plane was bad though. But since most products worked, that's more redundancy than most products.
The proposed solution is simple:
- GA requires to be in the high availability cluster
- test entire DC outages
Arguable, it's best to think of the edge as a buffering point in addition to processing. Aggregation has to happen somewhere, and that's where shit hit the fan.
Cloudflare's data lives in the edge and is constantly moving.
The only thing not living in the edge ( as was noticed), is stream, logpush and new image resize requests ( existing ones worked fine) from the data plane
You're being loose in your usage of 'data'. No one is talking about cached copies of an upstream, but you probably are.
Read the post mortem a bit more closely. They explicitly state that the control plane(s) source of truth lives in core, and that logs aggregate back to core for analytics and service ingestion. Think through the implications on that one.
It's not just streams, image upload & Logpush.
The data plane ( which I mentioned) had no issues.
It's literally in the title what was affected: "Post Mortem on Cloudflare Control Plane and Analytics Outage"
Eg. The status page mentioned the healthchecks not working, while everything was fine with it. There were just no analytics at that time to confirm that.
Source: I watched it all happen in the cloudflare discord channel.
If you know anyone that is claiming to be affected on the data plane for the services you mentioned, that would be an interesting one.
Note: I remember emails were also more affected though.
Which was still like ~12+ hours, if we check the status page.
>Eg. The status page mentioned the healthchecks not working, while everything was fine with it. There were just no analytics at that time to confirm that.
What good is a status page that's lying to you? Especially since CF manually updates it, anyway?
>Source: I watched it all happen in the cloudflare discord channel.
Wow, as a business customer I definitely like watching some Discord channel for status updates.
This wasn't about status updates going to discord only.
There is literally a discussion section on the discord, named: #general-discussions
Not everything was clear in the discord too ( eg. The healthchecks were discussed there), that's not something you want to copy-paste in the status updates...
Priority for cloudflare seemed to get everything back up. And what they thought was down, was always mentioned in the status updates.
However, I still fail to see your argument regarding Zero Trust and not being impacted. The status page literally mentioned that the service was recovered on Nov 3, so I don't understand what you mean by:
>The data plane ( which I mentioned) had no issues.
There's literally a section with "Data plane impact" on all over the status page, and ZT is definitely in the earlier ones. And this is given the fact that status updates on Nov 2 were very sparse until power was restored.
What I mentioned, was what I've seen passing by in the channel at the time.
I also saw no incoming help requests for zero trust tbh ( did some community help)
[1]: https://blog.cloudflare.com/introducing-cloudflare-stream/ [2]: https://www.cloudflare.com/press-releases/2018/cloudflare-st...