Ask HN: Why is there not more concern about the physical security of Cloudflare?

It's interesting to explore https://where.durableobjects.live/ - a tool that maps where Cloudflare's worker scripts actually run.

Notably, while Cloudflare has CDN edge locations in countries like China and Russia they don't appear to run workers there.

EDIT: I was wrong - I misinterpreted the map. A solid border circle around a location indicates "Worker-only Datacenter" (see the map legend) and there are indeed locations with those solid borders in Russia (including Moscow and Yekaterinburg) and China (Haidong, Lanzhou and more).

I doubt we could get them on the record for this, but I suspect this may be very deliberate. Maybe CDN edge locations can be run completely securely with forwarded encrypted traffic, while workers are at a higher risk of physical attack.

Nice map but are you sure it overlaps with Workers in general? Workers can run without DO.

That tool maps where Durable Objects run, not Workers. Durable Objects run in more limited locations for their "Durableness". For example, writes have to be confirmed by multiple nearby data centers, if a location goes offline they have to be able to go to a near one, etc. So they basically have their own clusters/regions.

Workers run everywhere. On every location shown on that map. Granted, you need a special license and Enterprise feature to use the China ones.

Yeah I got that wrong - see edit to my comment, there are indeed circles with thick borders indicating "Worker-only Datacenter" in China and Russia on that map.

Recently, many European countries have started using Cloudflare’s anti-DDoS protection for government services, which is fine since it can be done securely with end-to-end encryption in a zero-trust model.

However, these countries are now increasingly using Cloudflare Workers without realizing that this creates total uncertainty about where their data and business logic might end up—including potentially adversarial countries like Russia and China.

Data in transit and data at rest can be encrypted. But Clouflare Workers is neither. It's data being processed in-memory where it will be unencrypted.

> ... Cloudflare’s anti-DDoS protection for government services, which is fine since it can be done securely with end-to-end encryption in a zero-trust model.

Unless it's below L7/HTTP, the possibility of doing it securely is very questionable.

Up until very recently it was very trivial to conduct "domain fronting" of sorts but with colocations in hostile locations. So Chinese or Russian servers decrypting your TLS traffic no questions asked, and that was with their premium (DLS) offerings.

I suspect that if you're in a hostile country where CF announces their prefixes locally, it's still doable. Unfortunately that's a bit more difficult to test than it was before.

I'm the tech lead of Cloudflare Workers. I'm not actually the best person to answer the core questions here, but a few notes:

Cloudflare's Data Localization Suite gives you some control over where things run: https://www.cloudflare.com/data-localization/

Durable Objects specifically support jurisdiction restrictions, to keep your data strictly inside EU (for GDPR) or FedRAMP-compliant locations. https://developers.cloudflare.com/durable-objects/reference/...

The current set of locations supporting Durable Objects is mostly a function of where we have the resources available to operate the distributed database which we use as the storage back-end for first-generation DOs. We recently announced a new storage backend for Durable Objects, which is based on SQLite and a lot of in-house tech instead of an off-the-shelf distributed database. This new backend gives us a lot more flexibility in terms of locating data. There's a lot of work to do, but I would expect that we'll have more than just EU and FedRAMP as jurisdictions eventually. https://blog.cloudflare.com/sqlite-in-durable-objects/

Not specific to Workers: China is special. Your site will not be served from China at all (and your Workers will never run there) unless you've explicitly signed up for China network access. (I think the Chinese government requires a special licence for it? But I'm not an expert on this.) https://www.cloudflare.com/application-services/products/chi...

I don't personally know the current situation in Russia, or how physical security is managed in general (it's not what I work on, personally). I've heard some talk of trying to get the team to write a blog post about it, which I too would be interested to read!

> consistently refer to encryption in transit

For cloudflare to refer to encryption in transit is quite something, given that cloudflare is the largest MITM service in the planet. SO much traffic goes through them and they have it all in cleartext due to terminating TLS.

Remember "Encryption removed and added here".

Cloudflare uses AMD TSME to encrypt the RAM:

https://blog.cloudflare.com/securing-memory-at-epyc-scale/

This was posted with their Gen 10 servers in 2020. As of Sept 2024, they're on Gen 12 [https://blog.cloudflare.com/gen-12-servers/].

> Gen 12 Servers are currently deployed and live in multiple Cloudflare data centers worldwide

Judging from this, the fleet isn't uniform. How uniform is the fleet? Is it possible to be running on Gen 9 in some locations?

What's the threat model that ram interception is an issue? I think the upsell is entirely reasonable, you get charged more for weird compliance demands.

What's a threat model where RAM intercept wouldn't be an issue?

Isn't it difficult to actually analyze the RAM content for any sensitive data? There is so much noise in it

I don't think it'd be too hard to find something like private RSA keys in memory, for example.

A bit hard, but not crazy. You have to potentially deal with not knowing the mappings, but even random blocks will have obvious marks like Json fragments with "password=", crypto keys with common PEM/DER headers, high entropy 8/16/32-byte data, etc.

For fun, sometime, try this:

    sudo strings /dev/sda | more # press q to quit

You'll immediately see plain text of various things on your hard drive, complete HTML, etc. Eye opening.

No it isn't?

I recovered data of mine from ram many times

I think this depends on the adversaries you're concerned about. It's unlikely that breaking into a DC and dumping memory is going to be worth it without a very high expected payout. For one, you'd need very high technical expertise.

In any threat model where the attacker may gain physical access to servers, RAM interception is a concern. The original poster does care about physical access.

I'll mention that unencrypted in-memory data is accessible in pure-software ways too (which usually require root) - debugging tools like GDB, VM snapshots, memory forensics tools, /proc/mem, etc. So unencrypted in-memory data is a problem too if root access (including both by infra-provider insiders and external attackers) is part of your threat model.

Physical access opens up many possibilities to access the data, definitely not only the cold boot attack you're probably thinking of

On Azure -- they've been playing catch up this year after repeated congressional inquiries from breaches. It's only in 2024 that Azure has started to build a better device inventory on their infrastructure networks and started doing appropriate employee access control mechanisms.

Here is Satya's May Post, https://blogs.microsoft.com/blog/2024/05/03/prioritizing-sec...

Isn't this basically why modern server CPUs support Secure Boot and memory encryption? If I understand it correctly, it should be possible to set up your machines in such a way that 1) the server can only boot genuine firmware images, and 2) unencrypted data is only available inside the CPU itself. That is going to rule out most attacks which don't involve dragging the server into a high-end laboratory, is it not?

Tangential: how does memory encryption mesh with features like DMA (which I imagine would be key on high-throughput NICs)?

For DMA, DMA endpoint buffers will generally be unencrypted and live in the "host domain," outside of the secure enclave running trusted code. This has been used to construct exploits against AMD's Secure Virtualization (which relies on memory encryption), for example: https://www.usenix.org/system/files/sec19-li-mengyuan_0.pdf .

There are some emerging technologies like SEV-TIO and PCI-IDE which attempt to extend the trust chain down into PCIe devices, allowing trusted devices to form a relationship with trusted code in a secure execution environment: https://www.amd.com/content/dam/amd/en/documents/developer/s...

Anyway, the best protection here is to terminate protocol or application-level encryption (TLS or whatever sits on top of your protocol) inside of the secure environment, so that decryption only occurs through encrypted memory. It's slower, but this way an attacker sitting as a RAM snoop could only see encrypted network traffic hit the DMA buffers, which wouldn't help them much.

This answer covers it well: https://security.stackexchange.com/a/189978

Basically depending on the scenario, DMA region will either be unencrypted or transparently encrypted/decrypted.

They have some docs on the security of their server hardware, which is one of the ways they can assure that their systems haven't been tampered with by local staff.

https://blog.cloudflare.com/anchoring-trust-a-hardware-secur...

Why not use your own datacenter if it’s that important to ensure physical security? If the business can’t justify it, then maybe your use case isn’t that sensitive or important. Or maybe the business isn’t viable.

Because edge serving has numerous advantages the OP would presumably like to take advantage of. If you have customers all over the world, "using your own datacenter" either means operating tens of datacenters or giving some of your customers high latency.

Then share the nonsensitive assets via a CDN, and host a separate sensitive endpoint (with higher latency) in a more controlled environment.

I think anything which can be delivered to the web browser from CloudFlare could be part of the kill chain for a eager attacker.

If that's your threat model, then you either need to build your own CDN (from the data center up) or find a vendor willing to sell you the features you need. It won't be cheap.

Curious how they protect private keys for transit encryption. I'd imagine all their edge locations need to be able to decrypt traffic and therefor need a way to fetch corresponding private keys for everything they're proxying.

If physical theft is a concern, how do they prevent someone from hijacking the key distribution process?

I don't use Cloudflare and I don't have strong opinions for or against them. These questions are the same I'd ask if you were discussing any other company:

> which operates out of questionable ISP and IXP colocation facilities in various jurisdictions with dubious standards.

Why do you say that? Do you have signals that their colo facilities are less secure than they should be, and/or that Cloudflare hasn't gotten those facilities to beef up their security as part of their contract? Again, not saying this to defend Cloudflare. I just hadn't heard this before.

> Moreover, when we raise these concerns, they attempt to upsell us on their Enterprise EU/FedRAMP offerings.

That's going to be the case with almost all providers in the space. If you're asking for special treatment, you're going to have to pay for it. I don't mean that to insult you. At a past job, for various reasons we had strict compliance obligations that our data could not be accessed outside of the US. Some of our vendors used offshore tech support who'd have access to our data, and a couple times we faced a decision: pay that vendor $$$ to special-case our support setup to meet our requirements, or choose another vendor.

> Cloudflare has also deliberately restricted our ability to block non-Enterprise Workers, KV, and R2 from specific regions, leaving us with limited control over where our data is processed.

Same. Those fine-grained controls are often going to be an enterprise feature.

Again, I'm not saying this to defend Cloudflare in particular. They have their own paid spokespeople. I'm not one. Nothing you've said sounds particularly egregious though. If your data is sensitive enough that you're legitimately worried about someone sneaking in undetected and intercepting RAM, prepare to pay the enterprise tax with all your cloud vendors.

ultimately you trust the company and the jurisdiction in which they operate...if they give you the wrong answers then you should adjust your level of trust accordingly...thankfully there are other platforms and this is one concern that can certainly be better marketed

While this is mostly true, the "trust the jurisdiction" is hard to apply here. They operate their PoPs in over 120 countries, so it's close to "do you trust all countries?" - it's a completely different model than trusting an AWS region for example.

Isn't that the same with Lamda@edge, so AWS also operates lots of PoPs all over the world that run code for you (if you want to)?

Yup, you're right with Lambda@edge, I forgot about that one.

There are, roughly, four kinds of physical security one can have in a datacenter:

1) Nothing

2) Visitor logs

3) Locks and alarms on your racks, and/or (if you have enough) the rooms they are in. Remote monitoring is pretty common.

4) First party human security professionals

I don't have any special knowledge of Cloudflare's set-up, but 2 and 3 are by far the most common. Lacking #1 means your hardware just gets stolen. #4 is too expensive. #2 and #3 are where most people are, so probably something around there?

Sounds like you want an elevated level of security that cloudflare provides but you don’t want to purchase?

Cloudflare have some really interesting tech, like https://blog.cloudflare.com/announcing-keyless-ssl-all-the-b..., that I really wish we had an open implementation of so I could do similar myself.

> which operates out of questionable ISP and IXP colocation facilities in various jurisdictions with dubious standards.

what does this mean?

What about other providers like Fastly and Akamai? I've not looked into this but maybe they do a better job?

> Moreover, when we raise these concerns, they attempt to upsell us on their Enterprise EU/FedRAMP offerings

so you want the cheap plan with the features from the premium plan?

never heard of a story where physical security at any cloud provider has been a problem. are you worried about governments, or employees, or someone breaking in?

Just because there hasnt been a story doesnt mean its not important or critical

When I worked at AWS they were insanely hardcore about mandating physical access controls, that is all I’ll say, even to the point of ridiculousness. For all the things AWS does poorly security is not one of them.

If I were to guess CF is locating their PoPs at cheap peering points and the reason they are evading the question is because other customers in the facility have physical access to their equipment, which is both an expensive problem to solve and something that is not even remotely allowed at a real cloud provider.

This seems unlikely to me, at least in the US.

Even your cheapest of colo's offer locked cage areas. For someone on Cloudflare's scale, the cost is trivial.

I've been inside some really "low rent" colo's and even they would provide an escort to unlock your cabinet.

Obviously standards/expectations will vary from DC to DC. I'd wager the situation might be different in some of the smaller countries CF operates in around the world though.

cough https://www.datacenterdynamics.com/en/analysis/ovhcloud-fire... (physical security isn't just access control)

That's definitely a thing. Additionally, humans are surprisingly friendly in all the wrong ways when it comes to physical security (tailgating, "forgotten ID/credentials", etc.).

A compromised human is immensely more feasible than a physical break in, but almost all posts above fixate on the latter

I've visited data centres (with various impressing sounding accreditations) where the doors have been wedged open because the employees found the security annoying

I've also had DC employees, without authorisation: reboot my machines, give themselves access rights and then tamper with my systems

admittedly the latter was a long time ago, before any of this stuff was considered critical infrastructure

these days they'd probably end up in prison

you must not deal with compliance (and bless you for it)

Some of my own thoughts about this. I don't know, I don't work at CloudFlare. Anyway:

> infrastructure, which operates out of questionable ISP and IXP colocation facilities in various jurisdictions with dubious standards

What are these questionable facilities? When CloudFlare has installed those servers and software, and they have also made software that manages those servers, what is the problem?

CloudFlare has written some articles about some of those very many security protections they have. But that is a very lot of technical detail to explain, so if that is so important, their Enterprise/FedRAMP offerings fund CloudFlare to make possible to explain that amount of detail. But question is, do you really need that amount of detail? How much you have expertise to build same amount of security protections? Isn't it better to use CloudFlare Workers security features to concentrate on building your app? Alternative is to get your own bare metal servers, and manage them yourself.

With CloudFlare Workers, they have security features to keep code and data of each customer separate from each other.

The issue here is there's sort of a missing middle. The main cloud provides offer physical security without you needing fedRAMP etc.,.