Tunnel Vision: CloudflareD AbuseD in the Wild
guidepointsecurity.com
guidepointsecurity.com
This is a technique commonly referred to as "living off the land" where an attacker makes use of a tool like cloudflared to conduct an action that would otherwise be blocked by security tools. It makes the defenders job so much harder because you now need to differentiate between your devops team being cool and a legitimate threat inside your network by looking at the exact same indicators generated by the two. Looking for things like unsigned applications making outbound network connections are removed from the defenders toolbox.
Yes, cloudflared does the same thing as ngrok. You'll also find that ngrok is blocked in most corporate environments as well for posing an equal risk. As an attacker, you have a good chance of setting off alarms that (should) specifically detect ngrok.
I think the point of this post it to highlight that cloudflare tunnels need to be block by default as well and only allowed when there are specific approved use cases.
If an attacker is capable of installing apps on your server... you've already lost.
0. https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31...
You're right, but cloudflared is a whole lot easier to get running than previous RATs, and attackers don't need to operate their own c&c infrastructure - you can't defend yourself from such attacks by blanket banning Chinese, Russian, Iranian and North Korean IP addresses (which I honestly advise everyone to do if they don't have business in that country), and you can't easily block outbound to Cloudflare either as half the Internet is hiding behind them.
Basically, cloudflared lowers the price and effort for attackers dramatically, while the effort to defend against this threat model has now risen significantly.
If Cloudflare were to actually be willing to do something against being used by scammers, they'd put the ingress IPs for the C&C infrastructure on dedicated IP ranges and publish these in a machine-readable format so every reasonable person that does not use Cloudflare tunnels can ban them.
(Side note: "zero trust" needs to die)
I don't like the idea of making it easier to block certain services, because it goes both ways: it'd also be easier for bad guys to block good guys from using said services.
If the bad guys can modify your firewall config, you already have a problem.
I am not working with malware specifically, but in the past I've used ssh tunnels, random one-off websocket thingy we wrote, wireguard tunnel, frp proxy, and even AWS SSM agent to get access to machines with all incoming connections blocked. They are pretty simple to setup and generally cannot be blocked with whitelist block already.
(and I bet that for malware, they are worse than cloudfared. Based on CF's reputation, they take security reports seriously, so I would not be surprised if they take down malicious tunnels fast. While random VM on low-cost hosters will probably takes days to respond.)
These are known though, you can block them without causing issues.
> and even AWS SSM agent to get access to machines with all incoming connections blocked
SSM is awesome, but the ways it works... I have no idea how. I think though that it uses outbound connections, but unfortunately AWS SGs can't do deny rules.
This is why various compartmentalization techniques exist throughout the stack - principle of least privilege, network segmentation, acls, and so on. If something is compromised, limit the "blast radius" of what can be done with that compromise.
The other part is containment. By analogy alarm systems and guards. If the alarm system goes off and says an intruder is in the building, well by the "winning and losing" framework, the defender has already lost. (Of course by the binary framework, you don't even need an internal alarm system - once the breach occured you've lost so why bother?). Or you could try to contain the problem and send guards to stop the intruder before they do too much damage.
This is where the cloudflared usage described in the article is problematic. It's a tool that is less likely to trip alarm systems, and further can provide a wide range of access while doing so[1]. If it doesn't trip alarms, a small breach leaking a little data can turn into a big breach leaking lots of data. It absolutely should be considered when designing networks, security methods, etc.
I don't know what the solution here is, that probably depends on the individual/org doing the threat assessment, and probably varies by environment within that domain. Should the tool be blanket banned? Probably not it's also a really useful tool for a lot of people in a a lot of situations. Maybe cloudflare can split the tool in to individual binaries for functionality and have the tool called cloudflared just be a wrapper. Maybe they can change it some other way.
[1] this tool is used for a lot of stuff and may already be on the systems in question, so it's not even installing a program -- just execing it. Since it is used for a lot of different things, it may just be on a blanket "allow this app" list. And so on.
That is a deep misunderstanding of how security works.
The defender’s dilemma states that breaches are inevitable because defenders have to be right 100% of the time whereas attackers only have to be right once. This is why you have to defend in as many layers as possible.
A better way to approach it is to assume that at any given time an attacker can execute code on your systems (because they have knowledge and access to unreleased exploits) and work on ways to detect anomalies and behaviors indicative of compromise as well as limiting the blast radius.
The big thing that I found from my research on this tooling was the ease with which an attacker could a) learn things about a target environment, particularly by enabling RDP and accessing a device directly without generating any red flags and b) more importantly, being able to exfiltrated data over SMB that would appear to be local activity.
Sure, flow data would eventually show the outbound transfer rates suggestive of data exfiltration, but without any other correlative alerts, it's likely to get through before defenders have a chance to respond.
(!cf.bot_management.verified_bot) and (cf.bot_management.score lt 10 ) and len(cf.worker.upstream_zone) gt 0 and not cf.worker.upstream_zone in {"<zone>"} and (not ip.geoip.asnum eq <exception as>)
A request coming from someone else's Worker should not "bypass" any security -- it should be treated the same as if they were running their bot on any other cloud provider and making requests across the internet. If you find otherwise, please file it as a security vulnerability: https://hackerone.com/cloudflare
(Actually, it should be slightly harder to run this kind of abuse from Workers, because any request coming from a Worker has the cf-worker header identifying the domain that owns the worker. It looks like your block rule is actually taking advantage of this. Other cloud providers usually cannot inject headers like this so it's harder to trace and block abuse from them.)
(I'm the tech lead for Workers.)
"We definitely care about abuse generated by Workers."
That may be the case internally but it's not as evident through support. I base this on their ability to answer security-related questions, diagnose odd behavior, or mitigate problems. FYI, we are paying for your highest tier of support.
"it should be treated as if they were running their bot on any other cloud provider and making requests across the internet."
Part of this is how Site Analytics and WAF analytics represent worker data (or don't). Even though the worker modifies the contents, the IP is passed through as the end-user IP (at least the way we use them). These analytics systems need to do a better job of identifying anything worker or tunnels related. If this were being done through AWS it would be evident. We can use various mechanisms to mitigate this, but I wanted to clarify why abuse through Workers is different for our implementation.
Also, the process to block non-zone workers was undocumented, and it took a couple of weeks for someone to dig it up and only after several trials and errors. Support wasn't sure it would work, and there was no other documentation (at the time).
We have yet to get an answer about the high volume of tunnel requests from specific goes.
Combine this with the opaque stack you have that causes odd situations. For example, I added a worker that automatically retries 500 errors from the origin. Simple enough. This breaks Zero Trust, though, and causes a redirect loop. There may be documentation somewhere, but there isn't a good breakdown of how Zero Trust fits into a customer-defined worker, and a redirect loop is undoubtedly a poor failure mode.
There is a world of difference between the information you have and may provide as the Technical Lead of Workers, and what I'm getting via support or raw documentation.
I could provide you more information offline if you want.
Edit: Let me just add the other recent thread [1] regarding redirect loops in verification is par for the course. We had the same issue and tried for weeks for progress or resolution. The issue was never resolved, nor could we get information on why the failure mode was so poor, etc. That ambivalence adds up to expecting general ambivalence or inability to make meaningful progress. At some point, you say to yourself, "Why waste my time talking to support and putting in much effort and getting the run-around."
> Even though the worker modifies the contents, the IP is passed through as the end-user IP (at least the way we use them).
That should only be true when your own worker makes a request to your own origin. It should never be true for requests coming from someone else's worker. Again, please file a security report if you found a case where that is happening.
I'm sure it's just a coincidence that they turn a blind eye towards obvious bad actors that just happen to increase the demand for cloudflare's services
(booters being the classic example)
EDIT: Toned down the language.
Thanks
The biggest risk I see is if the target is already using this for legit use cases, since I guess it would be really difficult to discern between the two.
I guess renaming the malware binary to chrome .exe would have the same risk?
If you use this Cloudflare feature for other stuff, you obviously can’t use those as high SNR indicators.
Many tools can be used for this effect, but the fact that the configuration can be easily modified attacker-side through the cloudflare dashboard, and only the token for the tunnel is exposed "client" - side poses a serious challenge for defenders, especially IR teams conducting post-breach forensics.
But it is important to demonstrate the ways in which this seemingly benign tool can be (and has been) used to conduct nefarious activity if not properly detected and defended against.