Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees | Hacker News Reader

Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees

blog.cloudflare.com

230 pointsNicoJuicy164 comments

It's a seemingly simple and obvious way to lazily migrate your data, but if using Sippy means one less thing for the application code to worry about, and (I assume) is a free add-on, then it provides a ton of value.

I have to admit that Cloudflare has been killing it recently with DevX / OpsX. If I wasn't against that company's role in modern internet (as a user of Tor, their firewall is annoying to no end), I would have tried them out already.

This, CF is the only service that I find amazing, and that I do not use for anything. Compared to AWS, I think I prefer the "we're your unopinionated infra provider, if you want a WAF we have that too", vs the CF "block the world, especially the developing world, give zero craps about it". I fundamentally would be unhappy as their customer even if their service were stellar because I do not want my apps to be associated with this arrogant stance of "hey, your app blocks users from Nigeria".

The geographical blocks are not enforced by Cloudflare as a blanket ban, but are chosen by each account owner (it's a setting you configure). I've worked with a few companies that saw this as a very valuable service (like small domestic companies blocking international traffic, especially from Russia and China, because we had no presence there anyway and that cut down bot traffic by like 95%).

Likewise, TOR access is similarly configurable. Companies choose to block it because more often than not it IS bot traffic, and the few potential real customers who use TOR are deemed not worth the headaches of the rest of the network.

Cloudflare's WAF is really pretty granular, with a lot of toggles and overrides: https://developers.cloudflare.com/waf/managed-rules/

Anecdote: For small businesses with limited web resources, these are just everyday tradeoffs they have to make in order to keep hosting and security fees reasonable. At the place I worked at, previously we were spending tens of thousands a year on hosting and thousands more for a competing WAF that cost like 10x more and didn't work very well. Cloudflare let us move to a much lower hosting plan and cost like $240/yr and drastically reduced bot traffic. Not a single customer complained over the next year or two. It was a huge improvement in both performance and costs.

I don’t understand the parent viewpoint.

“I don’t like Cloudflare because they’re trying to centralize the Internet and block me”

It’s not as though Cloudflare goes out and randomly inserts themselves in Internet traffic and has some blanket policy of ruining TOR or blocking you.

Cloudflare has customers (site hosts) that have choice in the marketplace and choose them. The customer configures whether their services use Cloudflare or not. The customer configures TOR access, CAPTCHA level, geoblocks, and any other number of hundreds of parameters.

Then people get mad at Cloudflare when a site/host selects Cloudflare and configures it in a way that blocks them?

Cloudflare is selling what people want to buy and providing the service in the way they configure it. If you have a problem with that take it up with the site/host/CF customer, I truly don’t understand how/why they can or should be blamed for their success.

I think what you’ll find is that many Cloudflare customers are practical and pragmatic. Want access to our site over Tor? Sorry but Tor is 99.999% shady/malicious traffic we don’t care about. The risk vs reward isn’t there so blocked. Maybe if a customer says something we’ll enable it but that has never and will never happen so blocked.

Our PCI scans and auditing systems are showing weird traffic from Asia even though we have no customers or business there? Blocked.

Repeat this for any other number of factors and you can start to understand why Cloudflare has double the market share of their nearest competitor (AWS Cloudfront).

They offer a product suite site owners and hosts love. The collateral damage from a tiny fringe of legitimate users who get stuck in the CAPTCHAs, use tor, etc just don’t matter to the site hosts. If they did they would configure Cloudflare differently or leave them altogether.

Yeah.

I think cloudflare is even the only one that supports the "onion routing" to improve the situation for real Tor users.

Exactly.

It’s so annoying when I travel and I can’t access login to my Utility provider’s website to pay my bill.

I understand they don’t want get hacked but believe it or not, customers travel.

I agree, it is annoying. But it's also your utility's choice. Complain to them about it or choose a different provider if you can?

Security and convenience are always tradeoffs. I think 2FA is annoying as heck too (much prefer passkeys these days), or ridiculous password requirements, email passwordless login, etc., but those are all choices some admin or manager made on behalf of their business.

I found out a few months ago that my router has a built-in VPN that I just had to turn on. Now it doesn't matter where in the world I am, it can always look like I'm not only in the US, I'm actually on my home network. That might be worth considering if you're traveling a lot: it's about as benign-looking to providers as is possible.

I'd be afraid of doing that, knowing how insecure most home routers are. Seems like a zero day (or really even a "zero year", considering how behind updates can be) away from having your home router become part of a botnet...

It's just OpenVPN, and the version was up to date when I enabled it.

There’s always auto pay or give them a call?

That’s like saying it’s annoying that I can’t get my favorite chips at the grocery store because they don’t ship them to my country, so I’m upset at the grocery store for not having them.

I am convinced that while harder, there are more intelligent ways to block these DDoS attacks other than blocking entire geographies. I often travel between Africa and US, and there are things like buying furniture (home depot blanket blocks non US customers, but they would simply allow shipping only to US addresses), buying cars (there are large car sites that don't allow browsing from outside US, even if you've already have an account or bought from them in the past), etc.

I feel like geoblocking is the easy way out, because if developing countries suddenly started waving their cards en masse, these merchants would find a way to let them in.

Speaking of card waving, it likely only appears that developing countries are not a large customer base because to merchants, they look like US customers.

Most African countries don't have access to Visa/Master cards, so often they'll have a US account where they can transfer some of their money. Others might earn in the US (like remote workers), and spend considerably in the US.

Then, because most merchants don't ship outside the US, these customers would use shipping forwarders, like myus.com.

So when making the decision to "block Nigeria because we don't really have any customers there", they're likely not considerig this potentially large customer base they're alienating.

Even worse, these are usually customers that do not have access to credit, only debit, so for example, when buying large ticket items (like a car), they tend to pay for it all upfront, so likely great customers.

Then there are the business customers, the ones what want to buy containers full of merchandise. Those too get blocked.

As a business owner who geoblocks:

It's not usually a benefit to a business if a customer pays upfront.

Whether my customer pays by debit or credit, I get all of that money upfront before I let the transaction proceed.

Some businesses, like car dealers, actually make more money if the customer buys using debt, because they get incentivized by the loan company.

And lastly, the sheer scale of the US economy means that it's really not worth the hassle. All of Africa would be equal to one of the larger states (Wikipedia says $3T, Texas is 2.1T and Cali is 3.5T).

So it's vastly simpler, cheaper, and easier to deal with say 30m Texans or 40m Californians than literally 1.3 billion people in Africa or India, and you get roughly the same total addressable market and a fraction of the bots & scams.

Hence why many sites simply block non-North American traffic.

I wish we lived in a world that was more fair and open, but a couple of bad actors can really ruin things for everyone.

When I say pay "upfront", I don't mean that the upfront cash is better for business, but that usually, the credit industry is very good at letting people buy things they can't afford. Some one who pays upfront likely can afford and might have higher lifetime value. Someone to advertise to, upsell, or whatever.

Secondly, I also get it, there's only so many things a business can worry about, and supporting geographies with historically high fraud rates is not high on the list, this is why my gripe here is with CF that does not make it easier to improve this even though they know they control such a huge chunk of the web.

> Some one who pays upfront likely can afford and might have higher lifetime value. Someone to advertise to, upsell, or whatever.

100%. Richer people tend to be better customers. But that's another strike in favor of geoblocking non-US visitors.

When I was a kid growing up in Africa, I dreamt of a world where everything was accessible and purchasable and learnable everywhere, all the time, to everyone. Hopefully the internet turns out to be an equalizing factor and we get there someday.

Right now it's not really fair to expect business owners - most of whom are in non-tech businesses that require 100% focus - to keep up with the tidal wave of scams, hackers, and regulators originating from outside their sphere of concern.

> I am convinced that while harder, there are more intelligent ways to block these DDoS attacks other than blocking entire geographies.

Sure. Even by default, Cloudflare won't block entire countries. That's a CHOICE some businesses make if the default blocks aren't enough, and they don't have the time or resources to configure more nuanced WAF rules. (OWASP isn't exactly straightforward). Edit: For example, at that job I was talking about, we had different rulesets for different regions... China and Russia were completely banned, Africa was put behind stricter JS security checks and CAPTCHAs but allowed in, Europe had a medium security level (we did occasionally sell there, but very rarely), while the US had entirely custom WAF rules. It just depends on who we wanted to sell to or not.

It goes the other way around, too, you know. I've seen European and Asian sites that geoblock US customers. It's not out of malice, they just don't want to deal with the edge cases. Even if a foreign customer can access your website and buy stuff, dealing with international customs, consumer laws, credit card fraud, wire transfers, etc. can be a pain that's not worth it for smaller merchants. And if the foreign buyer is using a reshipper anyway, well, the reshipper can just buy the whole thing for them and deal with payments, etc. as an intermediary, like how Tenso/BuyFromJapan/JapanRabbit work.

Big companies have proper international presences, but for small local businesses, the amount of effort it takes to support international buyers just isn't worth the profit they typically bring in. Even on eBay, with its built-in international payment and shipping rules, sellers often won't want to bother.

This isn't really a matter of security rules, really, but just business cost/benefit decisions.

Besides, it helps businesses in each country stay local! Do you really want Amazon taking over everywhere...?

I'm aware of this, CF can be very granular, but businesses do not on average have the know-how or bandwidth to properly setup their rules to not come off like a...holes. So the effect is that most businesses behind CF come off like a...holes. My point is CF does not seem very interested in coming up with a better solution, like maybe a list of CF managed WAF profiles that work well and don't make both the businesses and CF seem like they do not care. Those profiles could be paid.

And yes! it's better to buy local, and Africa can't blame the US because our economy isn't there, and we aren't building all the things we should be building. But that is an entirely different discussion isn't it?

They do offer different profiles! By default the security is pretty sane, and offer many easy to choose default sets. Businesses actually have to go out of their way to make a custom country block via a custom page rule. So when you see a site block you, that's because that specific business chose to customize their rules specifically to block you. That's not Cloudflare's fault.

I work for a not so small company with a large international user base and wish I could have the option to geoblock sometimes. While you're not wrong about there being more intelligent ways to block traffic it's substantially more time consuming to apply and get it right so that you allow legit traffic and actually block what you need to.

We also aren't just talking about blocking DDoS and other common vulnerability scanning. Depending on your business there are other potentially costly fraud and abuse scenarios that you are blocking just by blocking other countries outright. Until there are tools to block all this that are as easy to apply as a geoblock, this will probably remain the unfortunate state of things. A lot of businesses just don't have the time or resources to manage all of this without applying geoblocks.

is shipping forwarding common?

i would think that it adds on a huge cost?

It does, but depending on your shipping patterns and volume, it can be worth it. For example, some provide storage so you can hold your merchandise until you have a container full, and then ship all at once by ocean freight. Other times, they have deals with shipping companies like DHL that might make it cheaper than dealing with DHL rates by yourself.

Absolutely this. Cloudflare simply provide website owners with the tools. I've used them on a number of sites. For example, someone provides a local service and is sick of hacking attempts from the same 5 countries. Block 'em. Or they sell products that can't be delivered internationally, so why expose yourself to pointless inquiries or hacking attempts?

Leaded gasoline was also popular and provided a valuable service to car ownwrs (more power, less engine wear). And yet we banned it.

It is not just about customers, you have to thibk about ecosystem as a whole.

What do you mean? You want to force every business to sell internationally?

The problem is; as a tiny SaaS owner, we still get massive DDOS attacks a few times per year. With other services I would pay a fortune; with CF it’s free. Downtime has people walk away and CF saved my bacon many times. If there are friendlier alternatives that work as effective then I have not found. Mind you, I don’t mind paying a few 100$/mo for such a service but on aws etc one attack burns that for more than a year. The one over Christmas last year would’ve been enough to just close the doors if I had mitigated that via aws.

> as a tiny SaaS owner, we still get massive DDOS attacks a few times per year

Wow, why? Extortion? Competition? Collateral damage?

Not sure about the parent commender, but a few of the small SaaS companies I've worked with were regularly targeted by carding and credential stuffing attacks. I don't know whether the attackers ever realised much direct benefit from targeting them, I always thought they were just soft targets for validating credit card or credential lists.

But if you don't have any DDoS protection set up, either of these attacks will essentially be L7 DDoS attacks when deployed at scale.

Those are completely different things though. If someone is trying to validate credentials/cards using your service, they're not trying to DoS it. Those types are usually using only a few IPs and are easy to detect/block (or even tarpit and help others) at app/proxy level. (Multiple failures for different usernames - block for some time)

You need DDoS protection when someone does not want your service to stay up.

Well no, cred stuffing often results in termination or suspension of service by the merchant accounts used by the victim. This denies them the ability to sell thier goods and services - generally this denies end users the ability to ise the victim's service just as effectively as a volumetric attack.

Further, cred stuffing is often automated by a botnet.

The two things are distinct, but have similar means and end results - they aren't completely different.

And a common thread is that they originate traffic from sleazy IPs: having a service like Cloudflare block those will protect against ton of internet background noise with very little downside for most businesses, so it’s unsurprising that so many sites do so.

I was surprised to find 1000s of google cloud and azure ips last attack.

There's a LOT of compromised instances on GCP and AWS. Attackers crawl for leaked credentials and access keys which they use to spawn more instances, run cryptominers on them, crawl for more keys, etc. They set these up in ways that are difficult for account owners to notice right away, such as by launching the instances in regions the account owner doesn't normally use or setting up large numbers of IAM accounts and roles to retain access if the new instances get purged. (Ask me how I know.)

I suspect a lot of the malicious traffic coming out of Africa is not direct attacks from cybercriminals but residential machines that have also been compromised to send malicious traffic. The only difference between that and cloud providers is that you cannot afford to block all of Amazon or Google. They have a level of economic privilege that the entire continent of Africa lacks.

The events I witnessed were all from rather large botnets. If it was just a few IPs then those particular companies might not have even noticed. Attacks like that aren’t always easy to solve with simple tools like says fail2ban.

Regardless of whether or not it’s the intention of the attackers to disrupt the service, that’s the effect it can have, and it’s something DDoS protections service will usually mitigate, especially the services that incorporate WAF functionality (which I think is pretty much all of them?…).

For small SaaS companies they are definitely not easy to detect and the card fees alone until you detect might make you not make payroll that month or go out of business.

They seem to be crawling the site but with 1000s-100000s of bots and ips. Happens several times per year.

When bot fight is on, we don’t notice anything.

With AWS, WAF unfortunately still incurs both the WAF request processing and the service’s own request processing pricing (such as Cloudfront or ALB), which means you still end up absorbing a large bill in case of an attack.

That'd be one distinct advantage of using Cloudflare over AWS, regardless of how opinionated it may appear; and you get to fine tune some of the settings if you're a paying customer.

I don't think they're especially blocking the developing world. They're blocking attacks. If you have a better way to do it, way so. Otherwise, their approach is the best way to do what they do, to your knowledge.

Anthropomorphizing technical services is probably not going to lead to good conclusions. Better suggestions will.

It depends on your site. If you are an ecommerce site that only ships to the US, issuing a challenge to people coming from Nigeria isn't out of line they are seeing lots of attacks from similar locations. Probably isn't a good thing for a local Nigerian news site though.

> and that I do not use for anything

Let's try. I suppose you know 1.1.1.1 but:

- Their WebAnalytics

- Flexible SSL ( instead of letsencrypt)

- Their free Hugo setup ( for your blog) -> Cloudflare Pages

- Buy DNS domains at cost

- 500 Cloudflare worker scripts for 5€ / month. Or 100 Cloudflare worker scripts for free

- Cloudflare tunnel - instead of ngrok or others. You can link it to your subdomain, other options have a paid option if you want to link a subdomain.

Is blocking users from Nigeria an official policy? Can anyone link me to either their official policy or to studies of which countries are blocked and how complete the block is?

Blocking suspicious traffic is a setting you can easily toggle off in Cloudflare.

Cloudflare doesn’t “block” anything universally everything is completely configurable (other than obvious exploits like mass-blocking the recent http/2 attacks)

Gotcha, thank you.

CF blocks "bad rep" IP blocks. I think nigeria part was a joke

Yes, I'm not Nigerian but I basically mean developing nations, including mine and others.

CF is very customizable, if you are a paid customer.

Cloudflare doesn't block TOR specifically.

It's due to their users and associated behavior that tor Exit nodes have an elevated bad reputation.

> https://developers.cloudflare.com/support/firewall/learn-mor...

> Due to the behavior of some individuals using the Tor network (spammers, distributors of malware, attackers, etc.), the IP addresses of Tor exit nodes may earn a bad reputation, elevating their Cloudflare threat score.

Customers of cloudflare have an option to improve experience for Tor users

> Beyond applying firewall filters to Tor traffic, Cloudflare users can improve the Tor user experience by enabling Onion Routing. Onion Routing allows Cloudflare to serve your website’s content directly through the Tor network, without requiring exit nodes.

Email the sites where you have issues and ask them to enable Tor routing.

Whatever happened to Privacy Pass? I seem to recall it being touted as the solution for maintaining privacy but keeping bot traffic out, but I haven't heard anything about it in probably five years or so.

I'm also not understanding how enabling Tor routing prevents bot traffic from hitting the site. The traffic gets served over a .onion instead, cool. But how does that prevent the bots?

What's the alternative if most of tor traffic is password attempts and bad actors how do you protect yourself from tors bad actors without effecting all of tors users. I work at a company that runs a large website top 1000 websites in the world, and we don't even have to block tor exit nodes since they trigger our bot and snap blocking rules on our firewall, how do we let valid for users through without letting all the malicious actors?

My take on this: if there is some DDoS taking place from same IP I am connecting from, that sucks for me but I'm willing to tolerate it (good old fail2ban). But having such a firewall all the time, even when you are getting less than 1 request per second from ToR? That's an overkill

If I occasionally get a DDoS from Tor, I'll probably just block Tor all the time, even if my current traffic loads from Tor are low. It's simply not worth the hassle of waiting until my servers start getting spammed, it's better to just keep the door shut all the time.

How would you deal with an attack though residential US proxies? Your method falls apart.

How many of us deal with automated password attacks is to issue questions that only locals or people with specific knowledge could answer. Change the questions and do everything custom.

It sounds like they have behavior-oriented rules that are just always triggered on Tor because Tor traffic has a disproportionate amount of bot traffic. I see no reason why behavioral blocking breaks down when an attack comes from an IP space that is usually more benign.

> How many of us deal with automated password attacks is to issue questions that only locals or people with specific knowledge could answer. Change the questions and do everything custom.

If I'm understanding what you're saying, this sounds horrible. What if I'm visiting an area where I don't have local knowledge? What about for the year or so after I move in to a new city? What if your assessment of what locals do and don't know is just wrong? There are a ridiculous number of failure modes in this questions-oriented approach. The only place this could possibly make sense is in some sort of internal company software, but even that context has better options available.

It is used commonly with facebook groups. Having to answer a question related to the group topic filters out spammers. I do it for country specific country sites requiring knowledge. The information can be googled if desire is high.

Facebook groups I can see, especially because they're often surrounding specific niche topics that you can reasonably expect people to have some shared knowledge of, and the administrator of a Facebook group doesn't have that many levers to pull to reduce spam.

At the country level (and for applications where you have enough control over your infrastructure to use a real firewall) I question both the efficacy and accessibility of a system like you propose—it's not that different from the old style "what is 2+2" CAPTCHAs, and there's a good reason why most applications have moved on from those. They're not a serious alternative to behavioral rules like what OP describes.

Cloudflare doesn't hate Tor. Blocking Tor is purely a customer configuration

>Blocking Tor is purely a customer configuration

...that's on by default and so used by the vast majority of Cloudflare customers making it effectively a Cloudflare configuration.

I'm pretty sure it is not on by default and cloudflare also provides onion routing by default so it even helps legitimate Tor users.

They may not block them by "default" but when all the onion browsers automatically have a high threat score for not running cloudflare's invasive javascript it's the same thing. Especially combined with the high "threat score" they attribute to almost all out node IPs. I don't even use tor and they still block my browser when I'm trying to read science journal articles because my browser cannot run their bleeding edge features they use in their spying javascript.

And everyone knows it because that's what the lived experience of trying to access cloudflare blocked sites on tor browser (or any other browser that's not made by a megacorp). It doesn't matter what cloudflare's intentionally ambiguous and probably disingenuous wording might try to imply. The only people who think otherwise have never actually tried using tor to surf the web.

Biggest gap with Cloudflare IMO is granularity of API permissions for tokens. I wish I could create an API token that only grants write across to a specific DNS entry (or regex of a DNS entry).

Original source: https://blog.cloudflare.com/sippy-incremental-migration-s3-r...

Ok, we've changed to that from https://www.infoq.com/news/2023/10/cloudflare-sippy-migrate-.... Thanks!

Yeah the linked post is a bad rehash of the original blog post. Thanks for linking it.

Not really. The submitted link shows you how to use it ( eg. The rest call).

This one shows you how to set it up in cloudflare.

While the info provided in both is indeed similar.

9 cents a gigabyte downloaded versus 0 cents a gigabyte downloaded is a pretty good deal.

There’s not much AWS can do about it because they must make untold billions from those sweet, sweet S3 egress fees.

I’d be willing to bet S3 egress fees make up about 60% of all AWS revenue.

S3 egress and inter-AZ traffic. The way they advertise multi-AZ should honestly be illegal, or at least include warnings about charges.

Like when you set up an RDS instance, the “prod” template defaults to multi-AZ (a good idea tbf), but completely elides the fact that if your app is in a different AZ, you’re going to start racking up $0.02/GB.

Same with NLBs and cross-AZ routing. Sure, it can be helpful, but yeesh.

Or EKS, since Topology Aware Routing is in no way a default.

Or NAT prices... there's a good reason why there's FCK-nat. Those nat prices are terrible

That's not the only thing that has excess fees in AWS/Azure/ ....

Eg. For cloudflare workers. If you're worker is making an outgoing request ( db / rest) it's not considered cpu-time and it's not counted towards that either ( in cloudflare ofc).

While this is a hidden profit of many cloud providers :)

Can you explain more?

Wait time isn’t calculated as compute?

Kenton describes a bit here what they measure and how:

https://community.cloudflare.com/t/how-is-cpu-time-per-reque...

Cloudflare is only billing that actual cpu-time.

Edit: this is a better resource https://blog.cloudflare.com/workers-pricing-scale-to-zero/

That's partially why so many llm/ai apps use cloudflare.

An API call can take a lot of seconds. While Cloudflare only bills cpu-time ( eg. 10 ms. ). Other providers bill those seconds too as "duration", while the CPU was just sitting idle.

I can see both sides because you are reserving that amount of RAM while your process is running. The Lambda price is also proportional to the RAM reserved.

Well.

I thought that this was possible because Cloudflare eliminated cold start delays.

So there's no RAM reserved either, I guess.

( can someone correct me if I'm wrong?)

I have to imagine CloudWatch makes AWS a lot of money too. Very often, when I look into an account from work that is spending a lot, CloudWatch is a large contributor.

Really? From Custom metrics, or logs? It's pretty rare that I hear anyone use it in production, there's usually either a SaaS like Datadog/New Relic or a homegrown setup with e.g. Prometheus.

I had been running https://dockeri.co with https://arc.codes/ for pennies a month.

Then, one month, I got a ~$500 bill out of no where.

Docker had changed an api causing my service to return 5xx errors all month. Each error was individually logged to CloudWatch - which racked up a ~$500 bill.

I moved to Cloudflare Workers that day and haven’t moved back.

Lol.

The Cloud really loves logging ( bills :p ).

It would be nice if Cloudflare implemented "Open telemetry".

It could reduce the cloud bill by at least 2. Logging is really expensive.

C suite types prefer to have a single portal for “all the things” and naïvely assume the price premium they’re paying for AWS is evenly distributed across all their products. It’s also a common assumption AWS is the best in each category.

A good number of people end up using cloud watch for all of the above, even though it’s (comparatively) mid.

All the other parts of AWS are there really only to keep companies using S3.

Anyone doing serious traffic with AWS will use cloud front and then you get additional savings on-top of that and you can negotiate a better rate then provided depending on usage so some people pay way less then the advertised price.

Sure, if you're a business that has to burn money on servers anyways, but a lot of people avoid AWS for side-projects and non-business applications because it can easily balloon to over $100/month with just a few services and moderate traffic.

Counter-point: it can also be extremely cheap.

I’m in a seemingly small subset of people that is very happy with AWS for side projects. Granted I’m not doing anything that requires many resources.

Also, CloudFront can't proxy bare EC2 servers, you have to shove them behind an ELB, which is additional cost.

Of course it can.

So, the CloudFront setup process only surfaces S3, ELBs, API Gateways, Mediastore, and Mediapackage domains as origin domains. I do notice that it will let me type in an arbitrary domain - is that how you're supposed to stick bare EC2 instances behind CloudFront? Just provide it something like realoriginserverplsdonthack.example.com and use some other method (e.g. VPC configuration) to prevent bypassing CloudFront?

Correct, you can put multiple instances (A records) there, if on route53 you can also use healthchecks, geo etc.

If you want to lock ec2 access to cloudfront only you can do it in SG with "managed prefix list for CloudFront".

Is Cloudflare requiring you to switch to Enterprise plans if your usage is above some threshold? In a previous discussion on HN about Cloudflare, a good half dozen people replied saying they got calls when their usage increased above some level that they had to switch to an enterprise plan, and AFAIK those start at $5K/month (from a brief talk with sales a few weeks ago).

We have tens of TB in AWS that we'd like to move to CF, but I'm reluctant to without being able to know if we're going to get a call demanding we switch to enterprise.

I specifically asked this question to Sales and they told me no. There probably isn't an official policy and it's up to the discretion of the people involved. In my case I had 100 MB of data on S3 that I was serving to a lot of users (250 TB - 500 TB of egress per month). This is free for me on R2 because you aren't billed for requests served from cache and we have a 100% cache hit rate. I was very up front about this to sales and they said they didn't care as long as we paid for some kind of support package. We were paying for the $20/mo plan but later elected to upgrade to the $200/mo plan. Not a bad deal since the data transfer from AWS alone was more than $20,000.

I don't want to take advantage of them and get on their abuse list since this is production. I'm happy to pay more! I just don't want to deal with negotiating an Enterprise plan. They ask you so many questions like "how many Page Rules do you want? How many Worker requests?" I just want R2. And this response confuses them too because they say "well R2 is pay-what-you-use..." I would honestly be happier with a $5000/mo "excessive R2 bandwidth" fee. But they don't seem to want to implement that.

$5k/month is just Cloudflare's opening bid. In Enterprise sales, you are expected to negotiate for a discount. Common rule of thumb is that Enterprise sales (in general, across vendors) start at ~$2k/month (this is what's necessary to pay for salespeople's salaries) and savvy negotiators will end up close to that figure for bare-bones plans.

In what world does 2k/month pay a sales persons salary? And what enterprise sales person is OK selling something for $2k/month/$24k/year when they carry a quota 20-50x that amount?

Enterprise deals start at 100-200k/year, minimum, IME.

So cloudflare quotes him $5k so that’s 60k. You’re already wrong there.

Then your 24k a year sales person number is assuming that the sales person will sell one license the entire year, an insane assumption.

> what enterprise sales person is OK selling something for $2k/month/$24k/year when they carry a quota 20-50x that amount

probably someone who spends no more than one man-week on this sale, which doesn't seem that hard if the customer is a small org and only wants to buy one feature? sure, if you expect to spend months negotiating and years implementing, you'll need to charge a lot to make up for the time, but we're not talking about that case here.

I'm not including some full Solutions Engineer / Customer Success Engineer kind of integration work on part of the vendor - I'm talking about, you go to the vendor's website, you see some feature you need (like SSO) is under "Contact Us" pricing, usually the barebones for that is $2k/month. This represents meeting with Sales for 15 minute calls a handful of times so they can qualify you, understand your usecase, present you with a quote for more than you need, then negotiate down.

Enterprise salespeople working in that segment understand you're small-fry and they'll kick you a small contract not because they're going to get rich off the commission but because they know that depending solely on whales is a high-risk approach that sooner or later gets them fired. Small contracts pad them out and provide more reliable monthly numbers as long as the sales process doesn't turn into a tarpit that isn't worth the money.

Data point: They call you once you hit 300TB egress per month on a _free_ tier and ask to start paying (nothing wrong with that, just a data point).

I’ve reached 500 TB a month at one point but never had any issue. Perhaps an email I ignored but nothing went down.

Is that on the free tier or one of the paid tiers like the $60/mo or $200/mo?

Free tier

Yes. They called a company i worked for, we used a lot of data. They gave us a few weeks to move off (what we needed).

Do not think its free for all usage.

AFAIK R2 is priced completely separately on a pay as you use basis. Have you got a link to these replies?

I've done some searching around, but can't seem to find the thread where this came up. It was in the replies to some HN post, and there were 4-5 top level comments all saying the same thing. Wish I would have bookmarked it or something.

This is amazing. I’m thinking of building the same thing for self hosted S3 (ie. Minio). I love AWS, they're very reliable and i feel like depending on them is not risky. But being 100% dependent on a single provider scares me a lot, and I often feel like S3 is the one thing that would cost me a lot if I tried to migrate. I have not accounted for all the data I have there, but it's lots of small files (all the user uploads), and what scares me is the per request cost of deletion (I think I read some horror story here sometime ago). So I keep pushing this off.

Bulk deletes on S3 should be done using a Life Cycle Policy to avoid the per object delete cost

This (although as others said it’s the list cost, not the delete). If you want to nuke a bucket don’t waste your time with the API and just age the files out. I’ve done this with buckets with tens of millions of files and five minute’s work.

Both list objects and delete object are API actions that incur the standard cost for "requests" ... isn't it?

PUT, COPY, POST, LIST requests (per 1,000 requests) = $0.005

GET, SELECT, and all other requests (per 1,000 requests) = $0.0004

"You pay for requests made against your S3 buckets and objects. S3 request costs are based on the request type, and are charged on the quantity of requests as listed in the table below" https://aws.amazon.com/s3/pricing/#:~:text=You%20pay%20for%2...

From the same paragraph:

“DELETE and CANCEL requests are free.”

Deletes are free, but you have to pay to list to delete if not individually tracking.

Turn on weekly manifest to get a list of files and then you can delete based off that. Much faster and better then listing billions of files.

What’s a weekly manifest?

I believe ludjer is referring to S3 Storage Inventory. This is a daily, or weekly, file produced containing metadata on every file within a S3 Bucket. It does not use the synchronous List APIs.

> You can use Amazon S3 Inventory to help manage your storage. For example, you can use it to audit and report on the replication and encryption status of your objects for business, compliance, and regulatory needs. You can also simplify and speed up business workflows and big data jobs by using Amazon S3 Inventory, which provides a scheduled alternative to the Amazon S3 synchronous List API operations. Amazon S3 Inventory does not use the List API operations to audit your objects and does not affect the request rate of your bucket.

> Amazon S3 Inventory provides comma-separated values (CSV), Apache optimized row columnar (ORC) or Apache Parquet output files that list your objects and their corresponding metadata on a daily or weekly basis for an S3 bucket or objects with a shared prefix (that is, objects that have names that begin with a common string). If you set up a weekly inventory, a report is generated every Sunday (UTC time zone) after the initial report. For information about Amazon S3 Inventory pricing, see Amazon S3 pricing.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/storag...

Ignore the replies about inventories. $0.005 Per 1k list requests. A list request returns 1k items. 1 million items for $0.005, or 1 billion items for 5 dollars.

Still use lifecycle policies (I visualized how it ages out 1 billion items here[1]), but list request prices are not a factor worth mentioning.

1. https://tomforb.es/visualizing-how-s3-deletes-1-billion-obje...

Deletions do not encounter costs, but obtaining a list of objects in order to delete them certainly does.

One thing is to migrate existing files and integrate it with your application, but you can also use Cloudflare interface to upload large files, e.g. vidoes, and avoid costly egress fees or having to pay more wherever you host your application. The steps are:

- click on the R2 link in the Cloudflare dashboard and add a bucket

- upload files to be shared (max 300MB)

- click the settings tab and add custom domain that you want files shared from

This prevented me from having to upgrade to Vercel Pro and saved me $240/y. A bit more on the topic: https://sometechblog.com/posts/don-t-overpay-for-bancwidth/

Feels like Cloudflare is slowly eating up AWS. This is a good step and benefits the users.

Can I use Cloudflare R2 and Cloudflare CDN to serve HLS video segments? I always hear about ToS issues with Cloudflare and video. I can never seem to find a straight answer.

They created this document for those exclusively using CF services for hosting content including Video: https://www.cloudflare.com/service-specific-terms-developer-...

In particular the following line should enable it:

> Unlike most Cloudflare products, the Developer Platform can be used to host content.

Also see https://blog.cloudflare.com/updated-tos/

> Over time, Cloudflare’s network became larger and more robust and its portfolio broadened to include services like Stream, Images, and R2. These services are explicitly designed to allow customers to serve non-HTML content like video, images, and other large files hosted directly by Cloudflare.

> Video and large files hosted outside of Cloudflare will still be restricted on our CDN, but we think that our service features, generous free tier, and competitive pricing (including zero egress fees on R2) make for a compelling package for developers that want to access the reach and performance of our network.

(of course this is not legal advice)

This really cleared things up for me, thank you!

Can definitely recommend Cloudflare R2, great value with no chance of any surprise egress fees.

Our last month invoice was only $0.26 for 26GB+ of managed storage.

The whole Firewall and rate limiting of CF is amazing and I like where they're going with the workers and Pages. I've also got a lot out our their free plan for many years but now we're on enterprise it feels a bit extortionate. Every email is another contract to sign and another 2k/mo cause we're going over the initial estimation. Much prefer the PAYG model from the likes of AWS and Google etc, even if it were more expensive, it feels more honest

But who guarantees that Cloudflare will not charge me for egress in the future? Sure nobody guarantees either that S3 won't becomer more expensive, but migrating to another provider just because right now they are cheaper without guarantees of their price in the future seems short-termist.

You can't guarantee anything from anyone like that, but they've made their position clear regarding egress fees:

https://blog.cloudflare.com/aws-egregious-egress/ comments https://news.ycombinator.com/item?id=27930151

If you want guaranteed long-term pricing then you have to put that in your contract with your host.

Does CloudFlare have an on-demand image optimization like Vercel? I love that I can specify <Image src="original_image.jpeg width="500px" /> and Vercel just took care of it for me. I am looking for something cheaper and works on native mobile apps as well.

Yeah

https://developers.cloudflare.com/images/image-resizing/

https://developers.cloudflare.com/images/image-resizing/url-...

Example: src="/cdn-cgi/image/width=80,quality=75/uploads/avatar1.jpg

About /cdn-cgi/image/

It's a fixed prefix that identifies that this is a special path handled by Cloudflare’s built-in Worker.

---

Price at Cloudflare : 50,000 monthly resizing requests included with Pro, Business. $9 per additional 50,000 resizing requests.

See: https://www.cloudflare.com/plans/#add-ons

As a reference: Vercel is $5 per 1000 source images. So Cloudflare is a whopping 25 x cheaper.

Price at Vercel: https://vercel.com/docs/image-optimization/limits-and-pricin...

that is a lot cheaper than Vercel in term of store image but egress seems expensive since they charge per cache image served as well. Do you know if Cloudinary is a better option?

Tbh. I'm mostly following Cloudflare + azure.

I checked quickly and cloudinary works with credits. Cloudflare mentions 1$ / 100 k. Images served, which seems pretty cheap to me.

The pricing I mentioned was for image manipulation, eg. Resizing ( not storage), which was what was asked.

I doubt they are cheaper though.

They have an image cdn that does resizing and such, but the per-request fees are rather high.

What do you mean? Cloudflare is 25 x cheaper than Vercel

Vercel : 5$ / 1000 requests

Cloudflare : 9$ / 50.000 requests

Google cloud storage support plz. Also warning to people to use R2 from the get go otherwise you will be stuck with weird architectures like this using both paying for both

Just a note.

Google Cloud is already in the bandwidth alliance, while AWS is not.

So you can send your files already to cloudflare at a discounted rate.

https://www.cloudflare.com/bandwidth-alliance/

Additionally, there are reasons to keep your data duplicate within AWS. There's no one-size-fits-all here.

The point is to make use of cloudflare free eggres to get your data where you need it ( since others have egress costs, but no ingress costs)

Eg. Train ml models in AWS, GCE or Azure ( where compute is cheapest) and make use of cloudflare to provide them with the data.

This should create a huge cost saving, since there's no eggres anywhere.

Example: https://blog.cloudflare.com/cloudflare-r2-mosaicml-train-llm...

Wow this is really clever. Would be cool to have an open source data migrator with the same concept

That thought was my first impression as well. However with a second thought i'm not sure if i'm convinced by the long time scenario that you would end up using this tool. After all it only makes sense to use this tool if you never intent to migrate the data completely. Which means, the data will till the end be scatered between AWS and Cloudflare with no apparent system other than the last access. Maybe I am overlooking something.

Nah. This is literally to have a migration path and to only occur once the Amazon eggres fees.

At the end, you can still decide to migrate all data or to abandon the not-used-till-now data.

Yes, it's free/very cheap to move S3 data to a tier like Infrequent Access or Glacier with lower monthly cost and higher retrieval cost.

"higher retrieval costs"

Getting that data is really high. It should only be used if you are sure it will never be accessed.

I've read quite a few posts where they complained about the huge bill, when they needed to get their data...

Not sure how it is right now, it's still obscure on their pricing page and before, you had to check the gotchas in their FAQ .

Doesn't mention if you have some granular control over deletes. Like, if an object has migrated, can you delete it in a way that it's not re-fetched from AWS? Or in a way that it is?

You can also use flexify.io, which isn't free but is much cheaper than AWS' egress fees.

They force you to use their DNS servers if you want to use their S3...

Incorrect. That's only if you do cdn.

True but they expect S3 bucket names in a certain manner .

this is awesome.

aws and cf are the perfect combo, and them competing on quality and price is great for consumers.

aws is the control plane cf is the data plane and entrypoint. what a time to build.

That's a smart way of doing it. Apt naming, too.

Sipping revenue from AWS, awesome!

for the egregious egress fees

They have no egress fees.

https://developers.cloudflare.com/r2/pricing/

I mean aws :-)

vampire attack, nice

Vampire hunter attack, nicer.

Great to finally see this here! Cloudflare is the 4th major cloud platform that (almost) nobody is seeing coming, and I believe that they will even surpass Google Cloud Platform's sooner than expected.

The only problem that remains is their support...

I'm writing a book about Cloudflare (launching very soon) where I share this and many other things to scale faster all while saving big on your cloud bills. You can join the waiting list here: https://kerkour.com/subscribe

100% agreed.

The most consistently amazing thing about Cloudflare is the clarity of their product positioning.

You have this common problem, we built a thing to fix it.

No 'change your problem into this other problem' gymnastics. Just 'pay us once you exceed the free tier, and it's no longer a problem'.

And furthermore, they seem to have clarity of platform vision, in that each piece does something very specific to help them compete efficiently against AWS/Azure/GCP (who have much larger resources) AND has synergies with their existing platform. E.g. edge compute, free/cheaper network traffic from compute/storage

Critically, Cloudflare seems like the only competitor to the majors that has their eyes on competing on price by capturing enough of the market of {some thing} that they can still make profits at extremely low price points.

Also, just glanced at their financials again, and they look exactly like you'd want to run a large company if your eye was on order of magnitude growth. They just pivoted to positive FCF in 2023, biggest expense is sales and marketing (over half their gross profit), and have exponential revenue growth.

If you want to read more about their financials of Q2, I found this very interesting:

https://softwarestackinvesting.com/cloudflare-net-q2-2023-ea...

I follow cloudflare closely and we use them at work. I agree with your statement :)

Their support is superbe, but it takes a while to access.

The only time we needed them, the chat option seemed relatively quick.

+ they pointed us to a tls connect issue at Azure with a very detailed analysis of why.

Thing is. If you see a cloudflare error page, it's probably you're hosting provider and not cloudflare...

Support and also all-around enterprise readiness. Even on the enterprise tier, their permissions management is a pale shadow of what IAM grants you on AWS or GCP, to the point where you will put your compliance as risk. No documentation on setting up SAML/SSO for their management console. It's very, very clear that their internal growth engines are set to ludicrous growth rates (to try and justify their outrageous stock price) and the organization is coming apart at the seams. None of which takes away from the fact that the core engineering is top-tier and the core tech product is best-in-class.

We'll see if NET survives public investor expectations.

I think they made their permission system much more fine grained during developer week.

Is your info still up to date? ( I'm not following this topic too much, but I do remember some things passing by).

---

Additionally, most of their investors are companies and not private.

There's a lot going on. One of the improvements that they did was in the sales department.

If those previous sales that were severely underperforming are now replaced by even average sales. Then expect a big rise in sales for Q3.

Reference: https://softwarestackinvesting.com/cloudflare-net-q2-2023-ea...

Yeah, for example, you can grant Edit permissions on Cloudflare Workers overall within an account, but you cannot grant permissions on a single Cloudflare Workers deployment. Any developer who has permissions in, say, a development Cloudflare Workers deployment will thus have full permissions to the production Cloudflare Workers deployment, or permissions to deployments owned by other teams.

please find the docs for SSO setup for the CF dashboard here: https://developers.cloudflare.com/cloudflare-one/application...

Why is this buried deep in the docs for Zero Trust and not part of user management? Why are there no references to it from user management, either in the docs or in the add/remove users screen?

So a cloud provider that does offer regular compute ( VMs )?

I find Cloudflares pricing webpage incredibly confusing.

What's confusing?

This is about R2

> Storage: $0.015 / GB - 10 GB free

> Class A operations (mutate state): $4.50 per million - 1 M. free

> Class B operations (read state): $0.36 per million - 10 M. free

Cloudflare workers:

> $0.15/million requests per month ( 100 k. / day included)

> Up to 30s wall time per request

Min. 5 $ / m. when exceeding the free tier.

https://www.cloudflare.com/plans/developer-platform/

On pricing page for R2 it says "Storage: 10 GB / month" is free ... what does it mean per month?

Info that there are "zero egress fees" is only available on R2 product page and not pricing page.

IMO R2 pricing page look like it only displays quick info and that there might be fine print somewhere, but there is no link to more details. It could be that's all there is to it, but somehow design feels off to me. Especially because of the "zero egress fees" info being displayed only on the product page.

Workers product page shows "Maximum number of scripts": 30 free, 100 paid. But on workers pricing page it shows "Up to 100 Worker scripts" for free and "Up to 500 Worker scripts" for paid.

Links to different sections (Pricing, Products...) don't have an option to open in new tab. IMO the whole website is weirdly organized. But maybe it's just me.

Concerning R2/storage:

Your are billed every month for your total storage. Every month you don't need to pay for the first 10 GB.

I'm not really sure if eggres has to be mentioned if it's widely known that Cloudflare doesn't bill for eggres. But I get your point.

Concerning Cloudflare workers:

Workers has a free tier and a paid tier at 5€/month.

The free tier has a limit of 100 workers and the paid tier has a limit of 500 workers.

Perhaps just scroll down a bit more on the pricing page of Cloudflare workers. I'm assuming you are checking it on Mobile and missed that.

About but being able to open pricing in a new tab. I noticed the same.

---

They also have a minor UX issue that if you want to go to the Web analytics page, the menu goes to the first child and hides. So you'll have to click it open again and click on Web analytics ( again, just an issue on Mobile)

Compare info on workers product page and on workers pricing page, not everything matches.

EDIT: I'm testing on desktop.

Ok. You're right.

Notified them on their Discord of workers, let's see if it gets picked up tomorrow.

Edit: It's going to be escalated and fixed ( Got a response within 44 minutes ... On a Sunday, nice).

Reply on news.ycombinator.com