How Much Traffic is Too Much Traffic For CloudFlare?
phoboslab.org
phoboslab.org
The attack continued and began to affect the performance of other CloudFlare customers, at which point we routed traffic to the site away from our network. While we encouraged the site owner to take advantage of the Enterprise tier of service given their needs and traffic levels, the site would have been brought back onto CloudFlare's network if they had upgraded to the Business tier of service ($200/mo) which included Advanced DDoS mitigation.
To be clear, CloudFlare does not bill based on traffic. However, resources are not infinite and when an attack against a Free customer begins to affect the performance of other customers we will take measures to protect the overall integrity of the CloudFlare service.
Matthew Prince, CEO, CloudFlare, @eastdakota (Twitter)
From the comments on the site: "I didn't notice any "attack" when CloudFlare began to route all traffic directly to us. It looked like normal web traffic - much of it, but no more than usual."
All we know is that the author ran on the free plan, and probably should have upgraded from the free plan when he started seeing his site getting large amounts of traffic.
In the end all is well, he got another service that served his purposes.
Yes, I know they did offer one hell of a starter/"sweet lolipop to sucker you in" pack - but that's still not what's being discussed.
It has been re-iterated many times in this thread - but CloudFlare had a sane person on the other end that was willing to open his wallet - that's something one should act on quickly.
The article was mostly moaning about communication (or lack of it). So can you talk about how these actions were communicated to the site/client?
http://i160.photobucket.com/albums/t167/vengethis/bullshit_d...
This reads like "We encouraged the site owner to pay 15 times more than they needed to."
Which is it, do they need the business tier, or the enterprise tier?
I've been a happy CloudFlare customer so far, but the lack of transparency in rules and pricing is concerning. You don't charge for bandwidth, but can disable sites at your discretion if it causes problem in your infrastructure? This sounds a lot like ISPs that offer "unlimited" bandwidth but start throttling you at some predetermined but unknown cap.
Doesn't this make CloudFlare customers more vulnerable to attacks, since an attack will result in a monthly fine for the rest of the site's lifetime? (whereas a normal site just pays a one time cost)
So... what you're saying is that you bill based on traffic?
On the other hand: CloudFlare comes off as terrifyingly incompetent here - which more or less matches with my experiences trying them out on a site serving maybe 100GB worth of traffic at most in a month. They seem to have missed dozens of great opportunities to upsell you on their paid service, and when someone finally noticed how much bandwidth you were using, they completely lost the plot.
What should have happened, IMO, is something like this: A panicked CloudFlare admin realizes your site is using 100TB/mo. Their first step is to send you a sternly worded email, explaining that for this usage level you need plan X, and if you don't upgrade within... let's say 5, maybe 7 days? They'll be throttling or limiting your service. Then you don't feel pressured to solve the problem right away, and they are trying to retain a customer that (presumably) they value.
Instead, they haphazardly change your settings behind your back (???) and then later take various steps to reduce your bandwidth usage before finally deciding that you need to pay them, without even figuring out how much money they want. Ridiculous.
To me this just says that CloudFlare is running on what may be a fundamentally unsound business model, and that by claiming their free tier doesn't charge by bandwidth (and not listing any limits) they're dramatically increasing the odds that customers will suddenly discover there are limits after all, and leave. If they were more up front about what the actual pricing structure is, it'd probably be more likely that people would start paying for the value that CloudFlare gives them.
He was happy to pay, they didn't capitalize on that.
All CDNs offer DNS level DDOS protection by default. Not because they choose to, but because they have no other option. After all, the same IP ranges are used by all clients and this makes it impossible to pinpoint the original target. (thus no one to blame/bill)
Every CDN does it, but only CF claims it as a "feature".
Do you know any good ones I can check?
Realize that people will pay money in order to not lose something. This is a very strong mechanism to use to increase the profitability of your products.
Also, similarly to OP, we are regularly (fortnightly) automatically put in "I'm Under Attack" mode without any prior warning or consent which is quite annoying as it tends to happen overnight so I am not alerted until someone checks the live site in the morning (it still returns a 200 so current checks don't pick it up)
"SECTION 10: LIMITATION ON NON-HTML CACHING
You acknowledge that CloudFlare's Service is offered as a platform to cache and serve web pages and websites and is not offered for other purposes, such as remote storage. Accordingly, you understand and agree to use the Service solely for the purpose of hosting and serving web pages as viewed through a web browser or other application and the Hypertext Markup Language (HTML) protocol or other equivalent technology. CloudFlare's Service is also a shared web caching service, which means a number of customers' websites are cached from the same server. To ensure that CloudFlare's Service is reliable and available for the greatest number of users, a customer's usage cannot adversely affect the performance of other customers' sites. Additionally, the purpose of CloudFlare's Service is to proxy web content, not store data. Using an account _primarily_ as an online storage space, including the storage or caching of a _disproportionate_ percentage of pictures, movies, audio files, or other non-HTML content, is prohibited. You further agree that if, _at CloudFlare's sole discretion_, you are _deemed_ to have violated this section, or if CloudFlare, _in its sole discretion_, deems it necessary due to excessive burden or potential adverse impact on CloudFlare's systems, potential adverse impact on other users, server processing power, server memory, abuse controls, _or other reasons_, CloudFlare may suspend or terminate your account without notice to or liability to you."
In other words, you can't host non-HTML, but you can if it isn't disproportionate, but if it is disproportionate, they can deem you to be a problem and cut off your service, without notice. That's not a contract at all. In legal parlance, that's an illusory contract -- when one side can modify their performance in any way at any time.
I use Cloudflare's $20 a month option and it worries me now that I might be deemed to be using a disproportionate about of space or bandwidth caching images, and then be cut off without notice.
EDIT: I love the cloudflare service and I'm not complaining. I just think their legal department needs to clarify this and the tech side of the house needs to be able to warn users when they are exceeding the bounds of what is acceptable.
> If and when your bandwidth usage gets high enough that one of our customer service people gets pinged about upgrading you, they'll also have a look on the dashboard to see what mixture of filetypes you're serving. If it's all static content, you're in trouble.
I'm not quite sure how else that could be phrased into legalese, than what they already have there.
From being shut off when you have a incoming DDoS of some arbitrary size to actually loading webpages slower than your server does vanilla, it seems the benefits of Cloudflare are mostly hype.
Both enabling "I'm Under Attack Mode" or routing the traffic direct are both supposed to generate an automated message to the customer letting them know what happened. We've reviewed the logs and don't see a message having been sent. I'm investigating why that didn't happen since I agree it is not acceptable.
At 100TB/mo., pure file delivery, you'd need to be an Enterprise customer. Let me know if this works within your budget.
An interesting proposition - If we take it at face value, $3000 for 100TB works out to be $0.03 per GB. That's pretty high these days. If you are buying downmarket (which is cloudfare like traffic quality) you can get a cdn deal for maybe $0.01 on a 6 month term with these kind of levels, and somewhere around $0.005 for an xc in the us or eu no commitment. Cloudfare should be buying at substantially better rates than these (or at least, they seem to imply it - calling bandwidth free) so it's it seems they have a similar problem as many freemium models - when most of your customers aren't paying you have to really hit the ones that do.
> somewhere around $0.005 for an xc in the us or eu no commitment
From whom? Or is this reverse engineered from a fully saturated link?
I was thinking of fdc in amsterdam in the eu on a 10gb unmetered no bw sla, but I also think they offer a similar (slightly higher) deal in denver. I was thinking of somebody else for the US - in SLC - but the name escapes me. If you actually could use the name I'll find it.
I shouldn't have said xc as really they expect you to buy power and space from them. Clearly not what I'd try to run an upmarket video cdn off of, but i would be surprised if people like fdc aren't who CF buys from.
Those kinds of offers are generally unusable if the bandwidth matters. This goes back to the oversubscription model, selling the same resource to multiple customers and letting the customers jockey for use, hoping most will never use it enough to catch on.
See this thread for more:
— http://www.webhostingtalk.com/showthread.php?t=1159276&p...
Note this analysis:
“The FDC offer is a shared 10 Gbps and i believe in another topic was explained, that you're supposed to stay at 1 to 1.5 Gbps usage. Even a pure 10Gbps peering port will cost more then $500,- / month, so a true 10 Gbps for anywhere close to $500,- cannot be expected from any provider worldwide.”
— http://www.webhostingtalk.com/showpost.php?p=8153934&pos...
And about the actual usable bandwidth:
“I always found this to be a problem with all FDC locations. So many places have such horrible speeds that no matter what speed they offer, I have a hard time making a good use of it.”
— http://www.webhostingtalk.com/showpost.php?p=8258933&pos...
Another comment:
“The server is great. The single-thread transfer speed isn't as much, but it's reasonably passable most of the time, considering the server cost. But I've been seeing some severe routing problems, with 5-10% of the net simply being unroutable much of the time, as well as intermittent packet loss. Due to this, the overall fail rate of this server is about 10x that of other servers I have in NL.”
If you're running an ad supported viral image host and can fit your popular content in RAM, this kind of thing may be acceptable, though in the long run users will tend to migrate to image hosts that serve their memes quickly.
TL;DR: Not all bandwidth is equal. You get what you pay for and what your provider pays for.
A site I'm involved with ended up on Leaseweb rather than a big-name CDN for much the same reason - we have 3 x 100mbit unlimited traffic servers, and all 3 have been pushing an average of 90mbps 24/7 for the past year or so.
It's not quite as smooth and reliable as a proper CDN, but it's an order of magnitude cheaper, and they've never given us any trouble about using too much of their "unlimited" bandwidth.
"Service Level Agreement (SLA) - 100% uptime
Industry standard SLAs often feature 99.999% uptime, also known as the five 9s. At five 9’s your website could be offline for as long as 5 minutes and 26 seconds each year. All CloudFlare Business and Enterprise plans offer guaranteed 100% uptime because we know that anything less than 100% is an impediment to your organization’s success."
It sounds a lot nicer than it is, much like "we never charge for bandwidth".
There are many cheaper options. The ones you mention seem to be the most expensive.
Take 100TB for example (the first result in that Google Search)
d. We strive to maintain a high level of service, and a lot of customers
depend on our high standards of quality. As such, we will not provide
Services to those that are using our Services for:
vii. Using the Services for a content delivery network or content distribution
network (CDN). An authorized CDN network offered through 100TB is
accepted. Special requests to use the Services to run an unauthorized
CDN network may be approved on a case-by-case basis. Failure to comply
with this policy will result in termination of this TOS, and you will
not receive a refund of the Fees.
Getting a server somewhere that has 100tb of outgoing is something different.
Contrawise ,if I'm paying $2-$3/megabit @95th on a 1 Gig Port, the amount of support I can expect during a DDOS is pretty minimal, so I end up having to take the hit - but my damage is limited to $3000/month so I don't really care.
Any time I see a "We don't charge for Bandwidth" service, I interpret it to mean one of (A) We'll throttle you once you exceed our unspoken limit, or (B) We'll discontinue your service. (Drop your port from 1 Gig down to 100 Megabits, or slower, traffic shape you, etc...) once you breach that limit.
There is no sustainable third option for those who provision reasonably high quality transit, and those who believe there is will one day wake up with their internet property offline, or seriously degraded.
A customer is a person who pays for a service. Someone who doesn't pay for a service (yet) is a lead. Not all leads are good business. 100TB of traffic does not sound like a good lead to me, not even at the $200 level.
Looking at the site, I see an IMGUR clone which was running for free off of CloudFlare's cache servers. I really don't understand the nonsensical comments on the article. WTF is wrong with people these days thinking that everything is supposed to be free? Are you all 16 and on an weekly allowance?
Commenter Matt had a very valid point that some sort of optimization of the stored (cached) files would have been a smart option for yourselves (less local storage) as well as CF (less to cache, less bandwidth). I'd recommend http://www.jpegmini.com/server (Oh wait, it's not free, now what... cry me a river)
I have a question for the Phobos peeps. Were you making money? Seems to me like you were... since you can afford the LeaseWeb servers. Instead of bitching publicly, perhaps you should have reached out to the company when you noticed your traffic levels were reaching antisocial magnitudes.
Grumpy Gramps (who used to pay UUNET $6k+ a month in 1997 dollars for the privilege of hosting a basic database-driven e-commerce site for a luxury watch brand on a guaranteed T1 connection)
This line in the article "I tried to monetize it through ads some time ago, but failed. Advertising-Networks don't want us as a customer, but I'm fine with that." was not conclusive that monetization failed via other means. Not a user of the site, but at first glance some choice links to pr0n affiliate sites would do the trick.
Throwaway ey... hmmm.
This is a common observation - less the user pay the worse and less objective the feedback is.
Once you get past the first few tiers of paid service, providers tend to reign in the bullshiting "technically true by widely misleading" descriptions and are upfront about what they provide. Properly managed expectations don't lead to as strong of emotions when a customer finds out what they're getting doesn't work for them.
Commercial users will save face by A) avoiding discussing anything that could make them look bad, ditching a vendor is a tacit admission that you went wrong when you chose their service. Additionally, if their service is built on top of the ditched service, it casts quality concerns on their product too. B) Avoiding disparaging anyone, ever, is pretty common play-nice-save-face.
Why should I give my competitor helpful advice? I got caught up in a web, I don't need to help the competition avoid my mistakes.
In a multiple person organization, the party angry about the service and the party who decided to buy/use it are different individuals and airing grievances/venting is an internal process that they don't reiterated in public.
In a free service, a failure to deliver is the only downside. If I paid for a service, I'm going to be too preoccupied beating myself up for making a stupid purchase decision and looking for an better alternative than bothering to share my findings.
These causes and more are all often balled up and interpreted as free users acting "entitled". Sure that exists, but its a classic case of reading too much into imprecise metrics when someone assumes the cause.
Furthermore, the feedback from free users isn't inherently worse, and is just as subjective as that from paying users, it just tends to be more negative. For all we know, any given product could objectively be shit and because only rubes will buy it, the free users are the only ones that can let you know.
They turned it off due to an attack and would turn it on again if upgraded to business ($200/mo).
Does this mean turn it on while under attack would cost, and after the attack would be free again? Or it would cost the upgrade to turn it on either way?
The process would have been automatic if he was a paying customer. Because he wasn't, the company felt justified in not trying harder to bring their resources back up.
Imagine the origin server would have been Amazon S3. The webmaster would have incurred an Amazon bill of $400 USD per day after the switch.
4chan probably also expects to have more than an email (and timely) endpoint with which to correspond with the provider of their most core site-service -- image serving.
Were you already using http cache of 1 year for all the images when this happened? Do you think it could have been avoided by setting it to 1 week, 1 day, or even 1 hour?
It's funny that I have to ask this to you instead of asking Cloudflare. They have really messed up on this one.
Whether he can tell us if he is rich or found another way to monetize adult or potentially offensive content, it is up to him to share.
Also, I learned a great deal of JavaScript while building that page and I just now learned a bit about load balancing and setting up Varnish - something which may come handy for godknowswhat. I never imagined making my living by writing a JS game engine either.