I think it's rather disingenuous for Cloudflare to publish their own benchmarks calling out competitors when they won't allow anyone to run their own tests and comparisons.
https://www.ilga.gov/legislation/ilcs/fulltext.asp?DocName=0...
https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act#A...
In the EU there are various ways of ToS are limited in their power, if that applies here I'm not sure.
But they are basically forbidding you to evaluate how well a product you (might be) paying for works, depending on what they advertised they might even prevent you from evaluation if the advertised product characteristics uphold. Especially in the later case I guess that the ToS clause would be void.
I never knew how true that anecdote was but that’s typically the boogeyman story that gets passed around for things like this
Consider the following scenario: You calling your ISP that you're going to do a benchmark. Then the ISP gives you the best service, at the expense of other customers bandwidth, for the duration of the test. Then after the test, the ISP removes the QoS modification, and reverts to the old behavior. That could end up with a biased, unrealistic result.
As for the latter, that benchmark wouldn't reproduce and would require extensive configuration (which in a sense is just a part of negotiation between the customer and the client). If a company got caught faking benchmarks run by competitors and it could be shown publicly they did, that's pretty much the end of that company's public reputation.
Frankly, that's a distinction without a difference in this context, at least in my opinion.
That's because you don't have a good enough reason to want to ask for permission though. If you were spending $xx thousands on edge workers every year you'd just ask, and you'd rule CF out if they said no. But, very likely, they'd say yes.
That's a fake opening with the same effect as prohibiting something by ToS.
Who's to say that the performance of the services won't coincidentally perform better after you've given your request to test?
I am sure they might mention some legalese of "avoiding putting arduous strain on the services which may cause disruption to our other users" as a justification, but that PR speak won't fly with me since there is a separate section that already references that.
I say this as a big fan of Cloudflare and I am sure they don't need to use such underhanded tactics to make their services look good.
The TOS clause, for reference: https://www.cloudflare.com/terms/#:~:text=(f)%20perform%20or... (ctrl-f benchmark )
> So instead, we ran the same tests (echoing headers and measuring TTFB via Catchpoint) against our own platform
The Cloudflare terms prohibit benchmarking without their explicit permission.
Am I the only having cognitive dissonance here? I was confused how they complained about not being able to perform benchmarks but then… ran a benchmark? There must be some distinction here that I’m missing
However, I see that the TOS says you may not "perform or publish" benchmarks. Seems this still violates the "or publish" part, even if Fastly avoided doing the performing. What would be the point of forbidding publishing if it is already forbidden to perform said benchmarks in the first place? The TOS seems to also forbid publishing on the benchmark even when someone else performs the Fastly part of the benchmark.
Edit: It must be the TOS doesn't apply if you don't use the service.
From "Cloudflare Self-Serve Subscription Agreement" https://www.cloudflare.com/terms/ , "2.2 Restrictions. Unless otherwise expressly permitted in writing by Cloudflare, you will not and you have no right to: [...] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent;"
From "ENTERPRISE SUBSCRIPTION TERMS OF SERVICE" https://www.cloudflare.com/enterpriseterms/ , "2.3. Restrictions and Acceptable Use. Customer must not: [...] (i) perform or publish any performance or benchmark tests or analyses relating to the Service, other than solely for Customer’s internal use;"
Generally, it sounds like the kind of thing that would be obvious to a lawyer (we want to prohibit random people badmouth our product, right?) but stupid to engineers.
The best outcome would be if Fastly and Cloudflare both dropped any language limiting benchmarking and sharing of results.
If they want to protect their brands, the most common sense requirement would be "We require you to tell us, within X hours after publishing, of benchmarks you're publishing and send us a copy of the article in which you published them (or link if publicly accessible)."