In the immediate finds I'm seeing, these two matter most to me:
>Cloudflare used a free Fastly trial account to conduct their tests. Free trial accounts are designed for limited use compared to paid accounts, and performance under load is not comparable between the two.
>Cloudflare conducted their tests in a single hour, on a single day. This fails to normalize for daily traffic patterns or abnormal events and is susceptible to random distortion effects. If you ran several sets of tests at different times of day, it’s likely that at some point, you’d achieve your desired outcome.
Neither of these are good. For one, paid uninhibited account would cost little to do this test properly. You don't know if you hit their rate limits, are de-prioritized over paid users etc. There are lots of reasons why this is could cause the metrics to be skewed.
The second one is an even bigger deal. Sample size matters, as does variance of inputs. This is just sloppy all around. 1 hour on 1 day will tell little of performance. Ideally, this should be done quite randomly, with a bunch of different random workload variances, and sampled over weeks, and from different locations (east coast, west coast, midwest, Canada, Europe etc). This would give a real sense of the overall network, and leave a lot less room for Fastly to hit back if the benchmarks are accurate.
Overall I think Fastly may have caught Cloudflare red handed in sloppiness of comparision. For Cloudflare, this is pretty poor execution. Unfortunately, if or when Cloudflare does a response piece and still backs their claim, they're going to look more suspect if they aren't more thorough with their methodology and reproducibility