Debunking Cloudflare’s recent performance tests (2021) | Hacker News Reader

Debunking Cloudflare’s recent performance tests (2021)

fastly.com

185 pointspromiseofbeans68 comments

First, I'd be shocked if Fastly didn't try to produce some counter arguments to the Cloudflare blogpost. I'm not saying Fastly nor Cloudflare are objectively right, however, there are a couple things that stood out to me here.

In the immediate finds I'm seeing, these two matter most to me:

>Cloudflare used a free Fastly trial account to conduct their tests. Free trial accounts are designed for limited use compared to paid accounts, and performance under load is not comparable between the two.

>Cloudflare conducted their tests in a single hour, on a single day. This fails to normalize for daily traffic patterns or abnormal events and is susceptible to random distortion effects. If you ran several sets of tests at different times of day, it’s likely that at some point, you’d achieve your desired outcome.

Neither of these are good. For one, paid uninhibited account would cost little to do this test properly. You don't know if you hit their rate limits, are de-prioritized over paid users etc. There are lots of reasons why this is could cause the metrics to be skewed.

The second one is an even bigger deal. Sample size matters, as does variance of inputs. This is just sloppy all around. 1 hour on 1 day will tell little of performance. Ideally, this should be done quite randomly, with a bunch of different random workload variances, and sampled over weeks, and from different locations (east coast, west coast, midwest, Canada, Europe etc). This would give a real sense of the overall network, and leave a lot less room for Fastly to hit back if the benchmarks are accurate.

Overall I think Fastly may have caught Cloudflare red handed in sloppiness of comparision. For Cloudflare, this is pretty poor execution. Unfortunately, if or when Cloudflare does a response piece and still backs their claim, they're going to look more suspect if they aren't more thorough with their methodology and reproducibility

Those might be bad, but comparing rust compiled to web assembly vs. JavaScript is much worse here

I'll assume you're right but can someone with more knowledge of why explain this? I get the whole compiled vs JS's event loop stuff but I've never worked with edge computing so I don't really understand the details of what matters here in the full stack of a request.

From my limited research in the past for web servers JS is not that far behind Rust. In fact they often rank better. For this "single query" JS (just-js) ranks 1st with a rust one at 3rd:

https://www.techempower.com/benchmarks/#section=data-r21&tes...

for "fortunes" requests JS is 5th place with Rust at 2nd and 4th:

https://www.techempower.com/benchmarks/#section=data-r21

etc, but again not sure how relevant this is

I am guessing warming up, i.e. JIT?

FYI, just-js is a js runtime specifically built for beating techempower benchmark. It is not for general use case.

The issue I have is that, if you’re claiming to test the performance of Cloudflare workers vs. Fastly’s similar service and, especially, if you’re going to write a blogpost about the statistical issues with Cloudflare’s benchmarks, you really should do your best to make the only difference the platform. So either JavaScript on both or rust on both so it’s obvious that the language and runtime aren’t the source of performance differences.

That sounds like the most reasonable take

At the time this was going down, cloudflare had web assembly and fastly didn't.

Fastly's complaint is "we don't offer the fast runtime so its unfair that you point out that you have a faster option while claiming you have a faster option."

Fastly’s compute at edge has always been based on WASM… there have been some pretty interesting talks by Tyler McMullen, their CTO about it

Cloudflare used JS for edge workers as their launch choice, and Fastly used Rust as their launch choice so of course it makes sense to compare the two most mature techs in their stacks

The funny side point here is this is two unprofitable CDNs squabbling while the unfashionable Akamai just keeps making money — an interesting case of first mover advantage

First mover advantage has mostly been debunked as a strategy, the history of tech startups is littered with mildly successful first movers (Friendster being the classic go-to anecdote) that later gets dominated by 1-2 tech companies who do it better.

Akamai's advantage is probably it's sales business and existing contracts. Workers is just business #1383 for Cloudflare which can take advantage of their freemium scaling model

So if first mover advantage has been debunked why is Akamai not only the largest CDN by sales but also the only profitable one too?

I'll agree that a one hour, one day test isn't the greatest, but cloudflare workers have a free tier- comparing to fastly's free performance isn't entirely dishonest.

What is dishonest, as a sibling pointed out, is compiling rust to WASM and pretending it is a more fair comparison.

Reading through both posts, I'm not exactly finding fastly's defense to be meaningful.

Title needs to be updated - this is from 2021! In the edge compute space that makes it irrelevant.

I wish articles like this had prominent dates!

It makes the results of the comparison irrelevant. It does not make the accuracy (or honesty) of the testing methodologies employed and reported irrelevant.

what happened next then? were there more follow ups?

Cloudflare said game on and is now faster than fastly everywhere

Well, except Liberia

I added more details here, with what I remember from those days: https://news.ycombinator.com/item?id=36733580

( Then noticed your post)

Uh, what?

Why should I, as a customer of a hosted, managed, SaaS product, need to ever care or think about the time of day as a factor for the performance of my product? If Fastly, as a hosted SaaS, isn't managing its own capacity well enough to deliver consistent performance at all times, that's on them.

Additionally, provisioning hidden limits between different types of accounts, especially limits that affect performance for a CDN whose objective should be delivering consistent performance, is deceptive at best, and malicious and false advertising at worst.

If you think that any CDN doesn’t have performance variations, you’re going to be sorely mistaken.

There’s a reason that many sites that require efficient content delivery have many different CDNs and optimize between them. Well, there are many reasons. But one of them is QoS.

Fastly could have picked a bad Cloudflare hour and do their own countertest.

For a product named Compute@Kintergarten?

The article outlines

- Cloudflare doesn't strictly allow benchmarking - Thus fastly compared against Cloudflare's own metric and numbers

My biggest issue with this is it comes across as desperate, defensive, and whiny.

Fastly would do better to go on the offensive and prove their worth rather than complain about everything a competitor says.

The only time I have seen someone use multiple CDNs was for live video content delivery. Can you name any other sites which have multiple CDNs? Excluding JS CDNS which people use for convenience..

Fair point -- almost all of the use cases I can think of revolve around delivery of streaming content.

But I think they're also probably the users most likely to drive performance variations too

I think the opposite is true: Video content delivery is not very sensitive to latency and even live content has a little bit of buffering. The raw network throughput capability of the CDN in the region of the clients is more important.

> Why should I, as a customer of a hosted, managed, SaaS product, need to ever care or think about the time of day as a factor for the performance of my product?

If someone ran a performance benchmark of your SaaS product many times over a few days until they got a result that looked bad in comparison to their own product, then proceeded to use that one sample in their post, you'd be pissed too.

Performance isn't constant, and especially for a (presumably low-priority) free offering, it's beyond sketchy.

Exactly. This is so obviously an issue I’m shocked anyone would disagree.

The question is: how many times did Cloudflare test before finding the worst hour? I rather doubt they registered their trials!

(But it could also be “just the one time” because they are that much better… who can say?)

Measure over a long period of time and you can measure the variance of the performance. Which determines if you should care about the time of day.

Let’s say cloudflare tested at 1pm their time.

And that happened to be 1am NZ time.

And it made cloudflare look good but vastly look bad.

But you run the same test at 1pm NZ time and fastly is faster cos cloudflare can’t handle the load.

Obviously it does matter to you as a sad providers because time of day does matter for when the tests are conducted.

For point 1, Is/Has Fastly ever offered a self-service paid plan? Last I tried, you had to talk to sales if you wanted to be able to pay them any amount of money for a regular amount of traffic or to access products like Compute@Edge.

So for Cloudflare's tests, I can see why they wouldn't want to directly say "we're benchmarking you, can we have an account", since that would require an entire contract to be set up and approved by the purchasing department, and it could give Fastly the time to flag their account to "over" prioritize it, potentially affecting benchmark results.

That doesn't change the fact that it's an apples to oranges comparison.

This is bullshit. Eastdakota refuted fastly's blame game then and this year blew fastly out of the water.

https://twitter.com/eastdakota/status/1671896844513734656?t=...

> Love that the only country where Fastly is fastest is now Liberia.

Of course this graph is probably bullshit too as we simply don’t know enough about how the data was generated…

> We use Real User Measurements (RUM) and fetch a small file from Cloudflare, Akamai, CloudFront, Fastly and Google Cloud CDN. Browsers around the world report the performance of those providers from the perspective of the end-user network they are on

Even the blog post they reference doesn’t cover much more detail

Key thing is to be measuring via RUM they must be injecting the script into sites they serve (and presumably the free ones) so there may already be bias introduced here, also why choose a 100kb file, how does the data differ if say a 10kb or 200kb is used etc.

Lots of questions about the methodology remain

There's always going to be something to complain about, I guess...

To counterpoint, I always found it interesting that cloudflare talks about population density ( big cities) on where to perform faster. Since that's where most of the population lives.

I haven't seen any other companies talk about that outside of cloudflare.

And it seems a more important real-life benchmark than deciding the difference between 10 kb, 100 kb or 200 kb...

Tbh, I think 100 kb. Is just the smallest relevant size which they thought, looking at their speedtest file sizes of 100 kb, 1 mb and 10 mb - https://speed.cloudflare.com/

> There's always going to be something to complain about, I guess...

Cloudflare are making the claim that they’re faster than others, and they collected the data they’re basing their claim on

They’re always going to try to present themselves favourably so that means we should be skeptical and ask questions about the gaps in the data and the methodology

Sure, I agree. But couldn't find much too doubt though.

Questioning 10, 100 vs 200 kb. seemed nitting to me, if I may say that.

I've seen a lot of bias against cloudflare since fastly's blaming and a lot seems exaggerated to me.

It's like they have to hold standards 20 x higher than the rest to make the same point. And tbh. They actually didn't earn the blame that happened then ( if I remember it correctly)

I'm sure fastly would be quick to point out any errors ( relevant or not) in benchmarking. And considering they didn't for a month and considering history, that also bears meaning.

Since they are the obvious losers here ( only faster in Liberia... ).

Additionally, I'm pretty sure eastdakota's statement "game on" and their unfortunate PR disaster ( they have gone much more silent since then), would mean they are double checking any benchmarking for errors.

At least, if I can guess cloudflare 's management correctly.

Note: fan of the firm + management. Got shares.

If I wouldn't think they are acting in good faith, I'd probably sell my shares.

See the previous discussion in 2021:

https://news.ycombinator.com/item?id=29465729

…in which Cloudflare's CEO directly address the first point (having a legacy clause in their Terms that used to prohibit benchmarking, clause they removed quickly after), and in which one of the main engineer behind Cloudflare Workers underline that there is "apple vs oranges" comparisons for a while from Fastly before that blogpost.

I really recommand reading those previous discussions, you'll learn a lot about the context of this blogpost and what Cloudflare think of it!

Fastly is the only official PyPI mirror, and I was once asked by a client to figure out why PyPI downloads weren't working, on a host that otherwise seemed to have good connectivity (downloads of similarly-sized files from non-Fastly sites worked just fine).

It turned out that Fastly ignores/blocks ICMP unreachable messages, which breaks Path MTU discovery[0]. The host was behind a VPN and I verified using Wireshark that it was asking Fastly to lower its MTU to accomodate the VPN packet overhead, but Fastly was ignoring the messages.

Setting MSS on the host's default route (`ip route change default advmss 1380`) fixed it, but it seems basic networking tips like "don't block all ICMP" should be table stakes for a CDN. I nicely emailed Fastly about this, figuring they might appreciate knowing they had something misconfigured, but never heard back.

Ironically, Cloudflare has a really good blog post about this issue[1], which I used when explaining the problem and solution to my client.

[0]: https://en.wikipedia.org/wiki/Path_MTU_Discovery

[1]: https://blog.cloudflare.com/path-mtu-discovery-in-practice/

Unfortunately you can't really rely on PMTU discovery these days. Not only is it regularly filtered, it slows every connection down since you add a RTT for the ICMP packets for every TCP host you talk to. In this case I would say the VPN client would be at fault for not adjusting the TCP MSS.

Relying on MSS to not break layer 3 is going to be interesting when QUIC is more prevalent. PMTUD should be better supported.

QUIC unfortunately defaults to 1370 bytes to avoid having to deal with PMTUD :(. Endpoints are free to negotiate higher and the protocol does provide a way to generate arbitrary sized ping frames, but it's likely most implementations will just stick with the default.

> using Wireshark that it was asking Fastly to lower its MTU to accomodate the VPN packet overhead, but Fastly was ignoring the messages

It could also have been a middlebox somewhere between the vpn egress and fastly causing issues.

Looks like it's still reproducible. I just tried artificially lowering the MTU on the interface of a bastion host in a datacenter, with only one router between it and a major IX (no middleboxes), and running apt-get against the Fastly Debian mirror:

  20:55:33.402893 IP 23.141.40.254.43004 > 151.101.10.132.443: Flags [.], ack 1, win 504, options [nop,nop,TS val 4088102127 ecr 3823759533,nop,nop,sack 1 {2889:3881}], length 0
  20:55:33.580057 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [.], seq 1:1445, ack 397, win 285, options [nop,nop,TS val 3823759807 ecr 4088102127], length 1444
  20:55:33.580314 IP 23.141.40.254 > 151.101.10.132: ICMP 23.141.40.254 unreachable - need to frag (mtu 1420), length 556
  20:55:34.155995 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [.], seq 1:1445, ack 397, win 285, options [nop,nop,TS val 3823760383 ecr 4088102127], length 1444
  20:55:34.156214 IP 23.141.40.254 > 151.101.10.132: ICMP 23.141.40.254 unreachable - need to frag (mtu 1420), length 556
  20:55:35.308019 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [.], seq 1:1445, ack 397, win 285, options [nop,nop,TS val 3823761535 ecr 4088102127], length 1444
  20:55:35.308275 IP 23.141.40.254 > 151.101.10.132: ICMP 23.141.40.254 unreachable - need to frag (mtu 1420), length 556
  20:55:37.739916 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [.], seq 1:1445, ack 397, win 285, options [nop,nop,TS val 3823763967 ecr 4088102127], length 1444
  20:55:37.740191 IP 23.141.40.254 > 151.101.10.132: ICMP 23.141.40.254 unreachable - need to frag (mtu 1420), length 556
  20:55:42.347790 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [.], seq 1:1445, ack 397, win 285, options [nop,nop,TS val 3823768575 ecr 4088102127], length 1444
  20:55:42.348049 IP 23.141.40.254 > 151.101.10.132: ICMP 23.141.40.254 unreachable - need to frag (mtu 1420), length 556
  20:55:49.539608 IP 23.141.40.254.43004 > 151.101.10.132.443: Flags [F.], seq 397, ack 1, win 504, options [nop,nop,TS val 4088118265 ecr 3823759533,nop,nop,sack 1 {2889:3881}], length 0
  20:55:49.659124 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [.], ack 398, win 285, options [nop,nop,TS val 3823775887 ecr 4088118265], length 0
  20:55:49.659189 IP 151.101.10.132.443 > 23.141.40.254.43004: Flags [F.], seq 3881, ack 398, win 285, options [nop,nop,TS val 3823775887 ecr 4088118265], length 0
  20:55:49.715144 IP 23.141.40.254.43004 > 151.101.10.132.443: Flags [R], seq 829588935, win 0, length 0

(FWIW, this thread nudged me to email noc@fastly.com again.)

If it weren't for Hacker News I never would have learned about this genre of blog post, where two for-profit companies talk shit about each other's products until they both look bad.

It's really entertaining.

Seems particularly common for edge related companies for some reason

It's kinda weird seeing a blunt response (criticism of their competitor) from a company, but I like it. Feels more genuine, I guess.

Given that they compared rust-wasm on a selection of a rather odd number of nodes (673- what went into that particular selection?) and the title of "lies", it just reads as someone at fastly is bitter.

All the article does (which they even admit, blaming it on "beta" status) is that, if you want to run javascript on an edge server, cloudflare is faster. On top of that, you can do it in a free tier, whereas fastly says their free offering is only meant for a short trial.

Wow, someone is salty. I note that Fastly doesn’t specify how they selected their catchpoint nodes either.

And they decided to compare Rust against JS because that somehow makes more sense? That’s bizarre.

Rust was the first language Fastly supported so their support is more mature, AssemblyScript was a later development

Not sure how many nodes Catchpoint have so 675 might be the natural limit

This should really get "(2021)" added to the title, especially since the title implies it being recent (when it is not).

Added

Thanks - I should have looked more carefully before posting

2021.

IDK - anecdotally I've used fastly side by side with cloudflare enterprise and I don't think they're stretching the truth too far saying that they generally perform better.

> Lies, damned lies: debunking Cloudflare’s recent performance tests

It's not recent any more. This blog post is a year and a half old.

Sorry, I should have looked more carefully at the article (the date is at the bottom for some reason?). It looks like an admin has updated the title to be more accurate now though, which is helpful.

In which year do we think we’ll get leaks confirming Cloudflare was an NSA operation?

Recent Cloudflare updates on performance (2023): https://blog.cloudflare.com/speed-week-2023-network-performa...

Cloudflare is now too big to fail. They are in the category of being able to get away doing horrible, biased, unethical shit and no one can do a thing because its now a battle of lawyers vs lawyers. They are not all bad, but its like saying Google does shady shit? Of course they do, but what are you going to do about it? Money is power, and power is everything.

Far from it. They are popular among the tech crowd for sure bur akamai and others are very popular too. Most sites don't use them either. Many if their products overlap with cloud providers' offerings too.

Im just saying they are big enough you arent going to fight them.. Look how loose their T/C are. They serve a purpose, but any liberty they decide to follow you are at their whim unless you have legitimate money to fight. And for 99% its not worth a 1 year fight and the amount of money. Just like all big tech, its largely un-realistically-regulated so they can do what they want.

Yea, like other providers (eg Twilio, Stripe) they can pick/choose who to allow as clients. Kick ClientA out because they did some vague, hand-wavy violation but allow ClientB (a direct competitor to A) continued use.

Its not just these mentioned, many corporate ToS give them too much power and it's insurmountable for individuals or small business to fight back.

Are there any competitive counter examples, or is this just par for the course? Not saying that makes it ok, but now I am curious.

dont use them if you dont like them, it is a free world

That's not the argument

(2021)

Reply on news.ycombinator.com