Hacker News Reader

Comment by tylermenezes | Hacker News Reader

As a user, Fastly is really, really good. Akamai is the only serious competitor, but they are a lot more traditional in their sales and configuration. (You can theoretically use Azure as a middleman as some have pointed out, but they don't support custom SSL, and configuration changes take literally hours-to-days to propagate.)

Cloudflare is fine but they are still an order of magnitude slower, which is why we switched. I just tested and Cloudflare is still taking 48ms their own DNS server to resolve our Cloudflare-hosted DNS, but only 14ms for Fastly to establish a connection and send the first byte.

That speed does make a difference. When we switched from Cloudflare to Fastly we had about a 7% increase in completed sales.

I think their ROI is still too low for most small businesses, the only way that actually makes sense for us is because it's free for open source/nonprofits. But I can imagine it's a big deal for larger ones.

View on HN

I highly doubt your claims about Cloudflare being an order of magnitude slower, they are consistently one of the fastest CDNs:

https://www.cdnperf.com/

Top 15:

    1 Google Cloud CDN        36.70 ms 
    2 jsDelivr CDN            36.80 ms 
    3 Akamai CDN              38.00 ms 
    4 Verizon (Edgecast) CDN  38.30 ms 
    5 Azure CDN               38.50 ms 
    6 Fastly CDN              41.38 ms 
    7 Cloudflare CDN          41.80 ms 
    8 AWS CloudFront CDN      43.00 ms 
    9 CacheFly                43.90 ms 
    10 BunnyCDN               46.00 ms 
    11 StackPath CDN          46.53 ms 
    12 KeyCDN                 47.00 ms 
    13 CDN.NET                48.46 ms 
    14 G-Core Labs CDN        48.74 ms 
    15 Quantil CDN            50.20 ms

Everything about what the top comment said seems like Fastly marketing propaganda.

7% increase in sales? With what sample size? With what confidence? How did you isolate variables?

For reference, I work at a company with $110M in annual sales. We were planning to start using Fastly. For obvious reasons, we wanted to know how much of a net positive that would be for the business. So we wanted to A/B test it. For us, at least, it's not as straight forward as it would seem.

You can Google my name if you think I'm working for Fastly. The only connection I have is that they are our CDN. They donate it, as we're a nonprofit, but I have zero obligation to say anything about them either way.

7% increase represents an increase in the number of students/parents purchasing tickets to our events. (It's not like they're spending more individually, but more people have been willing to give us money.)

That's based on several years with no substantial change in the way we market or the design of our site.

> 7% increase represents an increase in the number of students/parents purchasing tickets to our events. (It's not like they're spending more individually, but more people have been willing to give us money.)

We got that part, but correlation doesn't always imply causation. Maybe that change was seasonal or just plain old growth not accounted for in your 7%?

How do you know the 7% increase was due to Fastly and not some other small unrelated change to your web site? A few 10 ms makes no difference.

> A few 10 ms makes no difference.

About a 1-2% difference per 100 ms seems well supported (e.g. [1]), at least if your page load time is already low enough. 7% is very high, but some effect is expected.

1: https://developers.google.com/web/fundamentals/performance/w...

I don't buy it. So what you are saying is 2% for every 100ms. So that means if we decrease page load time by 1ms, then we should expect to to see .02% increase in sales?

So if we do $10m annually, 1ms decrease in sales should boost sales by $2,000!

We would all probably agree that 1ms will make no statistical difference.

The problem with these studies is that most are dealing with much longer load times. Like 3 seconds vs 19 seconds! Obviously, that will make a HUGE difference. You can't then extrapolate that down to the millisecond.

The other problem is that many of these studies are basing their numbers on average load times. So they are comparing two groups and averaging load time. Group A averages 100ms faster than group B. And group B increased sales by 2%.

But what really happened is group A had 800ms load times across the board. And group be had 800ms load times for 98% of their page loads and 20,000ms on the remaining 2%.

So working with averages can be largely misleading.

I can't see the details of the 1 study that claims 100ms increments, but I'm very skeptical.

Yes, the 1ms will likely be statistically insignificant. But so are $2000 extra income out of your $10m. If anything, you have a better hope measuring the 1ms than the $2000.

I agree with your larger point that average latency is not a good measure. Even for the perception of a single user consistency is more important than a good average. For large groups the average is even less useful.

But intuitively I see lots of places where 100ms makes a world of difference. Just like there's a big, very perceptible difference between a 200ms animation and a 300ms animation, a time to full render of 200ms can change your experience compared to 300ms. The slower the page load, the more deliberate your movement. The closer you get to 16ms (1 frame) for a page load the smaller the investment for clicking a link, and the higher the willingness to experiment and explore. Some of that inevitably leads to conversions and sales.

I’ve seen the data and it’s true, but non-linear. Going from 19s to 3s is going help much less than going from 3s to 1s. The curve is almost entirely in the under 3s part.

I agree it may make some difference... but I don't think it can all be attributed to a CDN change. Maybe they moved some buttons around the same time as the CDN update. Who knows.

CDNPerf is an okay way to get an overview, but not a great way to actually compare CDNs.

That said, it’s not 10x or whatever. Maybe like 5-10% depending on location.

There are a number of features Fastly has which others (excluding eg Akmai) don't which make head-to-head comparisons more difficult. I have no doubt that Cloudflare is very fast in perfect conditions, but there were a much higher number of cache misses because there's less flexible configuration which did literally make the difference between 20ms vs 200ms in production.

Can you elaborate on this?

Enterprise customers are generally able to get their SE to set whatever cache key they like, and if that is insufficient they can use the worker cache API: https://developers.cloudflare.com/workers/reference/cache-ap...

As I understand it a charity may get the former thing for free through https://www.cloudflare.com/galileo/ , but the latter probably still has usage-based billing.

Disclosure: I work for Cloudflare.

Comparing CDNs is incorrect by an aggregator. I couldn't find CDNPref methodology. One patient goes to two specialist doctors and returns with two different assessments (observer error). The multitude of variations in Cloudflare and Fastly alone, make it hard to compare. Free plans should be removed, and maybe just compare Enterprise to Enterprise on the same price point.

Do you know of a better way than CDNPerf? They do 300 million requests per day from random users from around the world.

Cedexis is also a good tool to measure, while most of it is a paid product, they have a few reports you can see. they also list cloud providers, and have other reports such as throughput, etc.

https://www.cedexis.com/get-the-data/country-report/

Is there any documentation on how they measure it? If they're running tests from data centers / peering points or testing things which are highly-likely to be cached the results are going to be harder to generalize to normal usage.

DISCLAIMER: Akamai Product Manager here.

Deployment times for config changes have been operating at the sub-5m timeframe for at least two years. It used to take hours (never days) to propagate changes across the server estate, but not anymore. And while we were admittedly late to the DevOps train, we have made up lost ground. We have nearly 100 individual APIs to control almost any aspect of our products. CLIs if you don't want to write to the API. A Sandbox to test config changes locally. The ability to validate OAuth tokens at the Edge, cache GraphQL responses, and throttle and/or quota API traffic on a global basis. Hashtag "legacy CDN."

> Deployment times for config changes have been operating at the sub-5m timeframe for at least two years.

Fastly is sub 5 seconds.

> And while we were admittedly late to the DevOps train, we have made up lost ground.

Not anywhere close. Your cache invalidation takes forever. Ability to assign tag objects does not exist. Engaging "professional services" to make a config change like it is 2002?

> Not anywhere close. Your cache invalidation takes forever. Ability to assign tag objects does not exist. Engaging "professional services" to make a config change like it is 2002?

Akamai also has a fast purge these days, sub 5 seconds as well I believe. Works nicely.

Fast purge only works on certain kind of objects in certain kind of configurations, not to mention the idea of "I would like tom make 20k purge requests in a second via an API" is met with stares.

Fastpurge is a hack.

Interesting. I only have experience with the occasional simple (manual) purge and I could verify the object was invalidated quickly. Can you elaborate on an example config where this goes awry?

Fast purge works with all kinds of site delivery ( and site delivery based ) products. Unlike modern CDNs Akamai other products ( such as for example VOD and media services ) do not live in the same object space and hence are not fast purge compatible.

You would think purging a video stream would be the same as purging a standard site delivery object, after all it the stream is http(s) accessible .m3u8 and a pile of .ts chunks but that's not the case -- in some cases it can take up to one hundred and twenty minutes.

I see, thanks for the info.

I'm not talking about Akamai directly, I'm talking about the Azure interface, which is as close to Akamai as most HN commenters are likely to get. Akamai itself is great but the Azure interface makes you (and VZN) look terrible

Disclaimer: I work at a company that is both an Akamai and a Fastly client. I worked with Fastly first, then Akamai after. Opinions are my own and not the views of my employer.

There has definitely been a push to catch up, but I wouldn't say you're there yet. The Terraform provider for instance is not up to par with the Fastly one. Having attended an Akamai DevOps workshop not so long ago, there didn't seem to be an easy way to use the CLI tools to configure a property in an idempotent way in a CI/CD pipeline. Maybe things are different now.

Akamai has some advantages for enterprise customers though such as easy assignment of cost using the CP codes. That's very handy. The integration with Let's Encrypt is also very nice.

Fastly delivers in it's simplicity and documentation. Whatever use case you need, you'll most likely find it in their docs and on their blog. Being able to use VCL to configure your caching is in my opinion way easier (and with less limitations) than the Akamai rule tree (even if you don't have previous Varnish experience). Varnish's finite state machine makes it easy to configure and debug any kind of behavior you want. The Akamai rule tree has caused me quite a few 'WTF' moments. It's difficult to debug when something doesn't behave as expected and there's a certain amount of 'black magic' in the behaviors sometimes that makes it difficult to judge what the outcome will be.

The good thing is that both companies have different features and get pushed to incorporate features deemed useful by their customers at one or the other provider.

What's the purge time like?

Less than 5s across all 240K+ global servers

I find it difficult to believe that sales would be affected by a latency drop less than one tenth of a second.

How did you isolate the variables?

No, this absolutely checks with my experience - I chase 5 and 10ms improvements all the time because we've measured and know it increases conversion.

But it makes sense, too: if the metric here is average latency, that doesn't mean that some users didn't see a much more dramatic increase. Every tiny bit of frustration removed from the experience adds up.

>No, this absolutely checks with my experience - I chase 5 and 10ms improvements all the time because we've measured and know it increases conversion.

You've measured and know that a user seeing a 905ms load time converts more than one seeing 915ms?

The latencies stack up for any web page. It's not like "oh this file was served 5ms sooner", it's an accumulation of latencies over all assets, and the interactions, that are required to present an experience to a user.

Also, latency measured at the server is amplified once its received by the browser. And when the user's connection isn't great, all this is worsened. It quickly adds up. In fact, it doesn't "adds" up, it "multiplies" up.

If you’re serving so many elements that 5-msec-per improvements really accumulate, your problem is page complexity, not marginal latency.

Most pages nowadays are spamming requests to backend services, for business logic or not, that's just reality. Barely any actual real world money making website is going to have < dozen requests per page, and won't need a user initiated page load every few seconds to minutes.

A 10ms average improvement could mean 1 in 1000 customers went from 10s to 1s without any other change to other customers.

This is easily possible if you have a highly distributed customer base, and/or some small segment of your customers don't have good upstream peering with your provider.

>A 10ms average improvement could mean 1 in 1000 customers went from 10s to 1s without any other change to other customers.

Which is why I asked the question in the way that I did. I buy that a slimmed down webpage loading 10 ms faster on average will increase conversions because that makes the site usable for the visitors on bad connections. Moving to a CDN doesn't have that impact. It shaves off 10-100ms across the board.

> Moving to a CDN doesn't have that impact. It shaves off 10-100ms across the board.

I think this is where we disagree. I've seen (firsthand and through analytics) situations where using a CDN can dramatically improve response time in a small subset of customers (while also getting the across the board win for most customers).

I've also seen CDNs (Amazon's in the early days) that were signficantly slower than direct to linode, even with a warm cache. It's a weird world, and packet routing is hard.

What kind of storage do you use when chasing the 5-10ms improvements? I ask as someone working on super fast storage and memory and it's not always easy to find people who are aware of the differences in latency. I imagine you're already on NVMe SSDs, but if not, I'd be curious why (i.e. is software more of the issue than hardware?).

I am going to assume there is more to it then that. Even Amazon only says 100ms increments affect sales.

There was a difference of usable site time of about 500-600ms

> That speed does make a difference. When we switched from Cloudflare to Fastly we had about a 7% increase in completed sales.

Interesting. Is that for the flow that starts at https://www.srnd.org/sponsor => https://www.srnd.org/sponsor/pay or somewhere else?

(Those pages do load very fast.)

hi- curious about your completed sales stats. What are you selling? was it instant? was it gross top line?

We sell low-cost tickets to events to largely non-technical students. On the order of 10k sales a year, so we're nowhere near enterprise scale (which is also why we couldn't use Akamai). www.srnd.org if you'd like to learn more.

Reply on news.ycombinator.com