Making the Switch from Amazon Cloudfront to Fastly | Hacker News Reader

Making the Switch from Amazon Cloudfront to Fastly

blog.firebase.com

35 pointsmayop10031 comments

Wow, Firebase needs a new static asset management system. The fact that entire set of static assets have to be purged and can't be served with a 1YR cache timeout is a big red flag. Independent file versioning, whether through a version number or hashing, should be put into play.

That sounds like an interesting approach. We push out breaking changes to our server and client libraries several times a week. Since we invalidate our CDN on each deploy and set headers indicate as such, we can quickly and seamlessly migrate our end users without having them modify their <script> include.

At present, our library is unlike other mature javascript libraries that can make use of a very high cache timeout. How would the 1yr cache timeout work for the library that is changing several times a week?

I think he might be referring to a 1yr cache timeout on things like the logo.

Steve Souders blogged about how to serve scripts with far future expires without updating the client's script tag [1]. This technique initially loads the (potentially stale) js from the client's cache and checks for updates in the background, updating the client's cache if a new version is detected.

[1] http://www.stevesouders.com/blog/2012/05/22/self-updating-sc...

Yes, this is a great approach for javascript code. Usually, like in this case, there is something that will be faster than a round trip to check for if-modified-since or the ETag, which I would assume is used to do the invalidation in this case. Of course, YMMV, and if fastly is close enough to the edge and there aren't very many requests, this kind of optimization won't be required.

There might also be an opportunity to modify what is served from the customer's sites, depending on how all that is architected.

Keep in mind that we can't afford to have a constantly-changing resource name, as 3rd-party developers link directly to our assets. For our own website, this might work fine, but even then it's a pain and doesn't work for all assets (such as for your index.html pages).

Make the main asset that your 3rd-party devs load a loader, and not the lib itself. Teach the loader how to find out where the latest version is, and how to load it. Then teach it to sing, and, preferably, to dance.

We use CloudFront at work. It works well for us. We do a couple things differently though:

1. Instead of trying to purge all old files and upload new ones, we use cache-busting hashes in our static file names. This solves the stale-cache problem even for browser caches, not just CDN edge servers. Luckily, Django makes it easy to do this (https://docs.djangoproject.com/en/dev/ref/contrib/staticfile...). 2. Where we care about gzipping (CSS and JS primarily), we use our own server for the origin instead of S3.

Is there a good reason you don't append content hashes to compiled asset names (images, stylesheets, js)? That would remove the need to do cache invalidations manually, and comes out of the box with web frameworks like Rails.

Also, CF supports custom origins so you can point it to your web application and save all that screwing around with S3.

The main reason we can't easily use hashes to bust the cache is that the includes for our static assets (the Firebase JS lib) are happening inside our customer's websites. We could hypothetically insert a shim that pulls the include down the hash appended resource, but could still have the same problem with the shim itself.

The custom origin has alleviated many of our pains but wouldn't have taken us all the way. As people have begun to use Firebase to power Facebook apps (mandatory SSL for the canvas) and other SSL only environments, we needed the CDN to support it. We had a stop gap for some of our user (using Amazon's domains) but it would've added too much friction in the docs to explain the domain swaps for justify keeping this hack in place.

The primary feature advertised on fastly's website is a feature every real CDN (as in, "not CloudFront") offers: an API to immediately purge your content. It is not at all clear what differentiates them.

Looking at their feature plan breakdown doesn't help with this either: none of the CDNs I've used have limited the number of backend domains I've had, and all offered 24/7.

To be clear: that is not because I'm already paying the same $5k/mo minimum: my bandwidth costs have almost always been much lower than that over the last four years I've used CDNs; this is just not a common restriction.

Whether you are dealing with Akamai, CDNetworks, or EdgeCast, every level in the CDN world offers this feature. I believe CacheFly even had this mechanism. (Looking into t, I remember correctly.)

Meanwhile, their bandwidth pricing is insane (albeitimilar to CloudFront): their $/GB is a few times what I'm paying for a "real CDN", and is about what you will get if you call Akamai and then don't negotiate.

The real question is: how many points of presence do they have? CDNetworks has over a hundred, and Akamai has over a thousand. Are we talking "even smaller than CloudFlare" here? (Apparently, the answer is "yes: even smaller, they only 7".)

Alright. Let's go point-by-point.

The primary feature advertised on fastly's website is a feature every real CDN (as in, "not CloudFront") offers: an API to immediately purge your content.

Every CDN offers a mechanism to purge content, but they are not immediate. Edgecast takes up to 15 minutes, CDNetworks I've seen take 20, Cloudfront can take as much as 30. When we say immediate, we mean really immediate. Generally speaking, it takes about 150 milliseconds.

Obviously, we will negotiate as well when we're talking about significant amounts of traffic. And good luck getting Akamai to call you back if you don't have significant amounts of traffic.

Yep. That's true. We're a rather young company and are actively expanding. However, what is most notable about this is that despite having far fewer pops, we're still significantly faster than most other CDNs, especially in major population centers. We've put a ton of work into reducing latency inside our servers so as to make better use of the pops that we currently have.

And good luck getting Akamai to call you back if you don't have significant amounts of traffic.

Look, I realize this notion sounds right and fits with the common dogma about Akamai, but it is actually a lie: I have personally had long conversations with Akamai negotiating deals where I would have had no minimum commitment (although there were other totally reasonable non-monetary concessions involved), and their prices still beat the ones on your website (albeit only by a sliver).

(By the way, I am going to explicitly point out that if you had stopped after your first paragraph about how your API is different, I would now have just apologized and been interested to learn more about why people needed that, but this obvious and totally incorrect FUD about peoples' abilities to negotiate workable deals with Akamai is really bothering me. I wasn't actually "anti-fastly" before: I just found it expensive and confusing... but now?)

Also, that "2 billion requests per day" might sound silly to someone who serves that in an hour...

We like to think that the exact number of requests is less important than exactly how they're handled. While it would be cool to go "we serve a billion requests a second", we're still an early stage startup. We're spending more time making our responses even faster (< 1ms on the 99th percentile) and trying to provide things that no one else does (for instance, instant purging and surrogate key purging).

I must say - that kind of perf is superb, though top percentile and averages aren't super representative of the average customer's experience. What are your tp50 and tp90 like?

Your sub-150ms invalidation is equally if not more impressive, especially if you're talking about multi-region invalidation.

TTFB at the 50th hovers around 175 microseconds, 75th is at 250 microseconds, 95th around 450 microseconds.

As for the purging stuff, I do mean cross-region. So, it depends upon which node receives your purge request. 150ms is average, but really it's "network latency plus a millisecond or so".

Wait, so 95% of your customers experience a TTFB of less than 500 microseconds (tp95 of 450 microseconds)? I just want to make sure I'm understanding you correctly. Because that's awesome.

95% of requests. But yeah! And thank you. :)

I'm sorry that my statement is bothering you. I'm going based on numerous conversations with people considering using Fastly. It's quite possible that I have a skewed sample, however I'm not intentionally spreading FUD, for what that's worth.

See: http://en.wikipedia.org/wiki/Hanlons_razor

Just FYI, Cotendo has real time purging (and has for several years).

Also, your Asian/Australian pricing is ridiculous.

On Pricing, I must be missing something because All EdgeCast, CDNetwork and Akamai Resellers are similarly priced. And unless you get to lots of traffics for the majority these Big CDN player wont even deal with us directly.

Although with only 7 PoPs i could understand why you think it is overpriced.

1$ per gb for ASIA, only singapore and japan pop, simply way too expensive to be considered an alternative cdn.

I agree with you (and $1 per GB in AU? Really?) but they are just starting out. Maybe to American/European companies that don't care about the rest of the world, fastly would be ok. :)

That said, you can get much better (and just as fast) deals from EdgeCast and (previously) Cotendo.

Actually, tests from us and several of our customers have shown us significantly faster than Edgecast throughout the US and Europe. (I work at Fastly.)

CloudFront's "Origin Pull" method seems to do almost everything you are looking for with almost zero setup and management. No manual uploads and it matches all of your HTTP headers from the origin (eg, cache control and encoding). I've been using it for a few years and I love it.

Plus, CloudBerry Explorer PRO makes everything even easier and well worth the $40 (http://goo.gl/SM8dk).

Seems like their problem was that they used S3 behind CloudFront. Do they not know that Cloudfront supports custom origins?

Yep, that was precisely our problem :) We wound up using a custom origin with Fastly. We made the switch off of Cloudfront since they don't support custom SSL domains and the Cloudfront dashboard left a lot to be desired.

Also, the invalidation on Cloudfront is pretty painful.

Using S3 as an origin isn't a terrible idea, especially if you want very high availability for a low ops overhead.

We use S3 behind Edgecast, and it works great.

We also have a standby server and CDN in case we need to failover at the DNS level in the unlikely case either of those go down. This may be overkill for other sites, who probably should spend time planning for far more likely interruptions to availability (EC2 interruptions, etc).

CloudFront invalidation is definitely a pain, since they have no global purge or prefix purge (/bar/foo/*). CloudFront also won't handle Content-Encoding negotiation for you, though it will Vary on it if you ask it to. We used CF in the past, and had a bunch of machinery to purge just those assets that changed.

S3 is wonderful, but CloudFront is missing a few bits, despite having a great price at a low volume.

What about cloudfront's dynamic buckets? Those seem to work great for us so far.

Reply on news.ycombinator.com