Hacker News Reader

Comment by simonw | Hacker News Reader

We got burned by CloudFront about 18 months ago... we were serving our static assets (CSS, JS etc) through CloudFront and had bug reports from some users in eastern europe (I forget where, it might have been Slovenia) that our site was displaying without CSS. I got them to check and they couldn't load CSS for GitHub (which used CloudFront) either. We went back to serving directly from S3.

It's an infuriating bug, because I can't see how we could confirm that this kind of thing isn't an issue any more. I'd love to go back to CloudFront but I'm just not confident that it will reach all of our users.

View on HN

Please feel free to send me some details (address is in my profile) and I'll pass it along to the team.

I've emailed you. For anyone else who's interested, here's the support email we got (back in January 2011 it turns out):

> The following URLs fail to load: > http://cdn.lanyrd.net/css/core.221dbc4b.min.css > http://cdn.lanyrd.net/js/jquery-1.4.3.min.97be02d1.min.js > http://cdn.lanyrd.net/js/jquery.jplayer.min.72d89d00.min.js > http://cdn.lanyrd.net/js/lang.ENG.4f594a71.min.js > http://cdn.lanyrd.net/js/global.f0851851.min.js > This is on the basic page - http://lanyrd.com/services/badges/. As far as I can tell, no files from the domain cdn.lanyrd.net will load. > > Also, it seems the Lanyrd.com site doesn't can't load any resources from the CDN domain as well - the homepage is totally broken for me. > > Oh, and I'm situated in Slovenia, if that helps.

I replied and asked them to run "host" and "ping" against cdn.lanyrd.net and they sent back the following:

> Host cdn.lanyrd.net not found: 3(NXDOMAIN) > ping:unknown host cdn.lanyrd.net

I also had an incident a few months later where our assets failed to load for a period for me sitting at my desk in London - GitHub's assets were affected as well, which lead me to suspect it was a CloudFront failure. Unfortunately I don't have any notes from that.

How do you know that wasn't your DNS provider having troubles there? Should have had them do `dig` to see if it was a DNS issue on your end instead of blaming Amazon right off the bat...

It could well have been (that's why I'm sharing the details: so people can make their own mind up). Like I said, this was over a year ago so it's pretty hard to debug-in-hindsight.

Starting with "We got burned by CloudFront..." seems a little harsh when the only piece of actual data you have could just as easily point at your own DNS provider rather than Amazon's systems...

I ran into a bug where cloudfront takes a long time to server some requests. Not an isolated problem.

This thread has reports from other users - https://forums.aws.amazon.com/thread.jspa?messageID=246418

I had the same issue in one of my websites

1. http://www.webpagetest.org/result/111114_HM_263JS/1/details/ 2. http://www.webpagetest.org/result/111114_2N_263HM/1/details/

You can see that cloudfront takes upto 5 seconds for some images. And these are files which were cached in cloudfront (X-Cache: Hit from cloudfront)

Because of this issue, I moved away from cloudfront.

On the flip side, we use CloudFront for > 5 billion requests/month and couldn't be happier.

Last time I checked, at that volume one could get much better pricing by committing to a yearly contract through a traditional CDN vendor.

We use S3 as our origin, so using CloudFront makes sense from an ease of use and fastest response perspective. Also, CloudFront offers reserved capacity pricing for yearly commitments above a certain bandwidth level.

I encountered these types of problems on Cloudfront-powered sites all the time when I lived in Colorado. I frequently had issues using GitHub, Basecamp, etc. Only solution was to wait a few minutes and try again.

> I'd love to go back to CloudFront but I'm just not confident that it will reach all of our users.

Why are you confident you will serving directly off S3?

Because with CloudFront there are dozens of origin servers around the world, and problems like the ones I experienced could be caused by a DNS server somewhere putting someone in touch with an unavailable server. S3 serves from one location (the location where you created the bucket) and hence is less likely to fail in the same way.

Yes, but if that one S3 location is having troubles, all of your users are affected, not just some of them as when CloudFront has trouble at a single location.

Did you get to the root cause of the problem? We are about to trial Cloudfront on one of our sites and have discussed the possibility of it causing problems for some users.

No I didn't, and I admit I didn't investigate very deeply (I chose to switch back to S3 and move on to other things).

Reply on news.ycombinator.com