Cloudflare’s CDNJS vs. Google Hosted Libraries | Hacker News Reader

Cloudflare’s CDNJS vs. Google Hosted Libraries

baldnerd.com

38 pointsxxdesmus22 comments

I'm not sure what using wget is supposed to show. Using it 25 times in a row on a completely stable, completely idle 25mbps connection, I get speeds/times that vary by 100% consistently

(IE min is 496k/s, max is 895k/s, average is about 600k/s)

Using wget is a completely useless benchmark, from what I can tell. Using apache's little benchmark tool I get about 255ms vs 180ms average for 100 requests to each.

The interesting part is that for ajax.googleapis.com I get:

  Connection Times (ms)
                min  mean[+/-sd] median   max
  Connect:      175  228  28.2    227     306
  Processing:     0   12  15.1      8      90
  Waiting:        0    0   0.0      0       0
  Total:        189  240  29.8    235     350

and for cloudflare I get:

  Connection Times (ms)
                min  mean[+/-sd] median   max
  Connect:       19   28  13.1     25     125
  Processing:   125  155  28.0    146     246
  Waiting:       21   28   6.5     27      65
  Total:        144  182  30.7    175     271

IE for google, all the time is in actually getting a connection and getting the bits, whereas, for cloudflare, there is actually some time waiting for their servers.

Using 5 concurrent requests actually gives me a massive advantage for google (cloudflare takes roughly the same time, google goes 4 times faster)

Could you post the exact commands you used? That high connection time for google looks like SSL setup.

Non concurrent:

ab -n 100 http://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min...

ab -n 100 http://cdnjs.cloudflare.com/ajax/libs/jquery/1.9.1/jquery.mi...

Concurrent case:

ab -c 5 -n 100 http://cdnjs.cloudflare.com/ajax/libs/jquery/1.9.1/jquery.mi...

ab -c 5 -n 100 http://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min...

Interesting results, but let's think about a few things before we all switch to Cloudflare CDNJS.

1) Time to first byte. In my experience the biggest lag is the connection, not actually downloading the file (for small js, images, etc...) I don't know how to test this.

2) Caching, by using google for common libraries like jQuery your chance of the end client already having a local cached copy are much greater.

3) Reliability. I know google has been recently killing off it's products, and even I tweeted how they could break the internet by shutting off their hosted library API, but it's probably not going to happen. Can you say the same about cloudflare?

I'm not saying I like one more than the other, but these are some things I would like to address before switching my and my clients sites over.

One thing I will give a +1 to Cloudflare is the ability to add js to the CDN via github. https://github.com/cdnjs/cdnjs

Using wget for a test is just terrible. Here's a better test (and I'll put it here, so everyone can criticize it):

  1) start chrome in incognito mode.
  2) press f12, and then click on network
  3) load each js

I'm in Denver, this is what I got:

  Google:
  DNS Lookup: 14ms
  Connecting: 24ms
  Sending: 0
  Waiting: 46ms
  Receiving: 42ms
  TOTAL: 128ms (86ms latency)

  Cloudflare:
  DNS Lookup: 15ms
  Connecting: 43ms
  Sending: 0
  Waiting: 46ms
  Receiving: 49ms
  TOTAL: 154ms (105ms latency)

I can't take a test run with wget seriously when there are better tools so easily available. It's beyond lazy.. it's incompetence, or worse, dishonest. He couldn't even be bothered to run it more than once from his own computer(!)

Doesn't this have network effects? The CDN used by the most sites becomes the most valuable since a user visiting your site will have the library cached.

I've always felt the benefit was that loading assets could be a networking no-op.

You're absolutely right.

It probably varies enormously depending on your location.

Here in Vancouver, I get 1.25MB/s for Google's CDN and 1.15MB/s for CloudFare. Unless we have a distributed tool to test that on a massive scale, it will only reflect the greatly-varying differences between people's connections, "distance to CDN", DNS metrics and whatnot which I guess greatly impact the speeds observed.

Nevertheless I really like that this actually "exercises" the different services and provides some form of measuring and poking that improves the overall transparency and accountability.

Another point to make is server response time. I'm not making a claim one is better overall than the other, but at my location (boston) cloudflare's servers respond a bit faster. Just grabbing a couple different files between the two shows up to 300ms difference, but usually <100ms.

[edit] oh yeah, and this is all moot if your user already has it in cache, which is currently more likely from google.

Disclaimer: I'm not offering a perfect solution to the ideal test.

1.) Depends on how everyone is routing. The people replying directly to the OP's page are checking from different servers with god-knows-what kind of routing tables that the average web browser on the net does not have access to.

1b.) A better test would be to have people from different ISP's and different geographic locations testing this out. Routing is everything. In other words, let's see what results are seen by mom-and-pop from India, UK, Texas, Rhode Island, Montana, Nunavut, Brazil, China, and Nigeria.

2.) Already said, but worth pounding into the OP's head, Google is the most widely used for jQuery because everyone else is using it and harddrive cache is 1.21 jiggawatts faster than any networked CDN.

3.) Latency and overall speed need to be measured. What use is 88Mbps if you've got horrible latency? Sure, there are times when a courier on a bike is faster than FTP, but for small files that latency is a bigger deal.

That is impressive, but you've basically just checked whether cloudfare has a POP in Ontario, or atleast within 2-3ms of your datacenter.

They both have POPs local to me:

   eggplant:~ tsl$ wget http://cdnjs.cloudflare.com/ajax/libs/jquery/1.9.1/jquery.min.js
   --2013-03-22 19:29:21--  http://cdnjs.cloudflare.com/ajax/libs/jquery/1.9.1/jquery.min.js
   Resolving cdnjs.cloudflare.com... 141.101.123.8, 190.93.240.8, 190.93.241.8, ...
   Connecting to cdnjs.cloudflare.com|141.101.123.8|:80... connected.
   HTTP request sent, awaiting response... 200 OK
   Length: unspecified [application/x-javascript]
   Saving to: ‘jquery.min.js’
       [ <=>                                   ] 92,629      --.-K/s   in 0.04s
   2013-03-22 19:29:21 (2.03 MB/s) - ‘jquery.min.js’ saved [92629]

   eggplant:~ tsl$ wget http://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js
   --2013-03-22 19:42:05--  http://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js
   Resolving ajax.googleapis.com... 173.194.75.95, 2607:f8b0:400c:c01::5f
   Connecting to ajax.googleapis.com|173.194.75.95|:80... connected.
   HTTP request sent, awaiting response... 200 OK
   Length: unspecified [text/javascript]
   Saving to: ‘jquery.min.js.1’
       [ <=>                                   ] 92,629      --.-K/s   in 0.05s
   2013-03-22 19:42:05 (1.62 MB/s) - ‘jquery.min.js.1’ saved [92629]

Just to drive home how much locality matters:

   tsl@beast:/volumes/fast$ wget http://nas.***.com/jquery.min.js
   --20:00:05--  http://nas.***.com/jquery.min.js
           => `jquery.min.js'
   Resolving nas.***.com... 10.10.10.2, fd2b:2048:4633:10::2
   Connecting to nas.***.com|10.10.10.2|:80... connected.
   HTTP request sent, awaiting response... 200 OK
   Length: 92,629 (90K) [application/javascript]
   100%[====================================>] 92,629        --.--K/s
   20:00:05 (63.00 MB/s) - `jquery.min.js' saved [92629/92629]

Looks like my single core ARM6 nas is the best in my neighborhood.

Never seen that before - what is nas.*.com?

He censored the domain name with * :-)

> my single core ARM6

You need to double-check your methodology:

http://sworddance.com/blog/2013/03/22/always-check-twice/

The difference is almost certainly an initial poor DNS resolution the first time for which ever service had the slower time.

That said, a more complete js cdn is always nice.

I ran the test, did it five times for Google, five times for Cloudflare and for each and every single one Google won out (for the current location I am at ... work), the script I used is here:

https://gist.github.com/bertjwregeer/5225202

This way it will also output how long it spent in each step, including connect, appconnect (if you try the SSL version), how long it spends in pretransfer and how long it takes to start the transfer at all. Followed by the size of the download and the header size.

This way you can get a much better idea as to why it is taking longer for one service over the other. The files themselves are relatively small so are not an accurate indicator of transfer speed.

For small files like this the latency is much more important than throughput, not to mention more users will have Google's version cached already.

Indeed. Past 5 Mbps, bandwidth is basically irrelevant, and Google knows it.[1]

[1]: https://docs.google.com/a/chromium.org/viewer?a=v&pid=si...

Don’t forget that one of the advantages of CDNs is that the user may already have the file cached in their browser. This is why I use Google’s CDN for jQuery and CDNJS for everything else – more web sites use Google’s CDN for jQuery, so the user is more likely to have that cached. Not to mention the HTTP limitation of 3 simultaneous requests from one domain…

the point isn't how fast your CDN is, the point is how likely it is that the user already has the asset in browser cache.

On that note: http://www.stevesouders.com/blog/2013/03/18/http-archive-jqu...

network effect of caching aside, I decided to run tests from 95 nodes around the world. Most of these have above average connectivity, however some are true "last-mile" nodes. The following data set consists of just over 2000 runs over one hour of testing.

                  #tests avgTotal  avgDNS    avgPingRT
  CDNJS             1105  187ms    58ms      27ms
  GoogleHostedAPIs  1120  219ms    30ms      38ms

https://gist.github.com/dneufeld/5226469

Reply on news.ycombinator.com