Hacker News Reader

Comment by buro9 | Hacker News Reader

I'm doing something very similar to this, the setup I'm using is:

DNSMadeEasy has a global traffic redirector ( http://www.dnsmadeeasy.com/services/global-traffic-director/ )

That then sends a request to the closest Linode data center.

Linode instances run nginx which redirect to Varnish, and the Varnish backend is connected via VPN to the main app servers (based in the London datacenter as the vast majority of my users are in London).

I use Varnish behind nginx to additionally place a fast cache close to the edge to prevent unnecessary traffic over the VPN.

Example: USA to London traffic passes over the VPN running within Linode, and the SSL connection for an East Coast user is just going to Newark. If the requested file was for a recently requested (by some other user) static file, then the file would come from Varnish and the request would not even leave the Newark data center.

View on HN

I can't edit my post but I should note that there is an edge case I'm aware of where this kind of solution might not be the fastest solution for the end user, and this would likely affect what filepicker.io are doing too.

The edge case is that some DNS providers (Google, OpenDNS) already pick what they feel is the closest end point.

I read about that stuff over here a while ago: http://tech.slashdot.org/story/11/08/30/1635232/google-and-o...

And this comment explains it best: http://tech.slashdot.org/comments.pl?sid=2404950&cid=372...

I haven't fully investigated this, and I don't know whether it is affecting some users. But when I implemented my solution I was aware that it might be possible for some small subset of users, for this to not result in a faster connection than if I'd done nothing at all (the closest resolver to Google may actually be further from the customer than the local server I run).

I'm just betting that for the vast majority of users this does bring about a noticeable increase in speed.

Google runs lots of DNS servers. You (your ISP) pick the 8.8.8.8 closest to you. That will in turn do the lookup and get the linode closest to google dns, which should also be close to you.

If you're using a North American Google DNS server, you'll get answers that say NA. If you use the DNS server in Europe, you'll get answers that say EU.

I'm assuming Google doesn't try to sync and cache between 8.8.8.8 instances, but I don't see why they would. That's a lot of work for no benefit.

IIRC the google public DNS service has cache coherency intra location, but not inter.

From what I've seen end users hit a google dns cluster in the approximate geo area. However I e definitely seen odd peering of a public DNS node in EU hitting provider anycast nodes in NA.

That's pretty much it.

If a request via 8.8.8.8 surfaced at a DNS server in North America, and DNSMadeEasy (in my example) then answered that request with a "Oh, you must be in North America, well for you the IP address of the web site is"... then you might not have got the answer you expected.

i.e. You might be in Spain, and using OpenDNS (or Google) the DNS query against DNSMadeEasy might surface on the East Coast of the USA, and as such you'd end up at Linode Newark rather than Linode London.

That particular example is pure speculation, but it illustrates the point.

As I said, I believe that the amount this must happen is just a slight edge case and as a whole isn't worth troubling about. But it is there as an edge case I'm aware of.

And if someone reads this thread and thinks, "Hey, this distributing SSL stuff is a great idea.", then as always, caveat emptor and check whether any potential issues that might arise are an issue for you and your application.

That's a great idea. We're hosting with SoftLayer and have been considering doing something similar. They offer free bandwidth on the private network between data centers (and have pretty good pings between them). With a cloud server in each data center, you could achieve a similar thing while avoiding the need for VPN and not paying for extra bandwidth.

Is this roll your own CDN significantly cheaper than another provider? Or is there some other advantage?

Static files are still served by a normal CDN. This helps with dynamic HTTPS requests that change each time.

I was already on Linode and I'm only serving a few hundred GB of static files per day (with the Linodes I have this is well within my free quota).

In my instance (forums with current discussions) most static file requests are for image attachments in the very latest discussions, the hot topics. So Varnish fits this scenario really well. I didn't need a long-term storage of images in the CDN, I just needed to store the most recently requested items in the CDN.

Linodes are cheap, I was already using them in a distributed fashion to reduce SSL roundtrips, and introducing Varnish was a small configuration change.

I have tried a few other providers (most recently CloudFlare). But I was generally not happy with them, usually due to a lack of visibility.

I proxy http:// images within the user generated content over https:// when the sites are accessed over https:// . And occasionally I found that images would not load when I used a CDN provider for that. But never had enough data and transparency with the CDN to know why. Users notice this stuff though, so I'd have isolated users complaining of images not loading and no way to debug or reproduce it.

So I found that as my scenario made Varnish a good fit, and the bandwidth was within my allowance, and it was easy to do... well, I just did it.

I still experiment with CDNs every now and then, but largely I get more reliability and transparency from my own solution. I've also found this to be cost effective, though I would be OK with paying a premium if I found the reliability and transparency rivalled my home-rolled solution.

Reply on news.ycombinator.com