Cloudflare Package Repository Mirrors | Hacker News Reader

Cloudflare Package Repository Mirrors

cloudflaremirrors.com

88 pointsFoxboron57 comments

As much as I'd love to use CloudFlare's uniformly fast network for package update, I'd never use this simply because I already depend on them for too much stuff.

It's not like letting them gather logs on my computers updating would be terrible somehow, but if it's so easy to avoid donating more logs and more dependence on them, why not do it for the sake of preserving diversity?

Growing companies often exhibit a long term pattern that usually seems innocent at first manifestation:

- Offer something for free

- Become hugely popular at offering that free thing

- Alternatives slowly dwindle in popularity and/or begin to die off

- User 'choice' increasingly becomes concentrated in that single option

- Company introduces surprising policy change, but users can switch to an alternative if they don't like it

- Company introduces differentiating functionality setting its option apart from the alternatives.

- Users grow additional dependence on the new functionality, alternatives look increasingly obsolete

- Company becomes increasingly aggressive at introducing changes, as they now own the primary alternative

It's hard to imagine CloudFlare are somehow going to monopolize package management :) But nonetheless, free services like this generally signal the beginning of this pattern

At some point I'm sure e.g. DejaNews seemed like an innocent dependency, and that attempting to look far ahead into the future would not have revealed its eventual sale to Google and the loss of the only comprehensive archive of the early Internet community in existence.

I think this is still a good service. It is just a mirror. It is a way of giving back to these open source operating systems and cutting their costs. Thus embrace it but be smart about it so that no one is trapped.

Dejanews isn't the same. That was an archive of a dying technology. This isn't the same case. Usenet was a dying technology, ultimately killed by http.

Cloudflaree has always done the free thing to raise it's popularity. You are right that as companies get bigger it can be harder to keep revenue growing and they slowly start to cross lines they didn't before though. If that company has built reliance on it's free tools which have reduced diversity, when they start to cross lines it can be sort of a trap. This usually isn't intentional now but it is just how things work... Years from now revenue growth might require crossing a line...

I'm guessing this may be more a case of "We and our customers need this, so we may as well make it public".

Companies sponsoring free mirrors has a long history without anything going wrong though. The key is to keep using mirror lists, not a mirror. Anyone who tries to "differentiate" should be removed from those lists. The difference with their DNS and CDN services is that all the information gets funneled through them instead of being one of many used in a round-robin manner.

>>>It's hard to imagine CloudFlare are somehow going to monopolize package management :)

10 years ago it would have been hard to imagine that Microsoft would own the largest open source code repository in the world.... but here we are.

I do not find it hard to image CloudFlare eating Package Management at all

You’re definitely right there. So much of what CF is doing feels like it’s going to end badly.

Another service I worry about in this regard is let's encrypt. They passed 50% marketshare 2 years ago. And many systems have special casing support only for let's encrypt.

Is anyone from CloudFlare or Canonical reading?

Please parter to make mirrors for Launchpad PPAs. They're so painfully slow around the globe and there is tons of open source projects that depend on PPAs...

Second this. Cloudflare package repo gets me so excited about it and i wish they extend to ubuntu too.

I recently moved to Vietnam, and I sometimes get awfully slow connection speeds to the mirrors nearby my city, to a point that switching to US mirrors is faster. (ftp.[kr|sg].debian.org)

How is Docker hub from Vietnam nowadays? I lived in Cambodia from 2014-2016, and it was nearly impossible to pull Docker images which have large single layers.

Would be great to have a Cloudflare solution for this.

Off-topic. I visited Cambodia for 3 weeks in 2017. It's a beautiful country and I enjoyed every minute. My mind has pondered over living there long-term -- it seems so affordable.

How was your experience living there for a while, especially as a developer?

I surely miss the developer/tech atmosphere that was there in Berlin, Frankfurt, etc.

The life is quite simple, and the coffee culture is based around chatting with friends for hours, instead of sitting in a cafe and getting work done on a MacBook Pro or a Thinkpad X1 carbon.

If you live like a local, which is not to say any lower living standards, it can be pretty cheap. At around 2x the living cost of a local, one can live comfortably here. I pay around third the rent which was once near Berlin, and now spend about half on food if I eat out.

Internet is cheap but unreliable. I sometimes have to use my 4G mobile hotspot (1GB/day at ~€4 a month).

I was born and raised in an Asian country, so there isn't much of a culture difference to speak of. However, Vietnamese is pretty difficult to learn, and even after six months here, I still cannot get the tones right and speak more than 50 words. Indonesia in that regard is amazing.

If you are thinking to move to south East Asia, I can highly recommend Indonesia. They have countless mountains to climb, a good hiking/riding groups to easily join, similar living cost to Vietnam, and a much easier language to learn. Bahasa Indonesia is understood pretty much everywhere, and quite easy to learn. Internet is censored in both countries to some extent (Reddit doesn't work in Indonesia with a DNS blackhole that you can easily bypass), and has worse Internet in Indonesia in general though.

There are many startups based in Indonesia (such as Gojek, Traveloka, Tiket.com), some of which are owned by SoftBank a few layers up. I feel it's more easier to I best/startup in Indonesia too.

Where exactly in Indonesia have you lived? Bali is like two times more expensive compared to Vietnam.

Also internet connection was generally much worse for me in Indonesia due to their island nature I guess. In Vietnam it's super easy to setup two 50-100Mbps FTTB lines for something like $80 / month (just in case one of ISP go down).

Yogyakarta, Malang, and Bandung (all on Java).

It seemed to me that Siem Reap was the logical place for expat coworking spaces but the internet there was really bad back in 2008. Not sure how things have changed.

I haven't lived in Cambodia, but what's I dislike about it is that it's feels generally like it more poor than Vietnam in quite ugly way: there are more people begging on the streets; including children. Might be something changed in recent years, but I have no idea.

In Vietnam / Thailand you just generally don't see this except in few tourist areas that you can easily avoid.

As the other commenter hinted, Siem Reap is a great place for expats, and that’s where I lived as well. The population is much more poor than the surrounding countries, but they are all so incredibly friendly and happy despite all this. You do need to make an effort really going into the country side to see the “real” Cambodia, I strongly advise against using only the cities as your reference, as they are generally very “tourist/expat savvy”, by lack of a different definition.

I did freelance work for a few European businesses, mostly C++ and devops stuff, and for a long time I felt I was living the life. What drew me back, however, was ambition: before taking the sabbatical, I had successfully founded and “exited” (more like an acqui-hire with some pocket money) a startup, and I was craving for more. It was very difficult to find people that I could consider my peers in Siem Reap, and this left some void. This may also have to do with my nature as an introvert, they may have been out there, but most developers out there actually are web developers, which isn’t my cup of tea.

All in all, however, the whole experience of being completely detached from my life back home for an extended period of time has made me a much more stable person, and I would do it again in a blink of an eye. As crazy as it sounds, I can now better appreciate the “rat race” and its good parts, and know that I also miss being challenged in that area.

Feel free to reach out to me directly if you feel like it.

I was actually in the Puok district outside Siem Reap for the 3 weeks. Mostly stayed in the rural side living with locals. I also visited the city for the weekends and it seemed like a great place to settle.

I also don't tend to go out of my way to meet peers in my local city. I guess I just prefer larger conferences for networking.

Thanks for the input! Maybe one day I'll also get acqui-hired and move there :)

Internet I general seems to be pretty bad here in Hanoi. I do not really work with bigger images (~100mb alpine images tops), and it takes ~5-10 minutes in evenings which appears to be the peak time on my current VNPT 40mbps line.

I also live in Vietnam for 3-6 months every year so it's part of the reason I posted it. PPA is real pain since as far as I understand they're not mirrored at all or at least they haven't been before.

In my experience mirrors in Japan work best in Vietnam for whatever reason.

I ran rankmirrors (from pacman-contrib) on the mirror and the 5 upstream i got on https://cloudflaremirrors.com/archlinux

rankmirrors test the time to download core.db from each mirror. Basically this:

  curl -s -m 1 -w "%{time_total} %{http_code}" -o/dev/null https://cloudflaremirrors.com/archlinux/core/os/x86_64/core.db.tar.gz

The results vary a lot sometimes cloudflare mirrors is the fastest, sometimes it's the slowest. Getting anything between 0.07 and 0.30

  $ rankmirrors -t ./ml
  Querying servers. This may take some time...
     *   *   *   *   *   *
   Servers sorted by time (seconds):
  https://ftp.lysator.liu.se/pub/archlinux/$repo/os/$arch : 0.113249
  https://ftp.acc.umu.se/mirror/archlinux/$repo/os/$arch : 0.120371
  https://mirror.osbeck.com/archlinux/$repo/os/$arch : 0.127601
  https://archlinux.dynamict.se/$repo/os/$arch : 0.180493
  https://ftp.myrveln.se/pub/linux/archlinux/$repo/os/$arch : 0.248406
  https://cloudflaremirrors.com/archlinux/$repo/os/$arch : 0.302824
  
  $ rankmirrors -t ./ml
  Querying servers. This may take some time...
     *   *   *   *   *   *
   Servers sorted by time (seconds):
  https://cloudflaremirrors.com/archlinux/$repo/os/$arch : 0.070069
  https://mirror.osbeck.com/archlinux/$repo/os/$arch : 0.081506
  https://ftp.lysator.liu.se/pub/archlinux/$repo/os/$arch : 0.098763
  https://archlinux.dynamict.se/$repo/os/$arch : 0.125846
  https://ftp.acc.umu.se/mirror/archlinux/$repo/os/$arch : 0.127948
  https://ftp.myrveln.se/pub/linux/archlinux/$repo/os/$arch : 0.136529

How is this supposed to work? (I use Arch BTW) supposed I'm in a country X, does Cloudflare basically caching the request to server that are in country X? But I'm already close to that server and I'm always cautious with mirrors that are in country X, often they were out of date :(

> How is this supposed to work? (I use Arch BTW) supposed I'm in a country X, does Cloudflare basically caching the request to server that are in country X? But I'm already close to that server and I'm always cautious with mirrors that are in country X, often they were out of date :(

According to what I read in their main page https://cloudflaremirrors.com/archlinux, they seem to be caching the packages from mirrors and serving it via their anycast network.

This is my issue with it as well - it's basically their geo-sniffing magic routing you towards the closest officially listed mirrors for that country (glorified proxy); it makes the mistake of presuming that all those mirrors listed in the Arch country-mirrorlist are created equal in terms of reliability and performance, but they're not (we have a whole web page just to track out of date mirrors).

"geo-sniffing magic routing", Also known as BGP + CloudFlare Argo (https://www.cloudflare.com/learning/performance/routing-vs-s...)

Yeah Arch mirrors in particular seem all over the place on quality. In true Arch spirit you def want to configure that yourself

I'd guess if this gains traction, the caching on Cloudflare's side will usually make up for this, and if it hits the actual mirror behind it, the load on that one should be lower than before. It reads like they will try to proxy multiple local mirrors so time will tell how well they will pick amongst them. I'm switching for now to see how it will perform over a longer period.

The difficulty is if the mirror itself is behind, Cloudflare's cache for that specific endpoint will be correct (functional, working) but out of date compared to another mirror.

Example: around the end of June this year the mirrors.kernel.org infra was failing to sync (DNS issues of some sort) the latest content, so your upgrade would appear to have no new packages because the stale content was a valid repo, when in fact there were many updates and other mirrors were in sync.

Right that makes sense, so Cloudflare would actually have to make the caching context-aware, not just classic HTTP proxying.

I'm not aware of the technical how, but the Arch infrastructure seems to measure this successfully; a hook into this data might be a great way for the two to interoperate (so that Cloudflare always serves the freshest cache/mirrors). https://www.archlinux.org/mirrors/status/

The relevant code can be found here I believe: https://github.com/archlinux/archweb/blob/master/mirrors/man...

Repeatedly poke the maintainers of the mirror and the core developers alternately until everyone gets mad and does something about it. :P

Alternatively (/ realistically), perhaps CF would prefer the value-add of rerouting where there are newer packages instead of the singular approach of geo-routing. Definitely let them know.

I'm seeing multiple comments in this subthread, I'd apply the 2nd line above to everyone affected, if a bunch of people let CF know maybe they'll figure something out.

Is it me, or it downloads so fast that the progress output doesn't even show in the terminal ? On Archlinux.

I think the header Content-Length is missing.

range requests to resume downloads do not work either...

but when encountered i can see the progress of downloading that package with the failed resume attempt... quite weird behavior

I saw the same. My best initial guess is that this is somehow caused by cloudflare either setting the content-type header to text/plain or not setting the accept-ranges header. Or maybe both?

> not setting the accept-ranges header

Mostly that, because it doesn't support resuming partial downloads either (it says http server does not support byte ranges)

Oh ! totally makes sense, nice guess. and very interesting

I can't get it to work, packages upgrades show this error. Any ideas?

  error: failed retrieving file 'XXX' from cloudflaremirrors.com : HTTP server doesn't seem to support byte ranges. Cannot resume.

Clear your partial downloads using sudo pacman -Scc, another comment says that it doesn't support byte ranges so it can neither display progress, nor resume partial downloads

It would be really nice if web proxies supported this as a generic mechanism, which could be applied to Linux packages, ISOs, git commits, docker layers and more. I wrote up a very half-baked proposal for this kind of thing a while back: https://rwmj.wordpress.com/2013/09/09/half-baked-idea-conten...

Then you could set cloudflare as a proxy (or transparent proxy even?) and it could inject the objects without having to visit the endpoints.

Ideally soon repos will be mirrored to ipfs nodes and that will automatically pull the hash-addressed content from the closest proxy, which may or may not be cloudflare. The protocol is already there. Some initial solutions are already there too: https://github.com/victorb/arch-mirror

Another set of mirrors with decent network infrastructure can be found at: http://mirror.facebook.net/

(Not holding my breath for the kind of responses this comment might get ;))

Need Ubuntu and Raspian.

When I saw the title I though they were proxying npmjs or docker hub.

The install process of editing a file is a bit painful. Means few people will do it. Somehow would be nice to have this offered in the is as a built in optional feature.

Also Fedora and Centos. Please.

I hope this is expanded to other package repositories (Maven, Docker, npm, etc). I'd really like to make it easier for people at edge locations to build my code.

Now every time cloudflare has a hickup, not only is half the internet unreachable, you can't even update your server... nice /s

At least on Arch Linux you can specify multiple servers, ensuring you don't depend on a single one.

I know, but lets be real for a moment: Cloudflaremirror will probably be the fastest. This will cause people to configure it and because people are generally lazy they will not configure other mirrors.

There are also other mirrors on cloudflare-ipfs.com.

If you are interested in other mirrors, perhaps you should IPFS pin some yourself. :)

Been looking for something like this for a while. We see if more repos get added soon (Ubuntu and fedora would be nice!)

Before opening the page I hoped that it would be an npm mirror which can be used in case npm goes down.

I don't understand, is Cloudflare hosting their own mirrors or what?

I understand they just proxy (and cache). They are still requesting packages from the actual repository if they don’t have it cached.

But I agree — don’t really see the point. Why not just become an official mirror?

They have Cloudflare Workers (with limited disk space), but act as a caching proxy is still the main business of Cloudflare. So i think acting as a proxy make sense for Cloudflare.

Being a proxy and cache is their business. All it takes to set things up that way is then a little bit of configuration that points to the actual origin.

I think if they would be an official mirror there would be a fair amount of more work being involved. Eg you can’t just randomly wipe the cache anymore and need to have higher persistence guarantees, since you are not supposed to refetch from other mirrors all the time.

Reply on news.ycombinator.com