Amazon Cloud Traffic Is Suffocating Fedora's Mirrors
phoronix.com
phoronix.com
Just stop support brand's and corporation's if you don't get paid?
> The massive uptick in Fedora/EPEL activity puts additional pressure on Fedora web proxies for mirror data and then the mirrors themselves that tend to be volunteer run. Much of this new traffic is coming from the Amazon/AWS cloud.
Isn't that sensible thing to do?
You now get the stock epel mirror listing, not a custom package Amazon rewrote, and they'd need to add a custom mirror that provides an override for that package to make it work that way.
So I would _guess_ one or more large companies migrated a bunch of things onto cloud hosted EL7-based system after EOL meant god help you finding new hardware support.
(I would also guess either EPEL doesn't have a mirror on Amazon or it should have auto-picked it, or the auto-picker is bad, and either way, Amazon will probably go "oh our bad", since I believe they host local mirrors of things anyway...)
AWS is just being ignorant. If I were in charge of Fedora infrastructure I'd block them and send them instructions on how to setup a mirror.
XML schemas have had a similar history, tanking w3c.org servers.
/s of course
Lot of half-baked ideas floating around in the early years of the commercial internet, but the Java thing held up better.
But it does allow more flexibility. If you don't want to be tied to a domain name, you can use a URN with a UUID like urn:uuid:4603d9d3-e895-4000-9077-0ab0f2776e1e
Then you prefetch to fix that problem, and now some slow calls you hadn’t gotten to the bottom of suddenly aren’t.
I'm sure some braindead software out there attempts to retrieve namespace URIs, but it would surely be a drop in the bucket compared to traffic for schemas/DTDs (which are intended to be retrieved).
A user with "five million additional systems" on AWS?
Someone is going to be in for a big surprise when they get their AWS bill this month and realise there’s an infinite-loop bug in their instance spawning script.
I suspect there are only a handful of vendors with popular enough AMIs or container images that could account for this, though.
If someone blocks you it becomes an incident, a post mortem and you learn your lesson.
If someone blogs about it, or e-mails you, it gets added to a todo list and might get fixed in a few weeks by a disinterested intern.
> ADDENDUM (2024-05-30T01:08+00:00): Multiple Amazon engineers reached out after I posted this and there is work on identifying what is causing this issue. Thank you to all the people who are burning the midnight oil on this.
You worked in ops, but not in a context where your employer could get shamed by IBM in public on the pages of Phoronix and HN. Call it "cloud scale" ops, I guess.
This is why torrents would actually work fine in this model.
What a nice gesture! However my mirror-servers are https plus rsync (for some projects). But are there not some ISP's that block torrent-traffic completely?
Well, that's how software is, but that article is from 2008 and things have gotten a lot better in the meantime (I think archlinux wasn't even signing their packages back then).
If the distribution's private keyring isn't compromised and you don't have third-party repos, your packages are as trustworthy as your distribution team (and upstream).
I'm sure this will happen once the Linux community standardizes on a package format /s.
I wish more systems had this kind of feature. I have a fat fiber connection, I'd be thrilled to pop up an unofficial mirror for something like a Linux distro. I've tried mirroring Linux ISO torrents, but it seems almost nobody ever downloads from a torrent, so I end up never actually uploading any of these images.
Therefore to use torrents requires one of two choices, neither of which are really viable when you think about it. Either:
* Require everyone who uses the distribution to torrent releases of the full set of packages for the distribution. (And remember that packages regularly get updated even in LTS releases, due to backporting security fixes etc.) It means everyone using the distro really needs to be running dozens or hundreds of torrents.
* Everyone torrents only the packages they need, so access to packages becomes up to popular vote instead of actual importance.
Torrents are fine for isos but intractible for ongoing package management.
It shouldn't be any worse than hosting old versions of packages on an HTTP file server.
* You can mitigate the popularity thing by using webseeds; if it's popular then it's p2p, and if not then it just falls back to HTTP
* Alternatively, you can 80/20 it; only torrent the 10 biggest and 10 most popular packages (changing 10 to taste), which is a relatively small number of torrents to run while minimizing load on mirrors
Under normal conditions with well behaved clients, raw bandwidth for the large packages is essentially never the issue. Misbehaving clients, cache thrashing, IOPS are the sort of issues that cause pain for mirrors.
Then the mirror became too slow and couldn't handle the amount of package data and was shut down: https://github.com/RubenKelevra/pacman.store
It worked rather well and even automatically mirrored the packages on my LAN. Maybe it'll be back some day.
All of those packages are then available for pacman under the (unchanging) IPNS name and the packages themselves are stored in IPFS as usual files. All you had to was add `http://x86-64.archlinux.pkg.pacman.store.ipns.localhost:8080...` to your mirror list after running your ipfs node (or use a public ipfs gateway).
The website sadly is down now. It had a nicer explanation. None of this still works now however.
EDIT: the folder/list of files was managed by the user hosting the project. The files/packages themselves were distributed. That's just how IPFS/IPNS works. Somebody has to put them into IPFS.
EDIT2: technically the example I posted is a DNSLink which points to a (now gone) TXT record which points to an IPNS name (which is a mutable pointer) that contained the directory of all current archlinux packages.
I hope that was comprehensible.
I did try doing something fun with network-interface scripts and avahi to make this more dynamic - the idea was my desktop and laptop both ran squid proxies, and then dynamically set peer proxies if they detected each other on the network. It didn't work great given the overhead.
It really feels like there should be a better way to do this in general, and without having to break TLS directly for things which are not really "secret".
But every tool has its own config to set to use the Artifactory. One setting for the OS package manager (which is different for different Linux distributions), another for PyPI, another for NPM (or Yarn or whatever), another for Maven/Gradle, something else for Go, then I need to download this Postgres extension and build it from source - the list goes on. So almost inevitably something gets missed and one ends up not being as good a citizen as one ought, and then one day some random Jenkins job is failing because some external dependency could not be downloaded.
I wish there was an easier way. Like some standard mechanism for saying “for this URL use this proxy”.
I guess one could just use a proxy server (http_proxy environment variable) but with most things on HTTPS it needs to MITM the TLS which then means you need that certificate installed in the build process - which is another one of those “everything can do it but everything does it differently” problems. And in any event, MITM is a bad smell.
The problem is most clients don’t do GET https://, because in your old-school corporate web proxy use case, the proxy server is remote, and sending HTTPS requests to it over HTTP eliminates the security of HTTPS.
If only there was some standard environment variable like artifact_proxy, which had to be a localhost http URI, and which tools would understand as meaning “send HTTP GET to this proxy, even for https://, delegating all the TLS stuff to it, but only if you are trying to download a build artifact, not for any runtime use”
The hard part wouldn’t be implementing this idea (the local proxy server and the environment variable), the hard part would be getting all the different tool developers to agree to support it
Tools like Artifactory are a hack built upon a hack built upon a hack; redirecting certain HTTP requests to proxies would just be piling even more crap on top.
Content-addressing is a much cleaner option: identify files by their hash, rather than as the result of a HTTP request to some particular URL (many tools will already use these hashes to verify the result anyway; that's what "lock files" are for!). Content-addressed data is agnostic about how its retrieved, which makes caching trivial. There's no need to care about the data format, whether it's RPM, Deb, a source tarball, a patch, a Python "egg", or whatever.
For example, I lead a transition to Nix at a previous employer. We had a bunch of projects with various build processes (Maven, SBT, Gradle, PyPI, NPM). After wrapping these in Nix, the whole lot could be cached on S3 by simply copying files around (see https://nix.dev/manual/nix/2.22/store/types/s3-binary-cache-... )
Pulp is perfect for this but the demands on my time make it hard to see around the corner.
An astounding milestone for the English language.
Imagine what this sentence possibly could have meant in 1990.
I thought this title referenced cars driving in a south american rain forest.
A decade ago, each company used to have its own cache of all public packages for CI/CD, but it looks like that bo one cares anymore.
Even worse, many of those processes are running an 'apt-get update' (or equivalent), so there's no way to know what packages they'll get each time!