Hacker News Reader

Comment by xfbs | Hacker News Reader

It's amazing how much stuff breaks when GitHub goes down. I'm doing some Rust coding right now, the rust-s3 crate tells me that to look up what feature I need to enable (tokio + rustls), I need to look into the Cargo.toml. Well, the repo won't load, and nor can I clone it. Well okay, fuck that I can use the default dependencies. But no wait I can't, I can't even do a cargo build because cargo uses a github repository as the source of truth for all crates. No more Rust for me today :(

View on HN

In the 1970s-80s we managed with email lists and tarballs on multiple mirrors. Git itself decentralizes source control, and yet we all want to use single-point-of-failure Github.

Anyone remember when Github took down youtube-dl?

I wonder how much Big Brother data Microsoft is gathering on Github developers. "Oh, as part of our hiring process, just like we scan your FB and Twitter, we also scan your Github activity to evaluate your performance as best as our machine learning overlords can assess it. Do you create Github projects and then abandon them when unfixed issues accumulate? Our algorithm thinks you're a bad hire."

> Anyone remember when Github took down youtube-dl?

I would blame the law (DMCA), not those forced to abide by the law (Github)

Especially because they worked to try and make sure this wouldn't happen again.

The law also permits content to be restored immediately upon receipt of a counternotice. Somehow the data silos never get around to supporting that.

twitch actually does do that IIRC, not sure about others

GitHub can't, until at least the the public support is in their favour, because legally it's a hot water. This is not your ordinary copyright DMCA Complaint, this is the section 1201 concerning circumvention, which is held to a different standard. This is essentially RIAA telling GitHub "if you're not stopping this, we'll see you (not the YT-DL developers) in court".

> GitHub can't, until at least the the public support is in their favour

Public support on which grounds?

I can’t see any evidence a counter notice was filed until the EFF got involved, at which point GitHub immediately reinstated YouTube-dl.

Did GH ignore a counter notice?

It was an unfortunate situation. I don't entirely know how GitHub handled things before the Microsoft acquisition but it also wouldn't surprise me if Microsofts legal department is more risk adverse compared to GitHub internal.

> I would blame the law (DMCA), not those forced to abide by the law (Github)

It seems even in the context of US-based companies that some companies are a little more "trigger-happy" with DMCA and other claims compared to others.

Laws need teeth to be enforced. It used to be illegal to harbor runaway slaves, or to be a Catholic priest in Protestant England. So people built cleverly hidden Priest Holes[1] in houses to hide. But you can't really build the equivalent of a Priest Hole within Github, because Github is not your house, but rather your lord's castle.

[1] https://en.wikipedia.org/wiki/Priest_hole

> blame the law (DMCA), not those forced to abide by the law (Github)

What about those abusing [a misrepresentation of] the law (the RIAA) and the members who pay their dues (Microsoft)?

Don't create a Github account with your name then?

Yeah seems like this would be a simple workaround if companies did start adopting this practice

but that data is avaliable to everyone cuz it's mostly public?

> Do you create Github projects and then abandon them when unfixed issues accumulate? Our algorithm thinks you're a bad hire."

This data is public and to be honest I’m amazed we haven’t yet seen a YC startup promoting that they do exactly this.

Shhhh. The former Klout folks are probably listening.

It was still rad back in the 1990s and 2000s, then came in people who think that if they repeat a bullshit sentence loudly enough and pushily enough, people at large will equate it with truth: "Email is too inconvenient to use".

I'd say those people do have a point about how manipulation tactics work, but email being inconvenient is bullshit.

What's inconvenient is using proprietary Electron apps just for communicating, while not being able to communicate on my phone without downloading and installing a proprietary app.

On the other hand, you have email: open-source clients are available for virtually every platform, it doesn't turn my laptop into an electric bar fire, it doesn't drain my phone's battery (or if it does, I could just install another client!), etc.

It's quite sad how something as beautiful as email is being replaced by corporate garbage like Teams, which comes bundled with Office 365 (and that's a whole 'nother crapfest). Uch.

There was actually a movie about this very thing. Miguel De Icaza had a cameo. The villain company was a thinly veiled replica of Microsoft, and the CEO looked like Bill Gates. ( https://www.imdb.com/title/tt0218817/ )

In another thread a few weeks ago about code signing, I mentioned we should just hand Apple Source code for the apps it distributes in the store, to make review more thorough and reliable. Replies objected "Why would you give your source code to another company?" ........ and github is still a thing? ( https://news.ycombinator.com/item?id=28794243)

Some of the features in that movie would work nowadays, such as paintings detecting a visitor and changing based on the visitor or their mood or sound input (don't remember exactly). Wasn't there also something with satellites?

Also, ironic De Icaza ended up working for... the company he wanted to work for all along, ...Microsoft. Though Friedman left Microsoft (or at least stepped down as GitHub CEO).

De Icaza was also featured in the Finnish movie The Code [1].

[1] https://en.wikipedia.org/wiki/The_Code_(2001_film)

> Git itself decentralizes source control, and yet we all want to use single-point-of-failure Github.

This is pretty much why many organizations out there, as well as i personally for my homelab use self-hosted GitLab instances: https://about.gitlab.com/

Though in practice there are a lot of other options out there, like Gitea (https://gitea.com/) and GitBucket (https://gitbucket.github.io/), though maybe less so for alternative source control systems (e.g. SVN has been all forgotten, however that's a personal pet peeve).

Not only that, but i also utilize my own Sonatype Nexus (https://www.sonatype.com/products/repository-oss?topnav=true) instances to great success: for doing everything from mirroring container images that i need from DockerHub (e.g. due to their proposed removal policies for old images and already adopted rate limits), to mirroring Maven/npm/NuGet/pip/Ruby and other dependencies, so i don't have to connect to things on the Internet whenever i want to do a new build.

That not only improves resiliency against things on the Internet going down (apart from situations where i need something new and it's not yet cached), but also improves performance a lot in practice, when only the company servers need to be hit, or my own personal servers in the data center for my cloud hosted stuff, or my own personal servers in my homelab for my own stuff.

Admittedly, all of that takes a bit of setup, especially if you happen to expose anything to the web in a zero trust fashion (permissible for my own stuff, as long as i'm okay with manually managing CVEs just to probably get hacked in the end anyways, but definitely not something that any corporation with an internal network would want to do), but in my eyes that's still worth the effort, if you value being in control of your own software stack and the ecosystem around it.

It's probably much less worth it, if you don't see that as a benefit and don't want to be the one responsible for whatever project you're working on getting hacked, e.g. if you'd fail to patch out the recent GitLab CVE where exiftools could execute arbitrary code, which is probably the case if you don't have the resources to constantly throw at maintenance, in comparison to companies with 100x - 1000x more resources than you have for that sort of stuff.

Cargo has an --offline option. It's actually pretty possible to use rust totally offline - the doccumentation can be built, for instance, then locally served with (iirc) cargo doc.

Offline should be the default, not an option.

Otherwise nobody will notice how fragile their workflow is until it is too late to fix it!

I recently realized I was the only person running the API locally when suddenly all our unmocked tests started failing only on my machine.

All other tests, including the ones on the CI server, were connecting to the hosted integration environment for any unmocked call (and thus randomly modifying state there).

I build and host documentation on every commit anyways in the CI. And yes, that is true, I eventually figured it out (had some issues with it at first) but it seems like GitHub is back up so all's well anyhow. I do however wish that there was some public mirror that cargo could fall back to, wouldn't that make a lot of sense?

It would be good if such things would be hosted decentralized on something like say IPFS.

It's amazing how much stuff breaks when GitHub goes down.

That's right. Someone should really come up with a decentralized VCS. /scnr

IMO Git is decentralization-ready but the rest of the infrastructure necessary to make it practical is not widely available/in use. The necessary peer-to-peer and networks of trust are still not a solved problem, or if they are they've for some reason are not popular enough to be in wide use.

It's widely available and widely used, just not as widely used as github.

Setting up your own git server is not hard, but it's not as easy as just getting github or gitlab to run it for you. Way too many people take the easier path, even though the harder path is not actually that hard.

There are also multiple solutions.

It’s ok for small private projects, but user management is hard. You can’t just let any rando have commit privileges.

that's what things like gitlab (not the website) or gitea are actually for.

> Setting up your own git server is not hard

Bullshit. Getting it set up in a secure and public way is an order of magnitude or two more difficult than it would need to be for mass use which is what we're talking about here.

When you have a situation where a the vast majority of people find something too difficult, the problem (from the perspective of actually getting it happen) is not the people the problem is with the proposed solution.

> There are also multiple solutions

Multiple solutions does not generally make things easier/better even if it's desirable for other reasons.

You probably meant distributed decentralized VCS.

Beside the `--offline`, you can also use `--vendor` to include all the dependencies in a folder to be committed alongside your project. Useful when you don't want to rely on external fetch every time!

Both should be the default.

Rust has a robust memory model, but everything else about it insists on copying the fragility of the NPM ecosystem.

The recent hoopla around a bunch of Rust mods quitting revealed that my suspicions are precisely true — key Rust staff also sit on the NPM board!

Cargo is the one thing keeping me off the rust ecosystem. The fundamentals of the language are great, but the tight coupling of the rust language with cargo's package management really irks me - it introduces as many correctness and security problems as the memory model solves.

What languages have package management systems that solve those problems? Cargo does have options to fix these, like mentioned in other comments. I'm not convinced they should be defaults.

I'm suggesting that "languages that have package managers" is generally the problem. I think the go package manager gets the closest to solving these problems thanks to its reliance on one trusted source that has STRONG incentives to make sure that packages are available and trusted, but fundamentally, a language and a package manager are very different products, and I don't want them to be bundled.

You can view the source of a crate on docs.rs (see [1] for the Cargo.toml of rust-s3). Also I am pretty sure cargo only depends on GitHub for authentication for uploading crates and not for the actual contents. Trying to build an empty crate with rust-s3 as a dependency right now seems to work fine.

[1]: https://docs.rs/crate/rust-s3/0.28.0/source/Cargo.toml

As I understand it, the crates themselves are not stored on github, but the crate index is, as it uses a git repo to get "free" delta compression and auditing.

I can't stand builds that just reach out to the internet for things.

Yeah seems like basic hygiene especially given the supply chain attacks but also a lost cause. No one has the skills to do builds without the internet. Even forcing teams to use an allow list for the internet involving fighting a lot of angry people.

One of the few decent uses for containers is to enforce proxied internet so build process artifacts can be auto-stored for subsequent builds.

For the worst offender I am aware of, try building a flutter project... it silent internet gets artifacts from at least three different packaging systems (node, cocoapods, android packages), all of which have caused hard-to-debug failures.

How do you build software in 2021 then? Do you write everything from scratch?

... yes? I've often been ridiculed for wanting to build everything in-house, but events like these just validate that sometimes, to get the most reliable and best software, you'll have to reinvent the wheel a few times.

To be honest that's deserving of ridicule. Reinventing wheels is how you introduce fragility and instability, not prevent it.

One can use the packages provided by the distribution. That's how it worked since forever for things like C/C++.

For example distris like Debian even mandate that you build software for them only relying on things that are already packaged. No arbitrary internet downloads allowed during build. The build environment contains only what is provided by the stated build-dependencies of your package, which are themself of course Debian packages, and some common minimal build base (like shell and some utils).

What if the package you want isn't provided by the distribution?

What if the distribution's version doesn't support the features you need?

What if it can't be built by relying only on things that are already packaged?

How do you distribute your software so it runs on other distributions? Do you maintain a different build for each package manager?

What if you want to run on platforms that don't have standard package managers, like MacOS or Windows?

How do you update the packages without internet downloads during provisioning the build?

The C/C++ model has proven to be fatally flawed. It hasn't worked, that's why modern languages eschew reliance on the distributions' package managers, and greenfield C/C++ projects use the same model.

I'd go so far as to say this model is a key reason why we need containerization to deploy modern software today - since you can't trust software written on distro X runs on distro Y because it packages dependency Z differently all the way up to the root of the dependency tree (glibc, usually).

The fundamental flaw of this model is that it inverts the dependency tree. Build time dependencies are local to individual projects and need to be handled separately from all other software. With few exceptions, Linux distros make this mistake and it's why we can't rely on distro package management for new projects.

> What if the package you want isn't provided by the distribution?

You package it.

> What if the distribution's version doesn't support the features you need?

You package the version that does.

> What if it can't be built by relying only on things that are already packaged?

You package the dependencies first.

> How do you distribute your software so it runs on other distributions?

Give them the source so they can package it.

> Do you maintain a different build for each package manager?

Yes, that's the whole idea behind distributions.

> What if you want to run on platforms that don't have standard package managers, like MacOS or Windows?

Well, there you anyway downloaded random things form the internet…

But nowadays even those systems have package management that can be used!

> How do you update the packages without internet downloads during provisioning the build?

It's not about disabling package downloads (which come form a trusted source btw).

It's about disabling downloads form random places on the internet.

Also you can use a local package mirror. That's what the build systems of distributions do.

> The C/C++ model has proven to be fatally flawed. It hasn't worked, that's why modern languages eschew reliance on the distributions' package managers, and greenfield C/C++ projects use the same model.

Well, except for all packaged software out there…

> I'd go so far as to say this model is a key reason why we need containerization to deploy modern software today

Nobody needs that. Software form packages just works fine. All the Linux installations around the globe are a prove of that fact.

> since you can't trust software written on distro X runs on distro Y because it packages dependency Z differently all the way up to the root of the dependency tree (glibc, usually).

Of course you can. It works exceptionally fine. After you packaged it.

> The fundamental flaw of this model is that it inverts the dependency tree. Build time dependencies are local to individual projects and need to be handled separately from all other software. With few exceptions, Linux distros make this mistake and it's why we can't rely on distro package management for new projects.

More or less every distro does it wrong? And you know how to do it correctly?

Maybe you should share your knowledge with the clueless people building distributions! Get in touch with for example Debian and tell them they need to stop making this mistake.

By the way: How does your software reach the users when it's not packaged?

At least I hear "sometimes" that users demand proper software packages. But maybe it's just me…

I think you're conflating package management for distribution and package management as a build tool. This is the flaw that Linux distributions make. It's why flatpaks and snaps exist. On MacOS and Windows you distribute software as application bundles similar to flatpak/snap.

>More or less every distro does it wrong? And you know how to do it correctly?

Yes, exactly. For distribution you use snap and flatpak, or application bundles on MacOS and Windows. For building you use a package manager that can install vendored dependencies pinned to specific versions, and do it locally to individual projects so they are not shared across multiple projects. This is the tact taken by modern tools and languages.

It's not my idea! It's what everyone building software in the last decade has migrated to.

Packaging software so it can be used by any distro's package manager is not viable, and reliance on such an outdated model is why using Linux sucks for everything but writing software to run on a single machine.

> At least I hear "sometimes" that users demand proper software packages. But maybe it's just me

And when they do you almost never go through the official/trusted mirrors because it will never be up to date, you host your own (maybe put up some signature that no one ever checks) so they just `sudo apt repository add ; sudo apt-get install -y` in your install instructions.

> Packaging software so it can be used by any distro's package manager is not viable, and reliance on such an outdated model is why using Linux sucks for everything but writing software to run on a single machine.

Sorry, could you elaborate on why Linux "sucks" for anything but single-user, please? I haven't heard this argument.

> It's not my idea! It's what everyone building software in the last decade has migrated to.

Well, everybody besides the biggest free software distributors out there, which are Linux distributions…

Yeh, I get it. That's why we need Docker…

Nobody is able to distribute software otherwise.

Well, except all those "distributions"…

> And when they do you almost never go through the official/trusted mirrors because it will never be up to date, you host your own (maybe put up some signature that no one ever checks) so they just sudo apt repository add ; sudo apt-get install -y in your install instructions.

Moment. Does this mean someone managed to build packages that work on different versions of even different distributions, which is obviously impossible as we just learned?

This starts getting hilarious!

I'm sorry for you that nobody want's to listen to your great ideas. Have you ever considered that you're completely wrong?

This is why a lot of companies use a product like jFrog artifactory to handle dependencies. Instead of trying to download files from GitHub and other random places on the internet, the artifactory is a local mirror of all of those files, with servers on the artifactory which run the relevant services such as a npm repo of packages, a npm repo of packages, and so on.

This way, if GitHub falls off of the face of the earth tomorrow, the packages still build. Also: If some open source maintainer deletes an older version of a package, replacing it with a new one with a different API, the local package will still build.

(The local build process will have to be set up to use the local artifactory to pull dependencies, of course.)

Should be fairly easy for Cargo to solve this: why doesn't it already mirror its source of truth to GitLab and other Git hosters?

Probably because the level of effort required to configure that and then monitor/manage it continuing to operate doesn't make sense when GitHub has an outage or two a year.

I was working on my Nix config. I had just added a small command line utility and wanted to install it, but then got 504 errors from github.com. Annoying!

I guess same story for Go!

In Go, it's customary to use 'go mod vendor' to put your dependencies into the main repository. While it's not universally recognized as a good technique, it saves the adopters of this approach from the downtime today.

I’m not sure this is customary at all - I rarely encounter a vendor directory anymore.

Note you also need to build differently if you go this route: `-mod=vendor`, otherwise the directory will be ignored in modern Go.

With more outages of our amazing, but overly centralized development ecosystem, the popularity of this approach will likely surge. It helps that Go supports the vendoring workflow as it makes the choice practical.

As for building with a flag: true, but very minor, as it's rare to execute 'go build' directly. In most projects I've seen, it's either Bazel, or some kind of "build.sh".

I've never had to use -mod=vendor, so I just looked it up:

        -mod mode
                module download mode to use: readonly, vendor, or mod.
                By default, if a vendor directory is present and the go version in go.mod
                is 1.14 or higher, the go command acts as if -mod=vendor were set.
                Otherwise, the go command acts as if -mod=readonly were set.
                See https://golang.org/ref/mod#build-commands for details.

If there's a vendor directory, it's used by default. As for my two cents, I use it frequently when building Docker images so that I don't have to pass secrets into the image to clone private modules (but I don't check the vendor directory into Git).

Maybe the key point is to choose consciously and pick the option that gives the best combinations of tradeoffs for your situation vs just doing what is easy or copying what other people are doing without understanding you're making a decision with various tradeoffs and consequences. Tradeoffs that are a good fit in other contexts may be a poor fit for your situation.

If one of the goals of your build process is to be able to guarantee reproducible builds for software you've shipped to customers, and you depend on open source libraries from third parties you don't control, hosted on external services you don't control, then you probably need your own copies of those dependencies. Maybe vendored into version control, maybe sitting in your server of mirrored dependencies which you back up in case the upstream project permanently vanishes from the internet. But setting up and maintaining it takes time and effort and maybe it's not worth paying that cost if the context where your software is used doesn't value reproducible builds.

Google takes care of storing copies of any go dependency you use on their proxy, there is very little reason for you to maintain your own via vendoring. Maybe if you are a big enough organization you run your own proxy as an extra layer of safety above google but still I don't see the value of vendoring these days.

These days with go mod and the go team maintaining their proxy there is very little benefit to vendoring and any benefit is not worth blowingup up the size of your repos and messing up code reviews on PRs that introduce new dependencies.

I think Go dependencies should still work, thanks to Google's module mirror[0] (enabled by default), which has cache.

[0]: https://proxy.golang.org/

The ones using GOPROXY=direct will fail, although it’s used rarely.

Go uses a proxy for downloading modules, so there's no Github involved. And you could run your own proxy-cache if you wanted. In addition, your work machine has a local proxy cache of the modules you already downloaded.

Go doesn't use a repository as a single source either, which is another problem in of itself.

Reply on news.ycombinator.com