Hacker News Reader

Comment by alkonaut | Hacker News Reader

I didn't lift an eyebrow at the headline which is apparently an april fools joke.

But what is it that makes it so absurd that it works as a joke, which I'm missing? The size seems absurd (500GB) but why would the Debian source be 500GB?

To me it's in no way obvious that "Debian" would include any third party packages repackaged by the distro maintainers. Are Debian people maintaining N source repositories for third party code today, which would be the code merged into this hypothetical monorepo? Why?

View on HN

This includes the sources of all the Debian packages.

In general this is a good idea, and it is close to how nixpkgs works (which is, in some sense, a "distributed monorepo"), but git, the Debian tooling and so on wouldn't actually be able to support a good workflow here.

> it is close to how nixpkgs works

Not really...? Nix distributes package definitions that describe how to build and install the package. This is close to just about every other package manager, besides the fact that most don't have / use the description on how to build the package, and instead they just download a binary based on the package definition.

The Nix equivalent to this joke would be storing all of the package code directly in the nixpkgs repo, instead of storing links to the source code. Debian would ba able to support the same flow as Nix just fine. You can just use source packages[0]

[0] https://wiki.debian.org/Packaging/SourcePackage

I’d say it is close. A purer example would be gittup[1], which has its (very few) upstreams as submodules. But ultimately, Git submodules are just URLs and commit hashes stored in .gitmodules in the repository root, which doesn’t seem all that different from the URLs and hashes of contents passed to fetchgit et al. in Nixpkgs. The capability of being able to patch an arbitrary package locally and have every transitive dependency rebuilt automatically is also the same.

On the other hand, Debian and Fedora maintain full collections of sources used in their releases, while, AFAIU, if you need an old version of something from Nixpkgs, it has fallen out of the official binary cache, and the upstream is gone, you’re on your own. (There have been some efforts to fix that[2], but the first attempt was very limited in scope, and there doesn’t seem to have been a second one.)

[1] https://gittup.org/gittup/

[2] https://www.tweag.io/blog/2020-06-18-software-heritage/

Things don't "fall out of" the binary cache. Garbage collecting it is an active research area.

I think its less maintenance and more for pinning versions of software to ensure stability. I guess that comes with maintaining a package manager like apt.

Definitely with you on not realizing it was a joke at first. Don't google have a company wide mono repo, that I'd guess includes all search and chrome? Or is that just an urban legend?

The stuff in Google's monorepo is mostly their own code, not snapshots from upstream.

Chrome has brought in third party code. Which makes Debian mad and they do a lot of work to strip that out to use the upstream libraries. (Or at least that is how I understand it - I don't follow Debian closely)

Yeah, that's why I wrote "mostly" :)

Google's monorepo includes snapshots of everything upstream that Google's proprietary code depends on - starting from the kernel and glibc and core language toolchains, and all the way up to various open-source libraries and modules in c/python/java/whatever.

But 3p sources likely are for specific stable version? Linus can't break Google by submitting bug into linux kernel?

Yes.

Isn‘t git submodules a much more lean way at accomplishing this than a clunky 500gb repo?

Doesn't google have a lot of internal tooling to make that monorepo actually work for them?

The issue isn't so much that large monorepos don't work, it's that you are trading one set of issues for another set of issues. And most popular open source tooling (and I assume all debian-specific tooling) is built around solving the issues of many repositories rather than solving the issues of monorepos.

> Doesn't google have a lot of internal tooling to make that monorepo actually work for them?

probably tests coverage and workflow matters: you can work with monorepo if you can check how your change will impact rest of the ecosystem.

At its core, with separate repositories it's generally fine to pretend that any commit effects all code within that repo. So for example you might run all unit tests for every new commit in a PR, or run all deploy scripts when something is committed to master. Dependencies are usually set up one-way, so code pulls it its dependencies (by referencing specific git commits or released packages). So a change in one repository might mean that it pulls in new or updated dependencies, but software depending on you should only see changes once a change is made to its repository.

For a monorepo those assumption don't hold, any given commit likely only affects a tiny fraction of the content of the monorepo. But a change in a library might also immediately radiate outward to anything in that repo that uses the library, so you can't do a naive check based on the directory tree. So you end up with a build system like bazel that can evaluate the entire dependency graph to know what to run. And you have to do that for pretty much all tooling you want to be triggered by code changes.

Add on top of that the scaling issues you have with git as your repository grows

yes, and a lot of it's open source.

The main issue IMHO is that git doesn't scale for a monorepo. A monorepo wants to spread a single virtual repo over a large number of devices, with no device having the entire repo. In contrast, git wants to do the opposite: it's built for a large number of devices to have the same copy of the entire repo.

But aside from tooling and needing a horizontally scalable VCS, there's also the cost and engineering resources required to keep the cluster running, available, and high performance.

I think chrome is got its own thing if I remember from my internship. But almost everything in one repo sounds right.

Debian does include the third party packages. This is either a mirror of the git repository, or a tarball of the upstream code.

it's simpler than that: it's just running `dpkg-source -x` on every `*.dsc` file it can find. So that's "all of the source used to build all of the debs" (using dists/stable/main/source/Sources.xz to get the list.) (I think then just a single commit of all of it; the fun bit would be doing this for each release and looking at the size of the diffs...)

I meant what upstream Debian includes.

Reply on news.ycombinator.com