Hacker News Reader

Comment by cesarb | Hacker News Reader

One of these is not like the others...

> The problem was that go get needed to fetch each dependency’s source code just to read its go.mod file and resolve transitive dependencies.

This article is mixing two separate issues. One is using git as the master database storing the index of packages and their versions. The other is fetching the code of each package through git. They are orthogonal; you can have a package index using git but the packages being zip/tar/etc archives, you can have a package index not using git but each package is cloned from a git repository, you can have both the index and the packages being git repositories, you can have neither using git, you can even not have a package index at all (AFAIK that's the case for Go).

View on HN

The author seems a little lost tbh, it's starting with "your users should not all clone your database" which I definitely agree with, but that doesn't mean you can't encode your data in a git graph.

It then digresses into implementation details of Github's backend implementation (how is 20k forks relevant?), then complains about default settings of the "standard" git implementation. You don't need to checkout a git working tree to have efficient key value lookups. Without a git working tree you don't need to worry about filesystem directory limits, case sensitivity and path length limits.

I was surprised the author believes the git-equivalent of a database migration is a git history rewrite.

What do you want me to do, invent my own database? Run postgres on a $5 VPS and have everybody accept it as single-point-of-failure?

> Run postgres on a $5 VPS and have everybody accept it as single-point-of-failure

Oh how times have changed. Yes, maybe run two $5 VPSs behind a load balancer for HA so you can patch and then put a CDN in front of it to serve the repository content globally to everyone. Sign the packages cryptographically so you can invite people in your community to become mirrors.

How do people think PyPI, RubyGems, CPAN, Maven Central, or distro Packages work?

Sure let me put all that on my credit card because some guy doesn't like git.

The situation that PyPi is in is clearly worse: https://stackoverflow.com/questions/39537938/how-do-i-downlo...

You wouldn't be the one paying for it, like PyPi you would upload your package to them.

When you bootstrap your package ecosystem using git forges for hosting there's no index at all so I'm not really sure what the argument is.

The target audience for the article are people building these systems, so the people who would have to pay for the centralized infrastructure.

With git there's a sync protocol built-in that allows anybody who's interested to pull a copy of the index (this shouldn't be the default distribution model for the package clients, but anybody who truely wants it can pull it). PyPi is keeping their index private and you'd have to scrape all data through a heavily rate-limited API.

I think the article takes issue not with fetching the code, but with fetching the go.mod file that contains index and dependency information. That’s why part of the solution was to host go.mod files separately.

Even with git, it should be possible to grab the single file needed without the rest of the repo, but i'ts still trying to round a square peg.

Honestly I think the article is a bit ahistorical on this one. ‘go get’ pulls the source code into a local cache so it can build it, not just to fetch the go.mod file. If they were having slow CI builds because they didn’t or couldn’t maintain a filesystem cache, that’s annoying, but not really a fault in the design. Anyway, Go improved the design and added an easy way to do faster, local proxies. Not sure what the critique is here. The Go community hit a pain point and the Go team created an elegant solution for it.

I was thinking this too. I think it might be talking about operations like “go mod tidy” or update operations where it updates your go.mod/sum but doesn’t actually build the code. I would guess enterprise tools do a lot of checking whether there are updates without actually doing any building.

Reply on news.ycombinator.com