Hacker News Reader

Comment by galleywest200 | Hacker News Reader

> It should not be necessary to fetch the same git module up to 2,000 times per day.

Holy cow Google! Wouldn't it behoove us to check if any changes occurred before downloading an entire repo?

I don't understand why they have to do a fresh clone every time.

Or, as mentioned in the post, why they don't do a shallow clone if they have to fetch it every time for whatever reason. Seems like a weird decision either way.

Yep, a shallow clone is enough to get the latest version. And you can even filter the tree to make the download size even smaller given you only want the hash but not the contents (if the git server supports this feature)

A checkout with this can literally clone nothing but hash

    git clone --depth=1 --filter=tree:0  --no-checkout https://xxxx/repo.git
    cd repo
    git log

People using Go modules should be using git tags, right? They should have at least one hash already that should be infinitely cacheable, the tag commit.

Of course, I have seen alleged examples of Go modules using tags like branches and force pushing them regularly, but that kind of horror sends shivers down my back, at least, and I don't understand why you'd build an ecosystem supporting that sort of nonsense and which needs to be this paranoid and do full repository clones just for caching tag contents. If anything: lock it down more by requiring tag signatures and throwing errors if a signed tag ever changes. So much of what I read about the Go module ecosystem sounds to me like they want supply chain failures.

I don't understand the Go ecosystem.

presumably the proxy backend is stateless

At the end of the day it has to store, consolidate, and present that state to the service's consumers, so there is state somewhere.

> there is state somewhere

Yeah, on third-party code hosting platforms :). And maaaaaybe in some short-lived cache somewhere. I mean, why spend on storage and complicate your life with state management, when you can keep re-requesting the same thing from the third-party source?

Joking, of course, but only a bit. There is some indication Google's proxy actually stores the clones. It just seems to mindlessly, unconditionally refresh them. Kind of like companies whose CI redownload half of NPM on every build, and cry loudly when Github goes down for a few hours - except at Google scale.

It sure reads like someone wrote this the dumbest way that could possibly work, without a thought for what the effects would be.

could it be a CI service that builds the go-proxy server for testing and that build process does an initial clone of all sorts of go modules?

Reply on news.ycombinator.com