Don't couple your Go code to GitHub | Hacker News Reader

Don't couple your Go code to GitHub

iain.rocks

323 pointsbirdculture177 comments

> In my opinion, every commerical software development team using Go should be using custom domains for namespacing their internal libraries and packages.

I’d remove “go” from the above, i.e. I think same applies to other stacks.

Even using GitHub domain links in code comments gets problematic long term. Ie when a migration happens and those links start pointing nowhere.

I'm confused by the article, and this comment, treating these URLs as difficult to replace.

Why can't you just search-and-replace? Presumably all of them refer to "GitHub.com" and not much else code will, so I'd think this was an exceptionally easy case.

Even easier for comments, since them being obsolete for a few hours during a migration doesn't exactly break anything.

It's harder to replace in history.

I've gone through such a migration. Huge Go code base distributed across a lot of git repositories had to be moved to a different git host.

It was horrible. A team of people spent weeks. Different projects depended on different versions of the same internal libraries, we had to create a branch for each depended-on commit and make a version of that commit with the new URLs. The expressed goal was to end up with a system that's "exactly the same" as the old, just on a new host. Changing which version of libraries projects depends on would introduce unnecessary risk.

And the result is a code base where bisects are broken and where it's impossible to build an old version of any of the code without a ton of work.

> Different projects depended on different versions of the same internal libraries, we had to...

This is one of the main motivations for using a monorepo with all third party dependencies imported all the way in, and only one version for everything.

Of course it causes a whole bunch of other problems and is kinda expensive to scale, so most places won't do it.

Isn’t this exactly why the Go team recommended vendoring deps for years and years before go modules came up? Even now it takes one simple command to vendor them.

Google runs a monorepo which makes that kind of organization possible. Most companies are going to have a mismatch of teams using incompatible library versions.

Pick your poison. Each approach has some seriously negative trade offs in the extremes.

Well vendoring is a technical solution of the same caliber as adding module aliases to the go.mod. By that I mean, sure, it keeps the code compiling right now, but all your source files still contain references to abandoned infrastructure. It changes when you have to do the work, but unless you want your code to reference abandoned infrastructure forever, you still need to do the work.

> And the result is a code base where bisects are broken and where it's impossible to build an old version of any of the code without a ton of work.

That makes no sense? As long as you’re using go modules, you can just host an internal go proxy for internal modules similar to proxy.golang.org to archive the old versions, and they won’t depend on the old git host.

I think is usually a symptom of how hard it is for the organization to do things, the technical side of this is not actually that hard.

I say this because I’ve gone through a few of these, some with monorepos (the easiest, search and replace and you’re done), some not (yes the hardest but you can just vendor).

At least in the situations I’ve faced this was genuinely not horrible beyond just the organization itself being complicated about the solution.

How do you propose it's solved in an easier way, given the constraints that 1) you don't want references to old infrastructure in your code and 2) you want to do the move without changing how the code works? Is there some secret trick we missed which would've made this much easier?

Everything is on A. All code refers to A.

Clone Server A -> Server B. All code still refers to A.

Update code to refer to B

You can now EOL Server A, and B becomes the new canonical reference.

I think the key is simply that you can keep both A + B running during the migration, so you just need to be able to do a code freeze for the duration of the migration. And a single person can easily do this migration with a few python scripts and an hour.

Bonus points: code freeze guarantees your #2 - no changes to how the code works.

Of course, I'd assume that most complaints come from situations where this "secret trick" isn't viable

The "update code to refer to B" step is a massive undertaking. My whole original comment was about that one step.

I don't understand. If B already exists and is working, isn't this just Ctrl+F, compile, push? I'm utterly unfamiliar with the "Go" language

No, it's not. Here's a comment where I described a concrete hypothetical scenario: https://news.ycombinator.com/item?id=49871032

You vendor the deps

I don't know enough about git but shouldn't you be able to take all your git data (refs, blobs etc) and import it to your destination host and have this work without issues? It might be a lot of data sure but the infra is surely cheaper than multiple weeks of engineering effort

Moving the repositories over was no problem at all! But once all the data is on the new hosts, all the Go source files still reference the old git host, since imports in Go are URLs to a git host.

Similar pain as upgrading a module to a new major version (eg: v1 -> v2) [1], but at scale, because of modules dependencies: every module in the domain are affected at once as they must change namespace, like if everyone changed its major version.

https://go.dev/doc/modules/major-version

Even easier than that, you can use the 'replace' statement in your go mod to change where the Go build system will try to pull the dependencies from; you can point to a folder on disk or to another forge, and if you want total control you can indirect everything to your own artifact cache via GOPROXY (which can be something as simple as a static folder of source code).

The "module name is network path" is a convenient convention but not at all some "limitation" of the tooling.

And now all your source files contain URLs to abandoned infrastructure. Is that what you'd want long term?

How's that any different from java code using package names starting with com.sun?

com.sun is an identifier, nothing more. No tooling sees com.sun and assumes, "oh that means I can make an HTTP request to the IP address which sun.com resolves to".

In Go, the package identifiers are literally URLs. Go's own tooling assumes that it can make an HTTP request to the URL and that the response will be HTML with particular tags which Go's tooling will parse and use to resolve a git repository which can be 'git clone'd. Leaving them as-is when you abandon the infrastructure they reference literally means leaving dead links in your source code.

If it's not fetching from the URL then it's not really a URL, just an identifier that looks like a URL.

What do you mean? Go's tooling definitely tries to fetch from that URL.

Not if you tell it a different URL though

That's just the default, which you can easily override if you need to. IMO there is nothing wrong with treating Go package names as arbitrary identifiers like com.sun.* If you're spending a lot of time updating imports because the underlying repo has moved from its original URL, then just don't do that. It's unnecessary.

And leave URLs to abandoned infrastructure all over the place? No thank you.

Sounds like classic developer OCD to me. Old URLs used as Go package identifiers are completely harmless.

Until your tooling automatically makes a request to a now-untrusted server and trusts its response...

The Go tooling won't do that if you configure it properly (e.g. via a 'replace' directive in your go.mod or https://go.dev/ref/mod#goproxy-protocol)

Any tool that automatically assumes that a URL in a Go module import is 'trusted' is just a broken tool.

And if any tooling ever gets executed without the proper replace directive...

Any tooling that looks at Go module imports should respect the contents of go.mod.

If you haven't added a replace directive to your go.mod, then you certainly haven't replaced all instances of the old URLs with updated ones.

Go mandates lockfiles, called go.sum

It's a solution that keeps you working now and you can do the actual search and replace when scheduling allows.

AKA it solves nothing, just kicks the problem down the road.

welcome to fast moving companies.

> you can use the 'replace' statement in your go mod to change where the Go build system will try to pull the dependencies from

Heads up for anyone who doesn't know: this only works at the "top level". Any replace directives in your dependencies will be ignored[1].

So for example if you have a dependency tree like [main -> thirdpartyframework -> golang.org/x/net/http2], and thirdpartyframework uses a vulnerable version of `golang.org/x/net/http2`, you can't just fix it by patching the thirdpartyframework repository with a replace directive; no, because that would be too convenient. Instead, the replace directive needs to be at the main module, where it doesn't make sense and is inconvenient.

Even though I like Go, it really seems like they don't care about anything other than monorepos. As soon as you need to work with forks, mirrors, or even just private modules[2][3], the tooling actively works against you. Also using your custom module proxy is a pain.

[1] See: https://go.dev/ref/mod#go-mod-file-replace:~:text=replace%20...

[2]: If you've only used private modules hosted on GitHub you might not have noticed too much pain because the Go tooling has hardcoded behavior specifically for GitHub and a few mainstream forges. You don't find out about this until you try to self-host something like Forgejo on your own domain thinking it would Just Work(tm), but it doesn't, and now you're left wondering why tf it works with GitHub but not with your own forge instance.

[3]: I think there's no hardcoded code for SourceHut, so you might be able to experience the inconvenience by hosting private modules in there.

> Instead, the replace directive needs to be at the main module, where it doesn't make sense and is inconvenient.

Wait, what? If you decide, in your own application, to force all your dependencies to use a specific version of golang.org/x/net/http2, then obviously you'd want to be able to put this directive into your own application's source instead of going around patching 3rd-party repositories (that's just rude).

What about anyone else using your code? You can find and replace your own code, but do you have access to all of the code relying on your go library? Is it all yours? Customers? Other developers?

Even in the case where it’s an internal only Lu array, it can be complicated to refactor a library name.

Thank you for the concise explanation! :)

I did not realize it ran that deep - I'm used to distributing EXE files, not anything that would need to know internal dependencies like that.

I mean yeah, but other languages don't make it quite so easy to make this mistake. Once your Go project gets to a size where it makes sense for different files to live in different folders (which, given Go's folder-based module system, often happens at just a couple hundred lines of code), the most straightforward way which the tooling nudges you towards means putting GitHub URLs (or URLs to whatever git host website you happen to use) in your source code.

Or example.com/yourname if your module is never going to be imported from another project.

to clarify: I mean literally "example.com", not your own domain name

I do something like this in all my projects now. Except that I don't use example.com, I just use the name of the project. So if I'm working on a program called Frobnicator, I'll just have `module Frobnicator;` in my go.mod and do things like `import "Frobnicator/lib/blah"` in my source files. It works really well to be honest for non-library software.

But it's something you have to do as an active choice; the tooling, and your colleagues, will nudge you towards using URLs to your primary git host's web front-end. And it naturally doesn't work for libraries.

If the first part of the module path is without dot, it is assumed to be stdlib.

Many tools will fail if you break this contract.

Oh that's horrible. I've somehow never encountered it but damn.

yourname.internal/mymodule is even better. But don't expect to Go toolchain (especially "go get" or "go mod download") to work.

https://datatracker.ietf.org/doc/draft-davies-internal-tld/

I think this is one of the biggest missteps made in Go and in all honesty I don't think it was any more intuitive than going through a package manager. Probably less, looking back.

Everything else is fine. Generics, errors, whatever...

I think TFA is specifically targeted at Go because of how their package system works.

Yeah, the only problem is when your whole company decides to move to a different custom domain entirely, e.g. from contoso.com to xtools.cn (which doesn't even resolves if you're not on the corporate VPN), and half of your friendly teams, whose libraries you depend on, drag out their own migration by fiddling with their local /etc/hosts and .gitconfig and .ssh/config (yes, they also wrote bespoke scripts that do those changes to their CI machines; no, trying to run those same scripts on your CI machines breaks your own shims) instead of updating their source code.

oh man deep cut contoso ref (contoso being the fake company used widely in microsoft demos/documentation/examples)

I don't love that either, having all of your stack point to your own domains for how it can be accessed is nice. But in my opinion, it's desirable to be able to build your whole stack from source code. Of course you download external dependencies, but for code you own you should be able to build it from source.

I have seen too many cases of requirements on internal projects that prohibit devs from working because oops the vpn is down now or oops gitlab is under load and the pipeline for your dependency won't finish for the next hour.

I also think that vendoring or at least caching your dependency tree somewhere local, so you can build everything from ground up to a known state.

It's not only an issue of availability or depending on circumstances you can't control, but a matter of reproducibility hence dependability.

I want to be able to compile critical software when a zombie apocalypse hits, and I have a growing discomfort about trusting outside parties for their availability and benevolence.

In the new world of AI and all the supply chain attacks on centralised package managers, I think circling back to vendoring probably makes more sense than depending on the network in your build step. Which in some ways is always great, as you mention, in terms of working offline. Using your own domain doesn't help there either, except exposing the package if the domain expires.

Even if you don't want to go as far as vendoring (which can get a bit out of hand with, say, NPM), a middle ground is using Artifactory or whatever as a proxy.

Then be more deliberate about when you update. AI may even help here where static analysis doesn't, because you might be able to use inference to see if a dep upgrade is even needed. Unless it has a severe vuln you probably don't need to track latest.

This is hiding in the obfuscation of being overly specific. The problem is general. To name a package, one must be in a namespace of some sort. As the universe does not provide any sort of abstract ambient "namespace" we can all appeal to, all namespaces are human constructs. As human constructs, they may fail.

In this terminology, the article basically amounts to saying "This namespace can fail! The solution is, use another!"

But that doesn't get you anywhere, because the new namespace can fail too. In fact it is almost certain that it will fail sooner than "github.com's DNS and hosting" as a namespace.

There is no solution where you tie your software release to a namespace that can't fail because there is no namespace that can't fail. It doesn't matter if your favorite language uses DNS or has a centrally blessed repository or if it distributes a blessed list of package names with the language itself or anything else. The namespace can fail.

Therefore, the only thing you can really do is be resilient against failures, and in a lot of ways, the only practical solution to resilience is just to assume that if, in the future, someone has problems getting a package due to a namespace failure, they will not helplessly disintegrate into a puddle of tears while your code is lost forever, but that the future person will instead solve this perfectly solvable problem.

The article doesn’t mention any of these namespaces failing as the major issue… he is referring to the challenge of transferring hosting providers, which could be done for various reasons.

That is a failure; the old namespace no longer meets your needs. My point is abstracted over all of that. There is no namespace solution where that can't possibly happen. Switching from one solution that has problem X (and more generally than the article describes) to another that still has problem X isn't a solution, and given that problem X can't be solved, that means there is no solution. In the interests of not spamming replies, this is also my reply to dcreager.

Yes, but the solution differs. As the author says — updating DNS entries can be considerably less complicated than migrating libraries and their references

In this example, the old namespace is perfectly fine. The issue is the hosting provider. The point is that you can (and probably should) decouple the namespace from the hosting provider.

There are any number of other reasons besides resilience that you might want to change hosting providers. I do this, and I'm not worried about the resilience of GitHub at all. For me, it's a question of branding: I want to be in control of this code's identity, and don't want that to coupled with or coopted by the hosting provider I'm using.

There's actually a solution, it's NostrGit, decentralized git. No single point of failure, and your identity is a public key. You can also just host your git via http on a Tor onion or i2p eepsite to get a public key identity namespace that is also resilient to the operator of your namespace registrar trying to sabotage or subvert you.

Git is decentralized all on it's own. A relation that will not be named had a complete infrastructure failure that destroyed their version control servers as well. Everything was up in a few days pushed from developer laptops.

It's not a matter of whether or not something can fail; it's a matter of whether or not you can fix it when it fails. As we've learnt recently, Microsoft is finding it increasingly difficult to keep GitHub running.

This is the most useless critique of a problem you clearly don't have a handle on I have read today. good job!

It says the solution is use a namespace you control.

> it will fail sooner than "github.com's DNS and hosting"

When it does fail you can fix it. When github fails all you can do is stare at their status page.

> That is, if you move your git hosting to GitLab then you have to change your code!

You can also just use "replace github.com/example/example => gitlab.com/example/example" in your go.mod file and everything will keep working. That seems like a very pre-mature optimization for something that doesn't really matter.

And you'll just leave references to abandoned infrastructure in your code forever?

cmd+shift+r in goland, push to a branch, open a PR, ask devops to make the old location archived and prevent further writes. After a couple months remove access to the old one and see what breaks.

You know, normal development task, small amount of story points..

You're now arguing that it's easy, which is a completely different argument. The comment I responded to argued that it's unnecessary ("just do the rewrite in go.mod, leave source files unchanged").

For my thoughts about why it's not easy, see this comment: https://news.ycombinator.com/item?id=49870363

Yep that sounds pretty bad. Personally, we'd just eat not being able to build old versions, we archive previous builds, and in the case of a hotfix or whatever we'd use the go.mod rewrite trick. Not sure why you needed the commit/branch setup, did you change the history?

Have libfoo with a bunch of different versions. Authservice depends on libfoo 1.2.0. APIservice depends on libfoo 1.3.0.

Now you want to move from github.com to git.example.org. You change libfoo, authservice and apiservice to use git.example.org, that part is just simple tedious work. But the v1.2.0 and v1.3.0 tags of libfoo are old commits from before the move, so they still reference github.com! Now you need to branch off of the v1.2.0 and v1.3.0 tags of libfoo and do the same change there.

So we have only 3 repositories with only 2 dependencies and we already have to do the search/replace 5 times.

Imagine now that apiservice depends on authservice v2.3.7 just to include some type definitions. Authservice is currently on version 2.4.0 but the types haven't changed so apiservice hasn't upgraded its dependency. Now you need to branch off of authservice v2.3.7 too and do the job there. Oh and authservice v2.3.7 depends on libfoo v1.2.5, so now you need to make a branch off of libfoo v1.2.5 with the search/replace.

3 repositories with a straightforward dependency relationship, 7 search/replace jobs.

The numbers get terrifying as you scale this up.

What about everyone else using your code? You are going to ask them all to update their code as well? Many of them probably won't realize, and will get stuck on an old version that never gets updated, even to fix bugs and security issues.

By this logic, those folks should be pointing to their own hosted mirrors of the project, no?

I think the migration can be made pretty easy for users with deprecation notices and the `go fix` tool. Better explained in [1] (I have not tried it myself).

The idea is: the package maintainer pushes a final version of the old package that is a shim, and which has a deprecation notice. The shim imports the new package, and forwards all calls to the new package (and I suppose type aliases and variable aliases as well). The shim includes `//go:fix inline` annotations on all exported symbols, so that when users run `go fix` it rewrites their code to use the new package, by inlining their usages of the old package (which is now simply a shim that references the new package).

Not perfect. There is still a bit of a discoverability problem. Not all tooling warns on deprecated packages/functions (go toolchain doesn't, but gopls and staticcheck do). And users need to know to run `go fix`.

1: https://go.dev/blog/inliner#example-renaming-ioutilreadfile

As library maintainer, you can mark the last old domain version as deprecated:

    // Deprecated: use example.com/mod/v2 instead.
    module example.com/mod

As library user, it's your responsibility to keep your stuff updated. There may also be utilities in the various automatic dependency updaters that can migrate these things.

why not just use replacements to begin with if you don't want to bother with setting up domains?

"replace example => github.com/example/example" at first, then change to "replace example => gitlab.com/example/example" or similar when there's a migration

(i don't use go, but i hope this is possible and i am confused if it's not)

Yeah, I think putting github.com in your repo is a mistake but it seems massively overstated. Seems trivial to fix all the code, or to just use go.mod, or first use go.mod and then fix your code.

GitHub is almost forever. Your custom domain disappears when you stop paying the bills, which if you’re an open source developer has a higher likelihood than GitHub disappearing.

One day, we’re all going back to vendoring dependencies.

Why did we stop vendoring dependencies in the first place?

Because its not as convenient

Can you elaborate?

Updates mean you have to copy over all the code into your repo, which creates a large diff, and hope you aren't overwriting any local changes someone might have made.

It bloats your repo, both with the actual code, and the large diffs when you update it.

You have to manually track new versions, without something to tell you if new versions are available, or if your version has known security vulnerabilities.

If the dependency has it's own dependencies, you have to vendor those too recursively. And if multiple dependencies have the same transitive dependency, it is up to you to deduplicate them, and make sure you have a version compatible with all dependents.

Etc.

Disk is cheap.

Recursive dependencies have the same issues whether you vend them or not.

Ensuring that diffs to updated dependencies remain within a vendor folder is trivial.

So, what’s left?

> Disk is cheap

The biggest problem isn't (usually) disk space, or network bandwidth, it is that git operations slow down as the size of the repo grows. And it means that cloning or pulling the repo takes longer, which can be especially problematic for CI.

> Recursive dependencies have the same issues whether you vend them or not.

Package managers usually handle resolving recursive/transitive dependencies for you. Some have support for vendoring dependencies, but not all do. In theory, you could have similar tooling for vendoring dependencies, but in practice that often isn't the case.

These are all solvable problems IME. But it does mean you need people who are experienced at solving them or who care enough to learn.

> But it does mean you need people who are experienced at solving them or who care enough to learn.

That sounds like it's inconvenient to me.

The price of security and business continuity is the occasional inconvenience. Tale as old as time.

Vendoring isn't a clear win for security. It may provide some protection against "supply chain" attacks, but makes it more difficult for you to respond to vulnerabilities discovered in the version you vendored. And for business continuity, in many cases depending on an external registry is an acceptable risk.

And both of those concerns can be addressed by using an internal/private registry that mirrors the packages you need.

But in any event, your original question was to elaborate on why vendoring is inconvenient. Whether or not the benefits are worth the inconvenience is a different question, to which IMHO the answer is "it depends". Sometimes it is, and sometimes it isn't.

> Vendoring isn't a clear win for security. It may provide some protection against "supply chain" attacks, but makes it more difficult for you to respond to vulnerabilities discovered in the version you vendored.

How so? Whether a dependency is vendored or not, you still have to update it to integrate a security update to that dependency, don't you? The alternative is to not pin your dependencies, but that is far riskier overall.

> Whether or not the benefits are worth the inconvenience is a different question

I would contend that it is the most important question. :-)

> Disk is cheap

If I want to upgrade the disk on my MacBook Pro, I need to buy a new MacBook Pro with a larger disk. If I want to upgrade the disk on my work laptop, I’m SOL.

> Ensuring that diffs to updated dependencies remain within a vendor folder is trivial.

It’s not obvious to me how putting the dependencies in a vendor folder solves the diff problem. Does every code host allow you to hide diffs to certain directories?

And what’s the advantageous scenario for vendored dependencies? Is it just when the mod proxy and the upstream code host go down at the same time?

> If I want to upgrade the disk on my MacBook Pro, I need to buy a new MacBook Pro with a larger disk

Is this a significant risk in reality? MBPs today come with a minimum of 1TB of storage. Even 5 years ago I think the minimum was 256 GB. This is more than large enough for all but the most massive repositories, even with vendored dependencies. And you can always plug in external SSDs or HDDs or connect to a network server.

> And what’s the advantageous scenario for vendored dependencies? Is it just when the mod proxy and the upstream code host go down at the same time?

This is a useful homework assignment. Ask your favorite LLM or consult some respected release engineering books. Also consult your local AppSec and infosec teams.

> This is more than large enough for all but the most massive repositories

Consider the possibility that the drive needs to accommodate more than just a single repository?

I have lots of software, entire language toolchains, AI models, Docker images, application volumes, VMs, etc.

And this isn’t a theoretical concern; I have to free up space every few months.

> This is a useful homework assignment. Ask your favorite LLM or consult some respected release engineering books. Also consult your local AppSec and infosec teams.

So there is no advantage scenario that you’re aware of?

I’m aware of the advantages. I’m choosing to let you exercise your innate curiosity to learn them for yourself.

Lol this seems like a very believable and normal thing for a Knowledgeable Person (TM) to say

Because it sucks. But maybe it should suck? It would make us think twice before adding dependencies.

I use dotnet and I never liked seeing dlls and binary files in my diffs. I would argue if we are adding vendor code to our projects, we should demand the FULL source code instead of dlls. Maybe it is already possible with things like x unit. I have never given it much thought... But then that vendoree code has to come from somewhere as well, right? I mean there is something to be said about provenance or something here?

Sorry if this feels like a stream of consciousness because it is ↔

Vendoring dependencies doesn’t necessarily mean you have to include binary objects in your repo. They could be content-addressable artifacts in your org’s private blob store.

i.e. they could be NIH git-lfs.

I think its a good question. The short answer is: because tooling doesn't default to this or make it easy.

The answer to the question of why THAT is the case - is not so easy to answer. The main benefit I can see with using lock files instead of vendoring is that it saves a lot of storage and diff history from entering your repository. So clones are much faster, backups smaller etc.

I think Go used to work this way (automated vendoring) but it’s the only language I can think of that ever did in terms of standard tooling. It would be instructive to learn why that changed.

We didn't!

I vendor all my Go dependencies all the time for every project. It is the way.

Has this ever been necessary/useful? Genuinely curious because I’ve been using Go since 2012 and can’t recall a time when vendoring solved a problem better than “regular” modules. Like have there been times when the source went down and the module proxy went down (or didn’t have your dependency cached)?

In the case of a typical software enterprise having your domain gone means that you probably don't care about the code anymore anyway.

Unless the field of software development somehow totally stagnates, I'm confident that at some point I will move to a different hosting service, as github goes the way of sourceforge, travis-ci, and myspace. So my personal domain will continue to be relevant (it's lasted fine 23 years).

Maybe at that point it won't be using git either, or maybe it will - who knows.

Mine has lasted over 25 at this point, but one day I will die, my domain name will lapse because what I do as a career might as well be witchcraft to anyone in my family and then my modules will be lost forever. Sucks any way you cut it. Hopefully someone who found my modules useful will fork whatever they find and keep them going.

go mod vendor... works for me. Also it makes sense for an ecosystem like Go with strong standard lib and strong backward compatibility commitment.

Also: https://proxy.golang.org/

True, but beware of the domain name you're using. Because VeriSign may unilaterally decide to delete your domain name along with thousands of others [1] and you're back to square one…

[1] https://neil.fraser.name/news/2026/09/03/

I broadly like Go's "the import is the hosted location (or a pointer to it)" quite a lot, as it largely solves name-squatting and ownership and a lot more (while allowing major risks with domain sales/abandonment), but yeah - I really do wish they baked a SHA into the go.mod (not just go.sum) so you could find a library and get a known-good download from any proxy with any name. A few languages now have content-addressed imports/packages, instead of just adding hashes as verification, and I hope we see more in the future.

Signed modules / including the signature hash would also solve a lot, e.g. it'd mean domain sales no longer silently inherit full permissions. It's sorta a shame that Go keeps doing such a good job at a minimum-viable wheel-rewrite, but then lets it linger for so long without catching up to the rest of the programming world.

> It's sorta a shame that Go keeps doing such a good job at a minimum-viable wheel-rewrite, but then lets it linger for so long without catching up to the rest of the programming world.

How many mainstream languages have content addressed imports? I can’t think of any, so I assume I’m misunderstanding your meaning of the term because you seem to be suggesting that it is common and Go is the outlier for lacking it?

Go is not really an outlier for not having signed packages (there are a fair number that have it, but far from most)... but definitely stuck behind common accepted practice. By decades, if comparing against some (e.g. Java).

Which keeps happening with stuff they rebuild from scratch - an excellent and somewhat unique first showing, far beyond what most first attempts manage, but followed by near-complete stagnation while issues that everyone familiar with the field predicted from miles away pile up.

Coming from the Java side this is typical of Google libraries. They start off impressive and gain wide adoption but then they stop supporting changes the community wants, missing basic features.

Yeah, it still very much tastes like Google in many of the worst ways :/ clearly Google isn't actually running the project, it's far too well run for that, but the same general "why would anyone need [that thing nearly the entire open source world does outside Google's monorepo]?" ignorance pervades a lot of it.

Which is a shame because there is quite a lot to like about Go in practice. And in spite of it all I'm thrilled that it is eating into Python's share in a lot of places.

> I really do wish they baked a SHA into the go.mod (not just go.sum) so you could find a library and get a known-good download from any proxy with any name.

You already can use SHA in go.mod in exactly the same way you would use a version string.

I don’t know about using multiple proxies in go mod though.

Kinda. If there's a versioned release at the same SHA, the next `go mod tidy` will replace it with the version.

And while the go.sum file in a module is returned by proxy.golang.org (somewhat surprisingly), that only includes the module's dependencies, not itself. So you're still stuck trusting a goproxy to serve you the correct data.

At least the fallout will be less if you have to resolve your custom domain differently in the intranet due to something like this.

If github is "taken down" you can't just resolve the whole domain differently, you need to only resolve the packages differently. If your "Golang domain" is taken down, it should be a lot easier to hotfix until something proper is implemented (the quickest and dirtiest would be a hostfile-entry).

reading the article confuses me though I don't want to do the context switching by googling what 3rd level domain is about

All you need to know is that for some reason you could buy foo.bar.com without owning bar.com directly from the registrar.

It is being retired because very few people use it. It kind of sucks but this is how domains work, it’s not real estate.

Or if your goal is to open source code, think about the lifetime of the entity owning the domain: the source code might be of interest to users beyond the control of the domain name. Domain controled by an individual or a company? What if the domain is not renewed, or taken over by an hostile entity?

Deepcopy [1] is a Go package that is still heavily used despite its creator has disappeared 9 years ago. At least GitHub is a stable and trusted host as a distribution point and communication point for users.

[1]: https://pkg.go.dev/github.com/mohae/deepcopy

Great when a company goes out of business and the domains are dangling. Next one to scoop it up takes over source code that others new depend on. We've seen this many times already in other ecosystems. It is a very bad situation.

It is bad advice to move your packages under your own domain. You will never be as good as Microsoft to keep paying for the domain. There are no guarantees in life but I do guarantee you that when you go out of business, that domain is the last thing you will think about.

Thinking this through, using my domain might actually be much less risky than using a host's. If Gitlab closes down, and someone else buys the domain, then I'm forced to make a massive refactor with little notice.

If I own the domain, I'm likely to continue to owning it until I shut down my company. At which point, no one is likely to be running my code.

Possibly different if I was developing an open source library or something, but either way there's risk and need to be thoughtful of the risk versus impact.

Well, actually you don't download from GitHub - you download from the central mirror proxy operated by Google, and that will always return the same package given the same versioned reference, even if the original source differs from it.

Seems like an error to use a URL. This is perfect for a URN or some other form of URI.

It could be a URN that used the DNS as a back end (though that’s a lot like a URL) so better would be something more abstract with multiple possible resolvers and a signature.

The only useful alternative you're talking about would be a content-addressed one. A "URN" that relies on some kind of mutable registry lookup is actually a URL (both are URIs).

> One of the good features of Go is that you namespace your code with the location to fetch the code.

Then the rest of the article explains why this is NOT a good feature in practice.

I don't think it's unworkable either, but this is one of these little thing that Go decided to do different and convinced its fans that this is a great idea and all the other languages where doing it wrong. After a couple of road bumps appeared, instead of admitting there are some advantages to having official package names, we're now told that everybody should just set up their own custom domain with an nginx server or a Go Vanity URLs forwarder to serve traffic for their GitHub-hosted packages.

combine these domain aliases with the caching goproxy you're supposed to use, and they're well on their way to reinventing a traditional package index from first principles

This is great and I think for any company/individual who is going to ensure their domain is registered and maintained this makes a lot of sense. It would be catastrophic, but there is no reason Github couldn't disappear or otherwise change some policies that require moving away from it. The way Go works makes committing these package names tied to Github so much more weighty than simply the place you pull from.

The one thing that I was worried about was returning 301 in the example Nginx config. If you ever wanted to change the url that clients are redirected to then any browsers that visited the old url config would be forced to go to the old config's redirect url. For `go ...` and `curl` it wouldn't matter, but Chrome/Firefox will cache that 301 permanently and break the intended redirect. Not sure if this is really an issue in practice though.

Package management in Go always felt pretty hacky. Among the issues is the confusion between "what something is" and "where something is". At the beginning there wasn't even a way to have different versions of a package, with the justification that you should just fix an interface and keep it stable till the end of time, there was also no way to "lock dependencies" so at build time you got whatever that repo served you. It has gotten better since, but still some jankyness as a result of "growing" a package system.

Isn't this going to introduce the dependency of domain management? I understand the positive side of it, but put some infra level manamgment layer to individual Open Source devs, just a thought. But yeah, positives vs negatives weigh and pick.

Here's the primer for those who thing find a replace is the answer. Pretend you have a project with A 1.2.3 depends on B 2.4.6 which depends on C 0.1.1. If you are in github you get module github.com/example/A requires ( github.com/example/B 2.4.6 github.com/example/C 0.1.1 )

You now have many choices on how to proceed, but none of them will include A) being able to build old releases, or B) doing so without making changes to all dependencies.

One choice is go to the leaves, C in our example, and make a release on gitlab. Then go to B, and have B depend on the gitlab C. This is fine for rolling forward, but if you want to roll back you would have to rewrite all of C to use the gitlab url, and find all of the equivalent tags and repush all of them. Also tell your users that any binaries you've released now have new hashes. Then repeat this for every repo. It's not a small task.

This tries to solve the hypothetical problem of a library designer moving their codebase to another domain and breaking imports. But, this rarely happens in practice, so unless you plan to move hosting in advance, it may not be necessary.

That said, in the normal run of things you can mark your last release on e.g. github as deprecated, leave a note that it's moved to xyz, and wait for the users to migrate themselves. Migrating is fairly trivial in that it's a find / replace.

I don’t think I would use a 301 permanent redirect for this sort of thing. If you later want to switch from GitHub to some other provider, software that aggressively caches permanent redirects (like browsers) will remember the old redirect to GitHub, especially since you don’t have any cache control headers set. Maybe not as large a concern will CLI tools, but something to be aware of as this has bit me in the past.

I'm not sure that custom domains are a good idea. It's way too easy to end up with broken links once vanity domains go away.

However, Golang saves the hashes of all the modules in `go.sum` files, so we just need to add a way to do content-addressable fetches. This solves the issue of reproducibility for old builds. As long as you can find the module in some repository somewhere, you'll be able to build it.

The next step is supporting module _evolution_. We need a way to declare: "From this point onward, `github.com/company/someproject` is now `company.com/someproject`", so that all the references are to these packages are identical. This is possible on a per-package basis with `replace` directives, but this doesn't scale.

And this is not easy to solve in general (especially in the age of supply-chain attacks). If the initial project cooperates (or if Github can be convinced to help), perhaps at least a part of this can be solved by adding special "redirecting module" support to Go.

A GitHub URL at least is still an credible identifier, your custom domains is not, and likely will never be given how the system is accustomed to. Domain is for branding, not identification.

Unless of course, it's special domains that has identification built in, such as .onion which is generated in such way (cryptographic keys) no other people can easily obtain control even after the domain is no longer maintained.

If you really don't want to use GitHub, an alternative is just use .internal suffix (i.e. yourproject.internal/project) in combination with `replace` directives in go.mod. But that require your user to manually download/install your package and then edit their own go.mod.

> A GitHub URL at least is still an credible identifier, your custom domains is not, and likely will never be given how the system is accustomed to. Domain is for branding, not identification.

The post is about commercial software. If enterprise’s domain is not credible enough, then there are bigger problems.

I'm guessing by commercial software, you mean GitHub etc.

Well, I too dislikes their enterprise tune, "Changing is consistent, GitHub give you advantage (over other companies)" blahblahblah. Many developers I know are on GitHub because it was/is the place for individual developers to share code, not because the "company advantages".

HOWEVER, I can't ignore the fact that individual developers just can't pay enough to keep the machine running, so it's reasonable for GitHub to pivot towards commercial market. On the other side, GitHub is still benevolent enough to provide some enterprise-class service back to freetier users. This arrangement checks out for me so far.

I do self-host some of my projects too with Gitea. But I never dare to use my domain in my Go packages, due to long-term security concerns.

When I initialize my projects, I just use .internal domain, then if I decided to upload the project to GitHub, I'll change the URL to GitHub to make it more convenient to the users. Otherwise, manual download and `replace` directive like I suggested.

It's a rather internet focused language. Even if you don't point it to github, some other packages will and you still have to point it to something (i.e. gitlab, codeberg, or your personal git server). Then what happens if those servers are no longer available?

Go defaults to using a caching proxy, but you can use your own too.

Just like [^1] says:

> After spending years doing those FFI dances in both directions, I’ve reached the conclusion that the only good boundary with Go is a network boundary.

It looks like this extends into the package manager too. The only good boundary with Go programs is a socket. The only good boundary with the Go package manager (? if we can say it is one) is a domain and a DNS server (your own or somebody else's in the case of gopkg.in).

[^1]: https://fasterthanli.me/articles/lies-we-tell-ourselves-to-k...

fun fact, in the process of designing CUE modules, the Go team said they would use a registry if they could do it over, CUE uses OCI registries

MVS + OCI is the gold standard imo

CUE module proposal (implemented a while back) - https://github.com/cue-lang/cue/discussions/2939

MVS, the fast and deterministic dependency resolving algorithm Go uses instead of SAT solvers - https://research.swtch.com/vgo-mvs

Even better reason: you can later point this domain at an artifact registry. This not only gives you reliability and flexibility, it also secures your software supply chain. You don't need an SBOM or anything fancy to get started, just pull all your artifacts into a central source and improve it over time. Install an artifact registry anywhere you can run a container, use dumb static shared credentials, and start with "proxy mode". Later on you can pin or restrict versions, verify checksums, implement SSO, etc. This is going to become table stakes in the new security landscape.

You don't need an SBOM or anything fancy to get started

-- Curious how this avoids SBOM need?

An SBOM is an inventory; you don't need to make an inventory to pull files through a proxy.

Correct, that is what confused me, how SBOM came into the picture, now got it, thanks!

Thanks for answering my question! So yes, if I intend to use go and study the packages I'd have to rely on beforehand, if they're all hosted on GitHub should make me steer away from Golang. Or at least plan to cache the packages I need locally so CI always have something to work with when building.

Alternatively, could we ourselves build an automatic mirror so there's a redundant supply provider that doesn't depend on a maintainer's choice of git forge

https://news.ycombinator.com/item?id=49434625

https://proxy.golang.org/

This should be the first thing to learn when starting with Golang.

Let's take a project I worked on for a couple of years some time ago, it has this go.mod file: https://gitlab.com/nunet/device-management-service/-/blob/ma...

Is it just changing all instances of `github.com` to `proxy.golang.org`? Shouldn't `go build` warn about using third party proxies for dependencies? Should `go get` offer a flag to automatically use the proxy when adding dependencies?

That's all well and good, but what happens if you drop off the face of the earth due to personal issues, health issues or worse and your domain expires?

There are also a few Hugo templates for vanity import paths.

Here's mine: https://github.com/foundata/hugo-theme-govanity (e.g. used at https://golang.foundata.com/ )

And yes I'm aware of the irony of hosting this on GitHub... still figuring out a good workflow for maintaining our OSS on Codeberg and GitHub in parallel fed from the internal forge. The dependency on our own domain is real but we favor it.

"Package manager"-like behaviour like this is evil. Importing software/code at runtime from the internet (like Apt, Go, Pacman, Pip, NPM, etc.) is the equivalent of those dreadful online stub installer packages, and it's impressive it's as trusted a mechanism as it is.

Leave the Go code or Linux distro to link-rot for a couple decades and all imports like this will be dead or serving malware, leaving the software unusable.

Holy cow. When I was learning go I thought I was the only one crazy to think that is not a good default

I always wondered when I moved from NodeJS to Golang on why do I need to import everything from github. It would have been simpler to have a central registery to host your packages and import from there I think with secrets to read the private ones

i would like to talk about the article's main point of not tying things to github only , so it’s basically distribution vs theoretical decoupling. Sure, tying the module path to GitHub creates some lock-in, but GitHub is where the community and visibility already are.

"This namespace can fail! The solution is, use another!" while i agree with this but setting up different namespace just because you want to avoid vendor-lock , we need to think from maintainer's perspective as thi would introduces frictions and so are of the most open-source maintainers willing to take it or not?

Thanks for the article, it inspired me to come up with my own solution (and blog post):

https://a.l3x.in/blog/vanity-go-import/

> That is, if you move your git hosting to GitLab then you have to change your code!

Not to be cynical about this but I fail to see how running sed s///g after a very rare event qualifies as a serious problem.

Now, sure, given that the solution is so simple, it's a nice recommendation, but still...

I've been saved by the commit hash in the dependency when reviving an old Python project. I could locate a fork of the repo and use the same version to get the code going again. I ended up vendoring the dep.

I think we need better handle to the original location/author/project than a domain. Any of such "go" proxy is yet another SPOF.

We should lean towards IPNS (Name Seever part of IPFS) or some distribited ledger.

migration concerns is a task of any coding agent for 2hours with testings, a cheap one for such an easy task, grok, gpt 6 luna can easily do it with almost no tokens cost.

So you need a domain and an nginx server running somewhere? That seems like more of a liability than just controlling a domain. Isn't there a way to configure this this just using DNS and no web server?

So what happens when the domain provider gets bought and the domain in your OSS project gets auctioned and compromised? This is turtles all the way down it seems..

I have not worked with Go, so I thought I'd ask: why wouldn't simple find and replace be able to fix this? And why wouldn't a coding agent be able to do this for you quickly?

See: https://news.ycombinator.com/item?id=49870363

We hit this after an org rename. replace covered our modules; transitive imports of the old github.com path still broke builds.

How many other people have hit the pain of injecting ssh keys for gitlab into a docker container building your code in CI?

Compiling code shouldn't hit network by default.

why you need ssh key for CI? The CI_JOB_TOKEN or scoped PAT also works via http to pull/push the code.

I fear one of these days someone will copy a legit package and make it onto your search results, and then the Go tools pull in something extra.

Definitely had a private GitHub dependency disappear once. Vendoring saved us a massive headache downstream.

That's a beautiful solution. I love it! The alternative work went with was to just build everything ourselves.

Another way to fix this kind of issues would be to fallback on the Software Heritage which archives the whole GitHub.

This is something Nix and Guix are trying to achieve: by default, they fetch source code from the original endpoint, but if it doesn't respond anymore, they would query the Software Heritage archive [1].

Because go mod doesn't use SHA, it makes things a bit more complicated but Nix/Guix encountered the same issue and there are some mechanisms to workaround this issue.

[1] https://www.softwareheritage.org/2025/05/21/software-heritag...

braid has been a treat for us to manage external dependencies. Much cleaner than submodules.

Honestly google should have always had a package index even if it points tooling to the repo, but it would maintain the package name with the ability to change where the repo is.

I disagree, prefer to see directly which forge a module is hosted on.

Now you coupled it to 2 services you don't own, congratulations on making it worse.

Bonus points if DNS mafia deems your cool TLD to now cost 10x times more coz it's "premium" and you're stuck paying up or telling everyone to migrate

Not really convinced. I'd rather it stayed on Github so it doesn't disappear. Particular for businesses whose priorities might change.

> I'd rather it stayed on Github so it doesn't disappear. Particular for businesses whose priorities might change.

People can delete projects from GitHub. Businesses whose priorities might change may not keep an old project set to public up on GitHub because they won't want people to continue contacting them for support, or they don't want to be responsible for updating security vulnerabilities for projects they are abandoning so they'd rather just pull it offline, etc.

Whether the library you're importing is hosted on GitHub or not, never assume it will be there tomorrow. *Always vendor your dependencies.*

I believe they'd also be cached by proxy.golang.org? But companies should probably run their own proxy.

You shouldn't be using fully qualified domains at all. Use relative paths instead, like sibling-hosted git submodules.

I don't think Go allows relative imports.

Reply on news.ycombinator.com