Hacker News Reader

Comment by na85 | Hacker News Reader

I really liked using Go until I tried to publish a module with a version greater than 1.

What kind of packaging system chokes on v2 of a module? What kind of language recommends[0] with a straight face that you just duplicate your entire codebase into a v2/ subdirectory?

The module system is so obtuse and the documentation so poor that I will probably avoid go and choose other languages from now on. It really feels like they came to release time and someone in the meeting said "hey guys what if someone wants to release a new major version of their library?" and everyone else in the room had an "oh shit" moment because their amateur language design didn't address that corner case.

[0] https://go.dev/blog/v2-go-modules

View on HN

> What kind of language recommends[0] with a straight face that you just duplicate your entire codebase into a v2/ subdirectory?

Not Go at least. You misunderstood the post completely.

Here is an example of how you are supposed to change the import path for subsequent major versions (that is, backwards incompatible versions): https://github.com/jackc/pgx/blob/3599f646293c1b0d381214ab26...

A one-line change to your modfile and the import path is changed.

Thanks for this informed comment. A timely reminder to me to read TFA, because I took parent comment at face value on first reading.

That's incorrect. The go blog recommends copying your entire project into a v2 folder. They do suggest you can use tags/branches instead, but it will cause problems with GOPATH projects.

It recommends this, sure, but there's no need for it. Doing so would be purely for backwards compatibility with older versions of Go tools that don't understand modules and rely on GOPATH. If you don't need to support older tools, then there's absolutely no need to copy anything.

Maybe I'm missing something, but I think there is a need. If your code imports dependencies A & B and they depend on C at two different versions, how can you handle this unless C is structured with versioned dirs?

Go does not map the module path directly to a file system path. You can import v1 and v2 at the same time because they have different module names. In other words, this is allowed:

  $ go get github.com/linkedin/goavro@v1.0.5
  $ go get github.com/linkedin/goavro/v2@v2.10.1

Then:

  $ go list -m ... | grep goavro
  github.com/linkedin/goavro v1.0.5
  github.com/linkedin/goavro/v2 v2.10.1

Version 1 has a go.mod that declares:

  module github.com/linkedin/goavro

and version 2 has a go.mod that declares:

  module github.com/linkedin/goavro/v2

but both have their Go files located in the root of the repository.

GOPATH is luckily soon dead.

Which is kind of remarkable. Python still seems to try to support every package manager and format that ever existed, and that’s part of why packaging in Python is so awful. Nice to see a language that can move forward with such a major change in such a short time and with relatively little pain.

Python is a lot older and a lot more widely used, so it’s not as easy to just change things.

Python3 is pretty close to Go in age and was a purposely compatibility breaking, so it could have easily changed things at that point and they definitely knew about the problem at that point.

The difference between 2 and 3 is way smaller than people make it out to be. I migrated a large Python 2.7 project to Python 3.8 and it wasn't a particularly painful experience. I feel that way more effort should have been directed at making 2 and 3 compatible.

It took more than a decade for the ecosystem to migrate. IIRC, a lot of effort went into making them compatible, but some things are just fundamentally incompatible and also very difficult to automate (e.g., string encodings).

I learnt Python in 2013, when Python 3 still had some serious growing pains. I switched from 2 to 3 almost overnight circa 2015 or 2016.

The last few years I cannot help but feel they were very close to making 2 and 3 play nice together, but the idea ironically lost traction because of how much Python 3 adoption had accelerated.

Python 3 is easy for a Python 2 programmer to pick up, but that doesn’t mean it’s easy to port large systems from one to the other.

Yeah, that's the thing about having so little existing software, especially stuff that has been essentially finished for a decade or longer.

If Go is successful for any significant period of time, that will change.

Go is pretty established. By the time this module change was made, a huge swath of the cloud ecosystem had been written in Go, including core components like Docker, Kubernetes, Terraform, etc. It’s also the default language for which Terraform and Kubernetes extensions are written, as well as the default language for container infrastructure (i.e., container runtimes, etc).

I think Python’s packaging story is a bummer in part because Python leans so hard on C which doesn’t have a standard build system or a sane approach to building or packaging.

> Go is pretty established

Sure, and when it has been “pretty established” for a couple of decades, it’ll have a chance to have the community weight of important legacy projects that Python has.

This is a canned argument for any criticism of older programming languages.

First of all, it’s not much of a consolation to users that the packaging and performance are bad “because the language is old”, and secondly Python had already had half a dozen significant changes to its packaging format and ecosystem by the time it was Go’s age (distutils, cheeseshop/pypi, PEP241, setuptools, eggs, easy_install, etc).

Note that by this time, Python’s package management was far more complex than Go’s and it was also far more painful (and indeed, still is far more painful). Note that Go deprecated GOPATH when the language was already 9 years old, at a point when it was quite a lot older and more widely used than Python’s myriad package formats which are still supported today.

We can say “No fair! Go has the benefit of hindsight!” and true enough, but again as a user, I don’t much care—I just want the tool that’s going to solve my problem with the least headache.

And this isn’t even touching on the Python 2-3 saga, which began when Python 2 was merely 8 years old and lasted for more than a decade. If you want to argue that we should use the Python 1.0 release date for our comparison, then fine—call me in ~4 years if the Go ecosystem is on the precipice of a suite of breaking changes which constitute an existential crisis. My money says that wonky happen.

> This is a canned argument for any criticism of older programming languages

No, its a an argument that applies to the ease of making disruptive changes to languages with less legacy in the ecosystem (age is a factor, but not the only one; its also a reason why Ruby, which was a similar age with the disruptive 1.8 to 1.9 transition had an easier time than Python did with 2 -> 3).

It doesn't apply to all criticism, or even to “older languages” generally (at least not equally, though survivorship bias means it is likely at least somewhat applicable to older languages that it is interesting to discuss in the first place.)

Fair enough, but I don’t think we can argue that Go had considerably less “legacy” during its modules transition than Python had during its various package management iterations. I think Go is just quite a lot simpler than Python (or Ruby for that matter) especially in how it distributes software (just source code, minimal dependencies on C, etc) which makes package management a lot easier.

> A one-line change to your modfile and the import path is changed.

Another detail: since your import path change also affects internal imports between packages within your module, you also need to fixup a bunch of .go files in your repo (worst case all of them).

I still think it's not a big deal (it's all pretty straightforward non-magic). Just a small nitpick that it's not just one go.mod change.

Yes, any consumer must pick the new import path, although that is true regardless of strategy.

> Not Go at least. You misunderstood the post completely.

What? From the post:

> The recommended strategy is to develop v2+ modules in a directory named after the major version suffix.

> …

> We recommend that module authors follow this strategy as long as they have users developing in GOPATH mode.

Although with a reservation, they clearly do recommend it.

"As long as they have users developing in GOPATH mode."

It's documentation written when GOPATH was still a widely used for development.

I think nowadays it's less of a concern to have a module first code base with GOPATH unsupported. It's nicer to use a V2 branch. Or delegate V1 to a V1 branch and maintain V2 in master.

I’m not disputing that things may have changed since the post was written. But the comment I responded to claimed that the linked post didn’t recommend copy-into-subdirectory, which it clearly does.

How is a branch nicer than a subdirectory? Off the top of my head, it seems like backporting bugfixes might be easier using a branch (if you're lucky, you can just cherrypick the commit and it will work) - any other reasons?

I've been a long time golang user and while I like to give the golang maintainers the benefit of the doubt, I have also struggled heavily with go modules.

I think my main issue with it is that it does things implicitly without you telling it to. Running go commands will modify the go.mod file all the time, and this is largely invisible to you. As a result, your build is basically a moving target and seems to lose all reproducibility. It also makes the entire thing very opaque and hard to understand at times, since you need to understand the implementation details of go modules to fully know what's happening.

In comparison, take godep. While not perfect, it simply did what I told it to, and only modified the dependency state when I gave it a command. This meant I knew exactly what was happening in my build system, and could update dependencies when I chose to.

If anything, go modules is too smart for its own good... And I think this is a big problem for a build or packaging system. It needs to be explicit and simple. It's odd it was designed like this, given the language itself strives for these ideals.

I'm on the other end of the spectrum for this one. Breaking changes in a stable library (i.e. has reached v1) should be totally avoided.

Statically typed languages, fancy features like intellij's structural find and replace, they just don't cut the pain and annoyance of fixing an unexpected change in behaviour of a library in a large code base.

It's just horrific drudge work, utterly unenjoyable, thoughtless labour that needs careful attention.

The go module versioning is clearly designed by someone who has worked on a large many developer code base before.

I can have V2 and V3 linked into my binary at the same time - a blessing for incremental refactoring.

I think you have named the one good argument for this scheme: to allow a go module to exist in several versions in a project. Before the module system, this was just not possible. Which, in general, is a good idea. Programs today tend to have a too big dependency tree anyway, at least there should be only one copy of a module in the system. However, this is an idealistic view and depending on the number of dependencies, at some point it might not be possible to maintain this idea. npm on the other side goes totally crazy, every module import is local, you can have hundreds of copies of the same module in your system.

The go module approach seems to be like a middle line. It puts a lot of pressure onto module maintainers not to break things within a major revision. They seem to be aware that this is not always guaranteed, that is why the module system not only tracks the major revision, but the exact subrevision of the module. But at least it can enforce, that changes which absolutely do break backwards compatibility are on differen major revisions. Of which you are allowed several inside your project.

I really like golang, and have used it for years. But I really dilike gomod. It's not uncommon for using it to be a frustrating experience. It makes me glad that most of my work these days is not in golang.

The reason I like this is because of one of the principles that Go espouses pretty heavily - never break backwards compatibility. If I install v1 of your package, I want it never to break. You can always improve performance, add more methods, etc, but the normal Go upgrade does not involve any code changes at all. The tools formalise that for modules as well, and encourage building a new module altogether if backwards compatibility is being broken.

If it’s not being broken, it’s not a major version update, and major breaking updates are better off as new separate modules.

Except in practice in all real software (including the go language!) minor and patch versions break importers all the time in small ways, AND major versions are usually used to indicate significant new features, not just breaking changes.

So your claim is reduced to ‘I don’t want v1 to break my software in ways that the dependency finds important enough for a major version’.

Not quite as reassuring is it?

Plenty of security fixes and undefined behaviour fixes break importers all the time, and often importers and exporters disagree about the severity of that. Sticking with v1 will not save you that pain.

I preferred the situation before where there was strong social pressure never to break the main branch.

This is a cultural issue.

It's not inevitable that by developing a module over time that you constantly break compability AND you want all people that depend on you to upgrade version.

Go encourages the opposite approach.

If your API is not stable, don't publish a package for public consumption, or, make sure to clearly mark it as unstable, so people know what to expect.

One of the absolute best features of Go is stability. If I have a project written a few years ago, I can always run it against the latest Go compiler and know that it will still work.

Why does this matter? I'll give you a counter example.

In 2016 I wanted to work on a mobile application project and I just picked React Native because I heard it lets you easily create cross platform applications.

I worked on it for a few weeks, then left it for about two or three months. When I tried to pick it up again, I installed the latest versions of node and npm and upgraded all packages.

The project stopped working. I had no idea what went wrong. The error messages were obtuse and several layers deep. I tried to debug it for weeks to no avail.

As a result, I abandoned the project and built just an Android version using Kotlin and Android Studio.

It certainly is a cultural issue which is why it shouldn’t be enforced by tooling!

I also completely agree about the value of stability, which is why I don’t like this change which paradoxically encourages more churn (normalises v2,v3 etc for breaking changes, normalises large breaking changes and rewriting apis).

All APIs are slightly unstable in some sense (even adding fields can be a breaking change), so the aim should be to minimise version bloat and breaking changes and provide stability for importers, which the go language has done an amazing job at for the last decade.

To give you a counter-example the sengrid api in go has had multiple large breaking changes/rewrites and has used this v2/v3 etc scheme. It’s still a horrible experience as an importer and I’d rather they remained stable instead of introducing massive churn with the cop-out that the old unsupported version didn’t change.

In dependencies I import I want a perpetual v1 which doesn’t change over a decade and just slowly improves, not v31 - new improved rewrite this year! Which this rule explicitly encourages.

> All APIs are slightly unstable in some sense (even adding fields can be a breaking change)

Of course adding fields is a breaking change. It has never ceased to astonish me how little most programmers understand compatibility, I remember wrestling with libpng getting them to understand that e.g. no, "tidying up" the order of fields in a public structure you've published isn't safe back in the 1990s before they came to god and actually provided sane APIs which hide internal data structures.

Now, it's true that Hyrum's Law means any detectable change, even if it was never documented, will break somebody if there are enough people depending on your system. That's a big deal if you're the Linux kernel, or the HTTP/1.1 protocol, as it means even trivial "this can't break anything" fixes may break stuff. For example, as I understand it Google once broke search in Asia by changing an internal constant so that a hashmap would re-allocate (thus invalidating pointers to items in the map) slightly earlier. C++ code relying on pointers to just magically stay valid across inserts was completely wrong before they broke it, but anybody staring at a 500 page on google.com doesn't care why it broke they just want it fixed.

Most of us needn't much worry about Hyrum's law. That would be, as an old boss repeatedly told us, "A good problem to have" because it means you're having enormous impact.

> In dependencies I import I want a perpetual v1 which doesn’t change over a decade and just slowly improves, not v31 - new improved rewrite this year!

> Which this rule explicitly encourages.

I agree with the first part (I'd much prefer having a MyFunction2 in a package if necessary, than a breaking change to the package itself).

But my takeaway from the design was the exact opposite - that it discourages breaking changes, by making them a little less convenient, and that the purpose of the feature was to allow major versions to simultaneously exist to avoid build issues [1]. I think I got that impression by reading the long articles that the team lead put out about the design decisions [2], and observing that the standard libraries themselves rarely break compatibility.

Unfortunately watching various discussions some people do feel encouraged to put out new major versions since upgrades are "solved" (even though that's only from a build system perspective, you're still causing client churn).

[1] https://research.swtch.com/vgo-import

[2] https://research.swtch.com/vgo

Two problems here:

Major version nos are not in practice used for breaking changes but more usually for new features.

The unintended consequence is to make it more acceptable to add breaking changes and force upgrades to stay current.

It’s not a huge deal though and I’m interested to see how this decision works out. Personally I think it is none of the package managers business.

> I installed the latest versions of node and npm and upgraded all packages.

Laughter of recognition from this side. I am working on an application that has a tiny bit of Web UI. Basically one button, "Yes, I want this thing" and some text that guides the person staring at the button to decide if they in fact do want this or don't. I didn't write any of that code, and my Javascript knowledge is pretty rudimentary, but hey it's just a button right?

On Monday I'm sat (virtually) with the guy who wrote that part, pair programming the dependency injection stuff so that this button can actually cause stuff to happen. It builds and runs fine on my machine, we make lots of progress. During the week I focus on other parts of the system (including a red herring where we're convinced for an hour we have a DST Root CA X3 expiry issue but it's actually a syntax error in a Puppet file on the in-house node classifier), but on Friday we're ready to check the current state. However the guy who wrote the button code is out. No problem though right, I understand the larger system, just hit build?

Nope. I get a JSON parsing error, which, it turns out, is how npm or node communicates "I am the wrong major version" because of course it is. Incremental improvements to a single page with a button on it had resulted in needing a newer major version in less than one week.

The problem here is that we add too much extra meaning to version identifiers. In semantic versioning, if you're at 1.38 and want to release a cool new version with many cool features and a much faster engine, while still retaining backwards compatibility, you're stuck with releasing a 1.39. If you call your new version 2.0, you signify that it's backwards incompatible.

We should have two differently-looking version identifiers, one for compatibility-tracking purposes and one for signifying the scope of changes to humans. The compatibility version numbers could look like 2ah4, where the first number is the major version, the letter (or series of letters) is a minor, increasing alphabetically and going to "aa" after "z", and the last number is a patch.

I think they are always fuzzy and that’s fine, as long as tooling doesn’t make incorrect assumptions about meaning.

Different exporters and importers have different stability requirements and expectations - it’s negotiated.

It looks like a subdirectory is obit recommended if you want to support get old versions of the toolchain. Using branches or tags is probably the way to go these days

Yes this was a regrettable decision which causes lots of unnecessary pain and confusion and I wish they’d reverse it.

You can just use tags, no need to copy files, but IMO it should not be the package management system’s job to enforce draconian rules about semantic versions and import paths. Otherwise go mod seems pretty straightforward and sensible to me though.

This was just to support backwards compat with GOPATH for the brief time that the ecosystem was transitioning to modules. Now that GOPATH is rarely used, it’s not necessary.

I was talking about the requirement for v2/v3 in paths on major version as a regrettable decision, not the particular choice of duplicating dirs (which they recommended initially and as I pointed out can be sidestepped with a tag).

Ah, my mistake.

> and everyone else in the room had an "oh shit" moment because their amateur language design didn't address that corner case.

I doubt that’s how it went down. They tend to be very confident in their design decisions. Whether that confidence is warranted or not.

Reply on news.ycombinator.com