The Principles of Versioning in Go
research.swtch.com
research.swtch.com
While I don't personally agree with many of the decisions they've made, I still respect the choices they've made and I think there is plenty of room for languages with competing philosophies. Because of Go's philosophy of not catering to every demand, it's never going to become the next Java or take over all of Software Engineering, but I think the Go team is fine with that as long as it continues to be useful as a tool for solving problems.
Go is really close to C. In fact, Rob Pike's Go CLI programs on his Github looks pretty much like a standard C program using getopt. Interfaces are just a nicer virtual function table. interface{} is just void *. In that sense, it's already treading ground that is proven to work. Go didn't have to do anything revolutionary because C was already great.
I hope the Go team will continue to embody what made C great so the next generation can receive the lessons its taught us.
C is the JavaScript of systems programming.
Had it not been for UNIX workstations and universities adopting it, we would probably using BLISS, PL/I or something else instead and not bleeding billions of dollars per year fixing security exploits.
We would be better served with Modula-3 or Active Oberon, which Go failed to learn from.
Same for companies like MasterCard, American Express, Salesforce, Target, Twitch, Netflix, Twitter, Dropbox and Cloudflare.
>If an old package and a new package have the same import path, the new package must be backwards compatible with the old package.
Which are effectively fine conceptually. Every dependency system also supports that, you just make a new project named "thing-v2" instead of bumping a major version. Making it explicit and encouraging it when possible (instead of forcing breaking changes) is good.
The problem is that there's absolutely nothing to help you achieve that. Or detect failures to achieve it. Or anything. You can't even reliably rename a single package with existing refactoring tools, good luck copying and refactoring most of an entire repository correctly. If you do it wrong, you might have to do it all over again because it's probably another breaking change.
Without fairly strong tooling to push people towards correct behavior, "every change is breaking" is the default behavior. And the frequent accidental behavior even if you're being careful.
There were talks about "go mod release" from the very beginning, but you are currently correct here.
> (…) You can't even reliably rename a single package with existing refactoring tools. (…)
Strictly false. There is gomvpkg[1]. The absolutely ironic thing is that it's still stuck in the GOPATH mode, despite being a part of the official x/tools repo, but it's not “absolutely nothing”.
[1] https://godoc.org/golang.org/x/tools/cmd/gomvpkg
> Without fairly strong tooling to push people towards correct behavior, "every change is breaking" is the default behavior. And the frequent accidental behavior even if you're being careful.
I agree. Honestly, the slow progress in moving the community's favourite tools, including gomvpkg, to work in the new ecosystem is a massive failure on the Go authors' side. That just means that they haven't made the move easy and simple enough.
IMO renaming packages is a necessary prerequisite, but otherwise orthogonal to handling breaking changes. Personally I don't include it in the "something that could help" group.
Either way, it's wildly unreliable in my experience, so that point stands.
Also that'd mean it's a few years behind even basics like the vendor folder, so I'd be comfortable putting it in an "abandonware, effectively does not exist" category.
Renaming stuff in vendor: almost never. It could be useful for handling library updates though, if it worked and were possible: you could refactor it to the v2 path, automatically updating your code, and then pull the new update for real and be mostly done (for minor-but-breaking changes). I've done similar things in other systems for library updates, since there's usually zero automated migration help for library users, only authors, despite users greatly outnumbering the authors.
I don't deny that you could have issues with the tool, I simply have never encountered any myself.
* > as the dependencies of a project approach infinity, the odds of two branches of a dependency tree wanting to rely on two different versions of the same library approach 100% *
With experience, I've found that this is precisely the case, with bounds much, much smaller than infinity. The ersatz standard lib can't even standardize, there's underscore and lodash. I think eventually the community will Ostwalt ripen in a sense, small packages will dissolve by attrition and big ones will grow. But the argument is strong for simply decreasing n = number of discrete packages.
(As an aside, I don't like the term "SAT solving" for the problem of package version selection. By focusing on an implementation detail, it makes the problem seem scarier than it actually is. Register allocation is NP hard too, but nobody calls the register allocator "the SAT solver".)
If you don't know to update how will you get the fixes?
In general I never want to change any dependencies w/o explicitly thinking about the change and setting aside time for testing and breakage.
And at the time I'm adding some new direct dependency L, I will pick the version I want (presumably the latest). As for its dependencies, why should I want them to be the latest? I just want the versions that are most likely to work correctly w/ the version of L I have selected. Right?
I've stopped blindly updating my dependencies, as people seem to have very different ideas what semver means. minor and patch upgrades in node and ruby programs have broken my applications several times.
Even when semver is done correctly, a breaking change in a package usually also means that you have to upgrade to the breaking change to get other goodies like improved performance and bug-fixes. This is something that takes a lot of time. At work we've basically stopped doing breaking changes in our private packages, opting instead in for a scheme similar to what you have to do in Go for everything to work.
Versioning is, at the end of the day, a cultural problem. And not having the tools to do breaking changes well can actually turn out well.
At least, that's my current thinking. I may very well be, and often am, wrong.
More than once, the fix for a mysterious bug we encountered in production was just "upgrade a particular dependency to a newer minor release" (changing nothing on our code), since another user of that dependency had already encountered that same obscure bug, and together with the developers had already investigated it and found a fix. Had we been more diligent in keeping our dependencies up to date, even when they have no new features we want or security fixes, we would not have encountered that bug.
That said, the Go's team (and to a lesser extent Python's too) dropped the ball here. None of the challenges of dependency management was unknown or unsolved. Anyone who has used e.g. Maven should be more than familiar. (A 11 section blog post is indeed nice though, good for onboarding new programmers I suppose) If developer experience was considered a first class priority, this could have been solved from day 0.
(My personal favorite is NPM since conflicting dependencies can co-exist at the expense of greater disk usage, but memory is cheap these days. No one with an electron app on their desktop has any right to complain about the memory usage of plain text source code.)
Many of the package managers we have now are thought out. Some replaced existing ones while others were crafted by people who had worked on dependency management in the past. Cargo and Composer are two examples of this.
It's a straw man that keeps getting dragged out, but anyone who has used and understood this kind of dependency manager knows it's meaningless. You can definitely argue about its impact on community behavior, but it functions just fine, handles a ton of real-world issues reasonably, and it's nowhere near slow enough to care about.
https://github.com/pypa/pipenv/issues
Perhaps they need to "rewrite it in Rust" /s
Their competitor's isn't much better either when it comes to dependency resolution related bugs.
This is due to the fact that not all libraries on PyPI have properly declared their metadata and, as such, they are not available via the PyPI JSON API. At this point, Poetry has no choice but downloading the packages and inspect them to get the necessary information. This is an expensive operation, both in bandwidth and time, which is why it seems this is a long process.
https://github.com/pypa/pipenv/issues/2284#issuecomment-3952... (from 2018, may be outdated)
we compute a resolution graph that attempts to satisfy all of the dependencies you specify, before we build your environment. That means we have to start downloading, comparing, and often times even building packages to determine what your environment should ultimately look like, all before we've even begun the actual process of installing it (there are a lot of blog posts on why this is the case in python so I won't go into it more here).
------------------
As I stated in another comment on this thread, Cargo has to do none of this. This stuff is the source of things being slow, not the actual "Okay I have all the versions and constraints, what's the answer" bit.
Define “fine”. In my experience, Go with modules[1] solves a project's dependencies exponentially more quickly than dep. And that is especially noticeable on big projects with three or more dozens of dependencies. I don't mean any disrespect to the people behind dep, and the whole debacle was a massive miscommunication disaster, but go mod is just quicker.
[1] While we're at it, I'm still low-key mad that the Go authors called modules packages and packages modules. Gah!
Also, since you've mentioned Cargo. Its slowness actually used to be one of the main pain points about which the Rust people at one of the companies I've worked for in the past complained a lot during the water cooler discussions. Granted, that was a couple of years ago, and Cargo has probably made progress since then.
And any cargo slowness is likely the fault of rustc or I/O. It's received a lot of profiling work and the core dependency solver has never been a performance issue to my knowledge.
There are two things that really slow dep down. One is unavoidable; for the other, we have a plan.
The unavoidable part is the initial clone. ... Fortunately, this is just an initial clone - pay it once, and you're done.
The other part is the work of retrieving information about dependencies. There are three parts to this:
* Getting an up-to-date list of versions from the upstream source
* Reading the Gopkg.toml for a particular version out of the local cache
* Parsing the tree of packages for import statements at a particular version The first requires one or more network calls; the second two usually mean something like a git checkout, and the third is a filesystem walk, plus loading and parsing .go files. All of these are expensive operations.
----------------
For context, in comparison, Cargo has the same #1 problem. But for #2, the information on all dependencies is stored in the index itself; this means that Cargo can figure out what dependencies you need without any network calls at all, let alone downloading, git checkout, and filesystem walk.
I do not know how go mod compares, off the top of my head.
Left-pad is IMHO "too simple". Packages on the median should be in a 1-2 pizza team effort with some ongoing maintenance, even if only a few pizzas a year. Left-pad is like, 2 pizzas and a weekend.
(were it not for string.prototype methods) a viable package would be something on the order of Python's str.format(). Can left pad and right pad!
Don't have a strong opinion on whether left-pad should exist or not (or indeed, if server-side Javascript should exist or not), but simpler than left-pad, is yes? yes.
Natalie Weizenbaum and I wrote the package manager for Dart. My naïve backtracking version solver was fine ~98% of the time. 2% of the time it was catastrophically, exponentially slow. Natalie wrote a much better one to solve exactly that:
As a colleague of mine pointed out, the lockfile makes the solving a non-issue in practice anyway.
The exponential search issue could arise if people used explicit upper bound on crate minor of patch versions, but since the number of crate doing that is few far between, there's little change to encounter such a degenerated exponential case in practice.
Actually dep tended to produce suboptimal solutions, meaning in some cases it downloaded too many packages to solve dependencies. And that meant more I/O operations than needed. Plugging a full-blown SAT solver would have solved this issue.
Our education methods could do a lot more to draw this technique into its applications. For example, in game programming, satisfaction tends to come up rather abruptly when talking about physics solvers; not because it's only needed then, but because the jargon was appropriated from academic simulation papers, and it comes as part of a sudden bomb of math concepts, making it look more intimidating than it is. Non-realistic physics are just called "collision code", handwaved away and not given a great deal of formal attention, even though they are performing the work of solvers too, just usually with a mix of linear programming and local search. As such, a lot of new game programmers get stuck on collision problems and concurrency bugs that result from poorly constrained solutions, because they don't have the conceptual tools surfaced to reason about it with confidence.
Likewise there are plenty of CRUD-style apps where a solver is a helpful building block for creating views and filters, but it's not approached as one, so you just get a buggy feature.
I don't quite get your criticism here. Russ's Minimal Version Selection algorithm is a package version selection algorithm that does not require SAT solving. Your wording sounds like Russ wrote something like “Package version selection is inherently SAT, so we won't do that ever!”, but he did exactly the opposite by introducing MVS.
Maybe you meant that you don't like the term for “Cargo-like package version selection”, but that's a completely different story.
That algorithm also doesn't handle updates though, which is quite a big limitation.
What they aren’t supporting is incompatible dependencies being incorrectly advertised as compatible, then being worked around by other dependencies testing against those versions and explicitly marking them as incompatible, in cases where dependencies have a shared dependency with different requirements.
Instead it looks like they are trying to solve this problem outside of the dependency manager, by putting a lot of emphasis on preserving backwards compatibility within major versions being the correct Go practice (and probably also hoping that edge cases get fixed quickly in a collaborative way, since everyone is on the same system).
How well this approach will work relatively in practice seems to depend more on the behavior of library authors than any technical factor, without knowing that it seems hard to say that either is obviously superior.
Think of `go get` as stable and `go get -u` as unstable. If you're working on unstable, then things may break and you should report bugs and/or contribute patches.
Upgrading a library dependency in your module file and committing the change is how you change stable for your library. Downstream users have their own ideas of stable that will be behind yours, until they upgrade.
So Cargo allows independent dependencies to solve the problem of broken semver in another dependency (as long as a viable version exists), where in Go it would need to be fixed in the original dependency or the root module.
I still think which approach “solves” the problem will depend on user behavior - it’s easy to imagine forcing broken semver to be fixed at the source working better in some cases than relying on incompatible version lists being updated correctly across many libraries. But Cargo certainly provides more tools to solve the problem.
Another improvement might be having a way to say "give me just the security fixes".
At first this could be a new tool that you run instead of `go get -u`, so the Go team doesn't have to be the ones to start this project.
People who just want their code to work don't need to care about this. They can run `go get`.
There are always going to be people who just want stable code and repeatable builds, and others who are working on migrating the community to the latest code. (And they might be the same person at different times.) It seems like Go's module system might work pretty well at helping them collaborate while staying out of each others' way?
It's already possible: https://github.com/golang/go/wiki/Modules
https://github.com/golang/go/wiki/Modules#how-to-upgrade-and...
The linux world solved this with providing a package repository that maintains stable versions of software. This requires a lot of time and effort.
Many software projects simply don't have the labor available to support multiple releases of software. They run tip of master and that's pretty much it.
They reshaped the problem: all future revisions of a single entity must be backwards compatible because it's the "right thing to do", and thus, under this assumption, a much simpler solver will do the job.
Yes, this moves the problem slightly, since now there are other places that encode versioning information (the import path themselves) and thus some of the incompatibility detection checks performed by package managers are now delegated to the compiler (which will e.g. not find a given import, or complain that foo.Bar is not assignable to foo.Bar because the full import path has a different version tag).
Yes, many people do not currently care about the problem of backward compatibility and it will be hard in practice to make everybody raise their standards.
But, careful backward compatibility is one of the main ingredients that made Go successful in the first place.
A healthy ecosystem should follow the same rules.
Incidentally, this whole vgo kerfuffle can help highlight which section of the ecosystem is written without this rule in mind, offering a choice for both the authors of the components (improve their practices) and the consumers (e.g. switch to another library)
This echos some of the other observations of tooling not being easily updated and still not being there, yet.
Are those documented anywhere? Are they detailed, discussed, or otherwise dealt with. The people behind those package manages have a significant amount of experience with dependency management including many real world situations. It would be good to have some public discussions on this for everyone's benefit if this is really true.
> Principle #1: Compatibility
The whole section on compatibility doesn't sit right...
> It is intended that programs written to the Go 1 specification will continue to compile and run correctly, unchanged, over the lifetime of that specification. Go programs that work today should continue to work even as future “point” releases of Go 1 arise (Go 1.1, Go 1.2, etc.).
This doesn't seem to apply to the tooling. For example, `go get` has a new behavior since the introduction of modules. The behavior changed. So, the compatibility idea only extends so far. Is that hypocritical as an idea because people use the `go` CLI for scripting and the behavior of it changed.
> What does compatibility have to do with versioning? It’s important to think about compatibility because the most popular approach to versioning today—semantic versioning—instead encourages incompatibility. That is, semantic versioning has the unfortunate effect of making incompatible changes seem easy.
Incompatible changes happen. How many long lived pieces of software have needed to make incompatible changes while supporting people through that process. Do you rewrite? Do you copy the whole codebase to a new sub-directory and add more there? This is what the Go project has started to recommend. If you do that you loose history and some projects move fast (look at Kubernetes client-go).
Tools like npm, cargo, and dep provide a means to stay at older versions if that's what you want. The choice is up to the consumer of the package. With Go the choice is now up to the Go team by limiting choice. Is that a good thing? Not if you don't like the choice.
It does not apply to tooling. It is not hypocritical because it was never claimed that it applies to tooling. It has always been a promise about code.
When I promise my kid that I'm "taking them to the park" this weekend, they have no grounds to complain when I don't take them to a full-fledged amusement park just because "taking them to the park" would technically cover that too.
Yes, e.g. here: https://research.swtch.com/vgo-mvs
You can go back and find discussions from when that was first written, but basically it's accepted that the dependency version constraint system of Cargo, Dep, etc. has some real flaws, and Go's module system solves them.
I wonder if the posts lately are an attempt to win people over to their thinking or to win back popularity.
[1] https://www.tiobe.com/tiobe-index/go/
[2] https://redmonk.com/sogrady/2019/07/18/language-rankings-6-1...
(Maybe it's actually not something that could've been done better; maybe it's just the nature of the magnitude of the change. In which case I think the mistake was going "1.0" when you knew you had something so important not yet defined.)
Control has been taken away from package consumers
Ten-year-old+ programming languages with at the very least tens of thousands of users (I see "millions" claimed, I find myself a bit skeptical, but maybe... conservatively tens of thousands for sure, though) do not lose 33% of their users in a month. That'd be somewhere in 2-3 orders of magnitude faster than Perl 5 has declined. Implausible.
You're putting a "just so" story on top of noise. Nor is this special pleading on my part; "Tiobe is mostly noise" has been my position for years, along with a lot of other people. Signal probably exceeds noise for maybe the first 3 or 4 or maybe 5 slots, after that it has a history demonstrably absurd claims about the popularity of things like REXX, which spent many years in the so-called "top 20", which is just ridiculous.
Simply put, this rule is too strict, and biases modules' UX and semantics too far toward consumers.
Not all packages are so mature, and consumed by enough consumers, that they should be forced by the language tooling to maintain backwards compatibility. That requirement creates a great deal of pointless work for the majority, perhaps vast majority, of package authors, who write small and uncertain packages for use by themselves or a small number of their immediate colleagues. For these authors, strict backwards compatibility is not only unnecessary, it's actually undesirable, as it creates needless toil.
> The answer is that we learned from Dep that the general Bundler/Cargo/Dep approach includes some decisions that make software engineering more complex and more challenging.
For a minority of use cases, where packages are large and mature and imported by many, many consumers -- sure, maybe. But the vast majority of Go code, written by private organizations and used internally, doesn't fit this description.
The rest of mod looks fine but tying paths to major version changes was a mistake IMO.
1. I don't know why I find Russ's writings are so irksome, even unpleasant to read, sorry. (Please help me on that, maybe I am in "questioning authority" mode and cannot flip out of it.) I cannot shake that feeling of being patronized all the time without being asked. The whole article, no - bunch of his posts, - reads like "we know better". Is it a Google thing, or heavy Rob Pike's influence?
Yes, he is bringing some good points, and some very questionable ones. And for the questionable one (Compatibility section for example, or SAT solver question (nobody did it before?), or version naming, or paths for the sub-components...) it feels like he is trying to push explanation about his (and his team?) reasons and motivations behind these decisions, but... it is a monologue. There are more opinions and whole wide world of different software shops with own ideas and practices.
2. Go2 was a promised land, that later translated into "incremental, no code breaking" approach. Nowadays it feels like they are not even talking about it anymore!
Looking at general history of programming languages, pretty much every language has had it's "C++" or Modula-2 moments. It is a natural process - an evolution - when unneeded parts disappear, and useful ideas added/extended. Amount of gotchas in Go is not zero, and to me (YMMV) it feels like Go2 would be (was?) a good time to shake off under-thought items, and enforce some new rules. Maybe even add interoperability between .go and .go2 files, maybe even semi-automatic upgrade path to run `go2 fmt ` on old files a la to Kotlin from Java - I don't know. But it feels like an opportunity and momentum for a "peaceful revolution" is lost.
3. Even with non-breaking changes style of change they have managed to break stuff. I, for one, still sit on 1.12 branch - because when I tried to upgrade to 1.13 (twice now), and our pure-Go code just plainly did not build with some obscure messages (I think it was linking or module cache errors). So, why pretend? Code rot is unavoidable - do we like it or not.
99. There are couple more points about chilling effect of "we know better" on a community as whole. (One example would be this whole "error" debacle that visible reduced activity on issue tracker and golang-nuts mailing lists after "hey all, we are going this way because we already went there". Immediately after it felt like people just gave up and left. "Empty town square" feeling.) It would be interesting to see this year results of "the state of Go" survey, but I don't think we will see a worrisome drops there -- the amount of new people coming in would easily override any opinions of people leaving.
/end of rant, sorry for a long post
> nobody did it before?
As far as I know no other dependency system is the same as Go's.
> "we know better"
I mean, it is Rob Pike. How successful do you have to be before you get to say I know better? In this case it is better.
> Nowadays it feels like they are not even talking about Go 2 anymore!
This very post is about Go 2 (in the sense that there was ever going to be a "Go 2").
> pretty much every language has had it's "C++" or Modula-2 moments. It is a natural process - an evolution - when unneeded parts disappear,
I can count on one finger the number of parts that have disappeared from C++.
A project's leaders are always going to opinions about how things should work, and they can do a lot to listen better and understand other people's concerns, but they're still not going to choose somebody else's solution if they're not actually convinced it's better. Choosing the solution you think is actually worse on an important issue like this is basically a recipe for self-sabotage.
I think oftentimes people have unrealistic expectations of how open source software works. It's not a democracy. There's no way to win without actually convincing the project owners that your way is better.