Go 1.17 is deprecating the traditional use of 'go get'
utcc.utoronto.ca
utcc.utoronto.ca
My bigger complaint, though, is that the Go team have decided to ignore replace directives when you call `go install`. In a perfect world, I find a bug in an upstream library, submit a fix, and they accept it. In practice, lots of upstream maintainers check their issue/PR lists about once a month if you're lucky, so the best way to temporarily fix the problem is to put in a replace directive changing github.com/foo/bar -> github.com/me/bar until the PR goes in. Except `go install` will ignore that, so if you install a tool via `go install` you get the broken library, but if you git clone the repo and run `go build`, you get the fix.
In proper Go team fashion, they have declared that because sometimes replace directives are difficult, they refuse to deal with them at all--if you intend to distribute something for use with go install, you better fork the dependency and update all your source files.
I fought hard in the github issues to make go install error on replace as a compromise to ignoreing it which creates two subtly different build modes! Also by erroring we leave open the possibility to eventually change it to build correctly with the replace directives without breakage.
Add your voice that @version cmds should respect all replace directives that they reasonably can!
> In proper Go team fashion, they have declared that because sometimes replace directives are difficult,... they've asked for proposals that can be publicly vetted and discussed?
The line has been "submit a proposal, let's review it" as long as I can remember.
1. It doesn't propagate downstream. If module Z depends on module Y, and module Y depends on module X, and module X is faulty, it's logical for module Y to have a replace directive for a fixed version of module X. This does not affect module Z, which now subtly (or loudly, sometimes) breaks. That helps nobody. It's good if you have an application that you build in CI and ship to your own private servers, but pretty much useless for everyone else. (And people don't seem aware of this; the author of module Y thinks they've fixed the problem for their consumers, but hasn't.)
2. It doesn't compose. If module Z depends on module Y and module X, and module Y and module X depend on module W, and the authors of module X and module Y independently discover that module W is broken and develop separate fixes for it, the author of module Z really has no option to make their software work.
I think Rob Pike's "a little copying is better than a little dependency" is apt here. If you just copy the source code you want into your own application, the problem goes away. But, the problematic upstream dependencies are often large projects (Azure's API client is a big offender here; I've never successfully depended on it without a replace directive) that are impractical to simply copy-and-refactor. At some point, these upstream providers need to fix their shit or risk not being used.
This problem, by the way, is not unique to Go. I notice this problem come up with Javascript when a bunch of modules depend on faulty-module-0.0.1 with a security problem. faulty-module-0.1.0 is released with a fix for the security problem, but your app directly depends on neat-feature-1.2.3 and nifty-addon-23.47.1 and those modules break with faulty-module-0.1.0. So inevitably you show up at the issue tracker for neat-feature and nifty-addon where 6000 other people are yelling at the authors to release a version that depends on the fixed version of faulty-module, find that the authors aren't around, and are just sad because there's nothing you can do. Basically, depending on other people's software sucks, and so does copying their code into your application. go has the advantage of very clearly telling you "hey, you're fucked!" which annoys people, but you basically get into this situation no matter what programming language you use. It's just you might not know, which is scary. (I hope you use something like Sentry for your Javascript apps and have 100% test coverage.)
`go get github.com/pion/webrtc` used to pull into $GOPATH/src/github.com/pion/webrtc. That worked really well for new contributors. They get the code with one command and run `go test`.
I wasn't able to just update the docs and recommend `go install` because it doesn't work with old Go versions. Is the expectation to put in the docs a sh snippet that does if/else on the Go version?
I am grateful for modules though. Suprising the amount of people that still aren't using them (or aware at all!)
But GOPATH is deprecated so you shouldn't be relying on that anymore anyway.
The expectation is that the user just clones your Git repo to _anywhere_ on their PC and runs `go test` there. If they want to consume this local development version in their application then they use a simple `replace` directive in their go.mod to point their dependency at their local version.
If they're still using GOPATH then that's something they can probably manage themselves honestly. That's just a simple folder move. It's been like, what, nearly 2 years since GOPATH was deprecated?
Go can hardly be called a modern language. The usual argument we hear on HN is that it has an absolutely stellar backwards compatibility story. Are we losing that too?
I'm not sure what this means exactly, but it's a significant advance in simplicity and productivity in my experience.
> The usual argument we hear on HN is that it has an absolutely stellar backwards compatibility story.
This refers to the compatibility of the language, not necessarily the tooling. Even still, this is a single change, there's no need to panic about losing backwards compatibility just yet.
Another tenant of Go, the language, is specification before implementation. There specifically so that many implementations of the language are created and do not become dependant on quirks of a single set of tools and enabling users to choose which tooling they prefer.
Point being, there is a very clear line drawn between Go the language and the Go distribution. Guarantees provided by one are not necessarily applicable to another. Nor do they apply to gcc-go or tinygo.
No, the main thing, that GOPATH meant, is that Google (and everyone else) had to make their Go libraries open-source — since entire logic of GOPATH revolves around building stuff, downloaded from Github.
Once Google management decided to be serious about pushing Go to masses, they hurriedly rushed to erase GOPATH from history (just like they erased "Don't be evil" slogan from Internet after getting serious about doing business).
GOPATH has nothing to do with GitHub. It is one of the places that has special case handling, but is absolutely not the purpose of GOPATH. Case in point: the VM I have to build some legacy stuff using GOPATH does not have a _single_ dependency from GitHub. Not one. Nor does it contain any open source code, besides the standard library.
The logic of GOPATH and "go get" just so happen to revolve around building packages, fetched from source code repository. This logic used to be a stop-gap, written during early stages of Golang development, and makes it awkward (at best) to distribute proprietary Go libraries. I assume, that Google is introducing module system (and aggressively deprecating GOPATH) to work around this issue (among others).
I stand corrected.
I see what you did there...
I don't think your documentation has to be backwards compatible in this case. If you must split it into two sets of instructions based on Go versions, you can. But unlike many other ecosystems, in Go it's quite common to expect the developer to have moved to a somewhat recent version.
I will say I had projects using dep and/or relying on a GOPATH-style directory layout with my build scripts, but that ship has sailed with just about any new libraries one might depend on. I don't think anyone would blame you for your library moving on too.
People and organizations decide when they will move versions based on their own project lifecycles and requirements, usually (outside of the Go community?) not just because the language vendor says there is a new version and it's time. This is something that was hard for me to cope with as a Rubyist, who was always doing everything to be on the latest version.
I'm new to the Go community, but I can already see the effects of this in Kubernetes, and I don't necessarily think it's good the way things are. Kubernetes 1.16 was released in 9/19 and already EOL and out of support in 8/20, less than a year later. (It took some time for many cloud vendors like AWS to catch up, EKS only landed K8s 1.16 in April 2020. That's four months of usable life before it was officially out of date.)
Integrated things that worked in August of 2019, probably need work again in 2021. As a software developer I understand this and "yeah, we need to always be upgrading" is part of my mantra, and it has been clear to me since I was a teenager that I will spend time struggling with problems that I would not even know about, unless I run Debian "unstable" or an equivalent rolling release distro, where patches can be accepted from upstream at any time of year, whether or not they contain new features.
But I could never get my bosses to see it this way, working in a company that was not firmly embanked inside of the "Cloud-Native" sphere of the world. The "normies" I've always worked with didn't want to see new features popping in at any time, they wanted their own predictable environment that works the same as it did yesterday, to stay that way all year long.
(Edit: and for what it's worth, Ruby does a pretty good job of being stable all the time I've used it professionally for the last 8-10 years, and in spite of having chosen it themselves, they still hate it. Anyway, that's what they say they want...)
Well, then, good news, this is mostly an effect of Kubernetes being its own ecosystem. It's implemented in Go but it is not generally considered a good example of a Go project. First of all, there's a great blog post somewhere about not looking to the largest projects in any language in general, because they tend to become their own ecosystems, but I don't know where it is. It's right, though. Secondly, Kubernetes is an extra-specially-bad example of a Go project because it's actually a Java project that was ported into Go, and even today the architecture makes that very clear that's how it evolved.
The Go ecosystem in general is actually pretty good about backwards compatibility. If you're starting out new, you shouldn't even necessarily notice this "go get" change because for you the new way will just be how it works, and the new way is easier to understand for a new person than the older, more confused way is. Plus, you have to opt-in by upgrading your Go compiler, and that's really quite optional; if you choose to stick to one for a while, you don't miss out on many libraries, because the language hasn't changed much. It isn't until generics drop that you'll start seeing libraries that you must upgrade for, and if you ignore the libraries that are literally "we're deliberately using generics to do things we can only do with generics" I expect it to be even another year or two minimum before you start seeing other more generally-useful libraries start to require them. I certainly don't intend to go back to all my libraries and retrofit them with generics just because. I don't think there's a lot of pressure to keep up with the latest compiler all the time.
HN has a lot of people waiting with bated breath for Go stories to drop who like to come in and complain vigorously about whatever the Go team has done, despite not using the language themselves. I don't mean to discount people with real experiences, but HN's sound and fury are not always proportional to the real problem. (Just because someone don't like Go's type system and the fact that it lacked generics for so long doesn't mean that piling on and moaning about some not-all-that-significant change in how the tooling one wouldn't be caught dead using and don't know much about is a useful thing to do.)
Thanks for the perspective. That makes sense, I wrote how they don't really like Ruby even though it is very stable like they said they wanted, and rarely makes breaking changes (and takes great care to ensure they are well documented and easy to follow when they happen) ...
Then I thought about Rails, and decided not to say anything about Rails, (because it might undermine my whole argument, as that's where arguing from this position breaks down a bit.) Rails isn't really that bad with breaking changes, but it is where most of the breaking changes probably come from, and therefore probably worst thing to use as an example... like you say with Kubernetes in Go!
You'll find that true for any language and framework created within the last 15 years (and some that are older than that too). Rust, Nim, node.js, TypeScript, .NET, etc. And it's to be expected too, for the following reasons:
It's really more down to development pipelines. If your pipeline requires additionally installing the language and/or framework (eg Go doesn't ship by default on most operating systems, it's an optional install) then you're going to expect a recent version to be installed since you're having to install it anyway and since you're not going to break a mass of existing OS boilerplate that might depend on an earlier version being installed (as is often the problem with Perl, Python, shells and others).
Also if a language is newer, then there is often a higher pace of in demand features with each new release. If Go were 30 years old you can bet most of the features people might want would either already be implemented or developers that might want those features wouldn't be giving Go a second glance and instead working with other languages instead.
The longer you work in IT the more you see this cycle repeat over and over. It's almost comical how cyclic the industry is.
It actually uses the nice VCS library guts that `go get` does!
However I do totally understand why, as a person who has more than once become ridiculously confused because I accidentally ran go get in a directory that was a Go module. Somehow I even managed to make my home directory a Go module, leading to a lot more confusion.
Needless to say, they do need to fix that somehow. They probably should’ve gone the other direction and made a new go get command that was specific to modules. It’s a bit too late to do that now.
Go modules overall have been reasonably smooth but I’m a bit disappointed since this particular break will cause a lot of confusion. I think most likely in the long term it will pay off, but it’s probably unnecessarily abrupt. Maybe this is being done to lead into Go 2?
Honestly, I do have another wish: I wish there was a version of “go run” that could fetch, compile and run directly from the Internet. Now I realize that’s not particularly safe, but it’s not much unsafer than doing it in two steps, and man would it be slick for small demos.
I say this is a change for the better. The cli is more consistent. The argument made in the post seems more like a philosophical "wah, wah, I don't want to".
Edit: correction, Pip is older than Go. But for those downvoting me, the dates were just an illustration and don’t really matter to the point I’m making. That point being: after 11 years it should be safe to update a legacy interface.
> Initial release: 4 April 2011 (10 years ago)
Source: Wikipedia, pip (package manager).
Like other things in Go, the antiquated feel is actually the pursuit of retro more than the product of age.
Fair point. I’m on my phone so missed that section when scrolling through for the summary box.
> Like other things in Go, the antiquated feel is actually the pursuit of retro more than the product of age.
I’m really not sure the point you’re trying to make here but it feels like a baseless cheap shot rather than a constructive comment
https://pypi.org/project/pip/0.2/
Released: Oct 28, 2008
The earliest release I can find for Go at [2] is 2009-12-09, with 1.0 being released 2012-03-28.
Look at Java, the king of BC, even they've started deprecating packages in the JDK for removal.
It's inevitable. Otherwise it's like a brain that can't forget even the most useless piece of information. About an organism that just eats, but can't expel the waste.
There's a way to do change slowly, gradually and reasonably, but change is inevitable. If you don't want small changes, big changes will come for you.
I was surprised to see six months described as "never, in all that time".
The remedy in practice is pinning versions by project or even worse org. wide to be able to concentrate on the actual problem being solved if things are breaking too much.
There is a reason plenty of organizations are still on Java 8, which is the last major release before they started doing that...
The reason 8 is preferred for many is due to a) not understanding the new model b) depend on a dependency that does some shady things with JVM internals it should have never done in the first place, this did get “sealed” by the module system. Most of these deps actually have already migrated to 8+, and even if not, access to internals can still be allowed with a command line flag.
On the other hand, deprecating a critical tool of infrastructure leaves a sour taste. Many people would be pissed of. Better not including it in the first place.
IMHO such tooling should not be part of the language itself, and be left to the community.
In more practical terms, I expect this tooling is provided and evolved as a first class affair because Google needs it internally to build their golang projects.
I strongly disagree with this. "All batteries included" is one of Go's strongest points, especially if you look at gofmt, which set a standard for all code out there.
But I still prefer to have the occasional churn when "the official way of doing things" changes to having x competing tools doing the same thing, which has the potential of generating even more churn.
It's already making FOSS less accessible to people who are not developers.
Not to mention how quickly it rots.
I can tell you that in the wild, non go programmers do not use tools installed using the go compiler that they do not have installed. It's apt, brew, $WINDOWS_PACKAGE_MANAGER, flatpak, etc.
I imagine precious few people install docker or sysdig using `go get`.
At this point, I feel like the entire solution space has been explored between all the different languages and there is not a great solution from any of them. Personally, I’m much happier with Go breaking things than becoming a C/C++ hole of legacy going back like 50+ years.
On the timing and speed (only one version warning before removal): we can debate the right speed until the cows come home. But, Python 3 took longer than a decade to kill Python 2 as they were trying to give people “plenty of time”. In the end (IMHO), people don’t change unless they’re aggressively forced to because there is always something better to do with your time than upgrade software to prevent some possible future hypothetical breakage. That calculus changes when it’s not hypothetical at all.
While I can appreciate the care and engineering skill involved in upgrading a technology while still having backward compatibility, it turns out just to enable the worst possible behavior on the consumer end.
I think I like Apple's way of just forcing breaking changes down everyone's throats (e.g. removing CD/DVD, PowerPC->x86, 30-pin->Lightning, dropping headphone port, etc.). Sure the tech media complain for a cycle, but then everybody adapts and the world's a better place.
I vote for IPv5 protocol, which will allow to cal any IPv4 host via IPv5-IPv4 NAT transparently at nearest bridge.
WHY I cannot do `ping6 127.0.0.1`? WTF? Is IPv6 is too small for whole IPv4 address space?
Maybe you think it should just be exactly equivalent to ping6 ::1 - your ping6 could be updated to allow this but, like, why? That's not an improvement at all.
OK, so maybe it should be equivalent to ::127.0.0.1 - but that's very strange, an operating system could choose to have this do what you expect locally but it's not clear to what end and again it doesn't offer any wider improvement.
IPv6 is big enough that it contains the entire IPv4 address space - in several different places in fact although ::a.b.c.d is the most transparent. But, that's completely irrelevant to the problem, which I suspect means you still haven't quite grasped what the problem even is.
As it is, they locked the upgrade cycle, because you can't do IPv6 only without losing access to IPv4 (historically).
In the end folks did get work arounds going using 464XLAT
an_ipv6_function(address) {
if (address & 0xffffffffffff0000 == 0) {
// Use legacy ipv4 protocol ...
} else {
// Use modern ipv6 protocol ...
}
}
a) ping6 ::ffff:127.0.0.1 doesn't work too;
b) it's not an IPv4 address, because the binary representation of ::ffff:127.0.0.1 != 127.0.0.1. It's the IPv4 address space mapped to an IPv6 address space (IPv4-mapped-on-IPv6).
c) IPv4 addresses in IPv6 address space are deprecated: https://datatracker.ietf.org/doc/html/rfc4291#section-2.5.5.... .
In short, IPv6 and IPv4 cannot coexist, so they must be separated, so we must keep IPv4, because it's mandatory, and we may skip IPv6, because it's optional. So, all internet providers in my area are skipping IPv6, because they can.
All that problems are created by lack of backward compatibility in IPv6 protocol.
Why? «Because backward compatibility with IPv4 is unnecessary. IPv6 is so much better. Everyone will jump to IPv6 till end of the next (1995) year.»
But once you get past those, another layer of "solutions" see the immediate technical consequences of 2^32 not being enough but don't grok the bigger picture. For example let's just turn existing 32-bit IP addresses into a 64-bit IPng address by adding zeroes, and then all the old addresses are now networks with up to 4 billion nodes. But wait, all those addresses are assigned too now, so we still have an address shortage. Oops.
The least stupid people saw the new address scheme itself as logical but just felt like surely IPv6 should have resisted the urge to fix anything else. Sure, you're embarking on a multi-decade project that likely cannot ever be repeated, but let's just leave all the sharp edges and weird anomalies exactly as they are to minimise upgrade costs. This belief is less stupid, but it is still pretty stupid, that cost difference is a drop in the bucket, and this was our only chance to fix some pretty huge mistakes made when the Internet was newborn.
It would actually be cheaper for a lot of big outfits to go IPv6-only today (and then use edge translators) than drag their heels and try to stay on IPv4 as long as possible, but contrary to popular belief almost nobody really does "cost-benefit analysis" to decide what to do - they make a decision based on emotion and any such analysis is purely there to support that decision.
Buying more addresses is a budget line item you can see, but a lot of big IPv4 costs are hidden in existing practices that "always" have cost money but would vanish under IPv6, and some IPv6 costs are only there if you did no preparation. If you just count the need to replace those IPv4-only printer-copiers you purchased a year ago across the company, and forget that you intentionally chose not to ask bidders if they were IPv6 capable you're going to persuade yourself that it's cheaper to keep waiting.
It still would have taken a long time to deploy, because you still need hardware and software support in lots of places, but it might have been a little less long, and a little less bad along the way.
I don't think we got much of use from the new ways either. I can announce IPv6 ranges to my LAN from two different ISPs, but my clients will use an IP from one and send to the gateway from the other, so that doesn't help me.
On the cheapest budget cobbled together network, either one turns into a network broadcast. But even a bargain basement 1990s Ethernet chipset does onboard multicast filtering, so irrelevant neighbour discovery packets needn't wake up your OS whereas every ARP query must be drawn to the attention of every host's OS to confirm it's not for them.
However, spend a little more money and the network switch understands multicast, so now the data isn't even sent across links which don't have anybody listening to that address, whereas of course nothing can be done about broadcasts.
You couldn't have easily done this trick in IPv4 because the address space is too small, but in IPv6 choosing the Ethernet multicast addresses to make this work falls out very easily.
Lots of us on Windows are still stuck with .NET Framework, because no matter how Microsoft keeps pushing it, even their own products aren't fully ported to Core, let alone 3rd parties.
Yes, a simple voice call might use STUN, through two nats, fail to puncture a hole, and then be proxied over a server somewhere, but "it works". Noone is willing to may 1$/€ more per month to get IPv6, noone really needs it, half of IoT doesn't support it (ahem, esp8266/32), it's costly to implement and costly to educate users (especially those, relying on nat for "security").
Most of the IPv6 progress is because the governments mandate it for many stuff, big cloud providers are running out of addresses to buy, vendors want to sell new routers and switches... basically, everybody except the users.
Upgrade cycle should be one major version per year (may be less, but not more).
N+1 version does not break compatibility with N version, but introduces warnings for things that's going to be removed. Every warning must have very easy and clear way to resolve it, ideally automatic way, if language and tools support it. Developer must not redesign his application to get rid of warning, but rather just make few obvious mechanical moves.
N+2 version removes previously deprecated things and breaks compatibility with N version.
So you're going to spend few hours once a year to migrate to newer dependency and resolve their warnings. If your software was rotting for 10 years, you're going to spend a week to gradually upgrade, resolve warnings, etc. That should be acceptable to everyone. And those who prefer to live in rot, should pay someone to maintain those outdated libraries (or just live with bugs and vulnerabilities, that's their choice).
With programming languages and foundational frameworks it might be worth to be a little bit more conservative and extend that deprecation period to a few years. Just don't keep deprecated things infinitely.
I used to build things with frameworks and languages with yearly upkeep requirements.
> And those who prefer to live in rot,
What rot? If my application didn't change I'm not sure why my dependencies should need to.
Regarding golang, I hated go get and the go workspace idea when I first started with go.
Then I got over it, now I try to use modules and I just get annoyed, the whole system is confusing and flaky.
All this change is doing is pushing me further into rust. It has problems too but they tend not to do this kind of hand wringing.
Go is much more stable than Rust, you know how many Rust libs need nighlty to work?
Rust libs that use nightly do so because they're opting in to new features, not because released features are unstable.
Even then with the releases of const generics, proc macros, and a couple other big-ticket features, fewer and fewer Rust projects require nightly anyway.
(This is me being curious about your Rust comment, not implying anything about the relative stability of the two languages)
Anyway, thanks. Nightly does exist to be used, but our users largely report using stable, so when people say things like this, I am trying to see if there’s any blind spots.
I think the compiler should just ship with older versions of the language and different translation units can be compiled with older versions of the compiler. So long as the newer versions can interact with the older ABI (which ideally rarely changes), there's no problem.
* Fix all bugs in old version, or
* Ship known buggy code
Neither seems great. Better to just have archival versions with a "here be dragons" warning.
Thank god there's no known bugs in released software!
But more deeply, LTS ships buggy versions of the software. That's the point - only the truly dangerous bugs (e.g. compilation-triggered RCE or something) get fixed, everything else stays compatible even if it's otherwise undesirable or suboptimal. Code depends on those bugs.
For young projects, where it's kind of expected (trade-off of being an early adopter), or for small projects (which can be easily maintained by a third party), it's acceptable.
But for bigger/core projects this is far more problematic.
Just for comparisons, a lot of Linux distributions will maintain a specific version for ~5 years if not more (Debian, RedHat), and if for smaller components, they can do things like backport patches, or even create the patches themselves, however, for larger ones, they are reliant on upstream providing stable and maintained versions for the life of a specific version of the distribution.
Also, I've seen quite a lot of large and complex projects taking several years to reach production. In these contexts, having to constantly rework components (and re-test them, both individually and as part of the overall system) is not really feasible.
There is no ideal solution, maintaining backward compatibility is costly, and can lead to messy code bases, and it's legitimate to want to avoid that by deprecating API/functionalities. However, downstream (distribution, users of a given library) needs stability and generally cannot afford to rework their solution every 3 months (keep in mind that downstream generally relies on many upstream components, if a lot of these components break at the same time, even in minor ways, this can result in a huge headaches).
IMHO, a more desirable compatibility period, at least for big/core components, would be around 5 years.
I know with C, I can rely on it for 45 years. There is something to be said about this and the peace of mind that comes with it.
Obviously, this is a spectrum with extremities and many ways to go about it. We should not discount "Move slowly, and not break things".
Right, but that's at the expense of 50 years of progress. For the overwhelming majority of languages I strongly contend that's the wrong tradeoff.
Some organizations have a long path to production for runtimes and would prefer to skip releases. Requiring a code change to be tied to the runtime update makes things harder and encourages staying on the old release forever.
Soon I will be able to let go to Python 2.7, but not just yet. My industry is the precise opposite of "move fast, break things" because here the "things" happens to be what are called humans and their "break" scenario involves shrieking as their lungs are burned to leather, and no amount of Jolt soda poured into the floating holographic head of Mark Zuckerberg will suffice.
If you have a piddly language you and five people circulate on Usenet, go for it. Break it twice a day. But once your language gets serious, be ready for inertia. It's a property of mass.
And if one of your dependencies started using new features from 3.x it would break everyone stuck on 2.7, so you can't even use the new features in your library. So why would you bother migrating? So everyone was stuck in a game theoretical position where there was no first mover advantage because one straggler would invalidate all the investment.
Clearly not, as most libraries are now Python 3 only, so the switch did happen.
I do see a lot of people complaining but I see no practical solutions being suggested. "Just don't do a breaking change" isn't a practical solution when the problem being fixed was such a core feature of every language, strings.
You don't build a convincing argument that the python developers did nothing wrong when saying this 1.5 years after python 2.7 has been EOL.
>I do see a lot of people complaining but I see no practical solutions being suggested. "Just don't do a breaking change" isn't a practical solution when the problem being fixed was such a core feature of every language, strings.
A system should have been put in place to allow python3 code to import python2 code. Then people have an incentive to actually run python3 and use python3 features.
More people are stuck on Java8 which has been EOL for a very long time. They are often stuck because Android hasn't upgraded afaik. But Java11, 14, etc can use almost all Java8 code unless it uses I think sun.misc.Unsafe or something like that. The point is that people can use modern Javas with Java libraries targeting older versions.
Try shipping python3 scripts to systems that only have python2 interpreters. Doesn't work very well unless you also start to ship the interpreter and all its dependencies. I learned my lesson and just avoid distributing any Python code to customers now.
Docker tends to be the easy solution for all this stuff, and it's language agnostic.
> Doesn't work very well unless you also start to ship the interpreter and all its dependencies.
That basically applies to everything that isn't a statically linked binary.
https://developers.redhat.com/blog/2019/08/01/how-the-gnu-c-...
Every time libraries and compilers advance, you're effectively porting all the existing code. You have to fix any build breakage, and re-validate everything.
Moreover, Python is a C program. Just get the old Python 2 C code and compile it against "the same version of libc you're running" and it will work, right? Where is the problem?
I didn't care about Python 3 features, but you can't run Python 2 code on a Python 3 interpreter either and the Python 2 interpreter was abandoned. Meanwhile you can run ancient JavaScript code on both old and modern systems, ancient C++ on both old and modern systems, ancient Java mostly on both old and modern systems, etc. .
> Docker tends to be the easy solution for all this stuff, and it's language agnostic.
I think I will rather go with statically linked binaries before I dump docker on unsuspecting users. Just the pain of getting that dependency whitelisted by their corporate IT makes my skin crawl.
Every person saying "pythonic" instead of idiomatic would need to switch to "booshy" instead. The file extension would be .og (for old Greg).
Only pair programming would suffer as the main philosophy of the language would we coding in isolation.
Well, that, and the constant drinking of Baileys from a shoe...
I'm also sure they wouldn't have migrated.
The main issue is not that I give you X amount of time now please fix it coz you have generous time now.
The cost of such a change is ENORMOUS.
As an example, in a recently IPO'd company we had a team of engineers and we had 6 months of work for 10 engineers, round the clock, to fix the Python 2 to Python 3 migration issues alone! and the original creators of the code were long gone! .. and I could only imagine the plight of thousands of other folks in similar boats and other resource constrained boat-less entities!
Then there are distributed packages and libraries that are used by a large set of audience that are dev complete and no longer maintained as such. The cost of fixing those are much much higher.
It makes one more than wince at the thought of going through this exercise again.
Ruby 1.9 was released around the same time as Python 3, with a similar amount of large breaking changes, and no one was holding on to Ruby 1.8 the way the Python community held on to Python 2.7. You could keep using it if you wanted to, but no one did. Ember.js is another example that I've used that was refreshingly well-managed, with a clear (oftentimes automated) upgrade path between minor releases. Using semver and LTS versions, it didn't just break backward compatibility without a significant deprecation period. Those are just a few examples off the top of my head, but I'm sure other good examples exist.
When things are well-managed, you as a developer have the freedom to upgrade when it's a good time for you. When you have production software with real users or a business that depends on it, you can't always just drop whatever you're doing to upgrade, which is why backward compatibility is needed.
If it's done right, there's a self-imposed drive to want to move to the latest with all its improvements. On the other hand, with Python 3, they took away things that people cared about, leaving developers not only with no drive to upgrade, but an irrational impetus to hold on to what they had, even when the situation was fixed at a technical level.
I disagree, I think you're learning exactly the wrong lesson from the Python3 debacle. It took a long time for Python 2->3 because the Python3 developers ignored the need for easy backwards compatibility, just like the Go developers are doing. Python3 required all software, including all your transitive dependencies, to be simultaneously updated. Python3 was dead-on-arrival for many years until there were finally concessions made in Python3 to make backwards compatibility less painful (in this case, so that code could more easily be written to work in both Python2 and Python3).
If the go developers want people to use "install" instead of "get", sure! But give time so that the change can be distributed across the ecosystem before removing the functionality. There's no hurry.
What's the alternative? Deprecate warning old-style string handling and slowly nudge it out? Great, that's not actually decideable, since you need to know how a str is being used.
str are at the very heart of python. This would be a gargantuan task just to write the linter.
AFAIK Ruby's change was that they introduced a notion of encoding to strings. Python literally change the string types and its semantics.
Adding a bunch of other BC breakages was a good idea. Having been responsible for the migration of a large codebase the strings change was by far the most disruptive and problematic one, the rest did not even come close (the distant second was the change to rounding IIRC). Ripping the bandaid and adding a bunch of other BC breaks to clean things up or prepare for the future was I think an excellent idea. Most of those were easy to shim or work around (the syntactic BC breaks were especially trivial).
The problem, as GGP's comments correctly notes, was the incompatibility between Python 2 and Python 3: when Python 3 was first released it was impossible to write a non-trivial program which could run on both version.
Meaning if you wanted to port a project you had to convert everything at once. And as if the odds of that working were not infinitesimal enough, libraries had to fork their own codebases, and either try to maintain completely incompatible codebases concurrently or drop support for Python 2 while literally all their users were still on it.
No, its often misremembered that way; Ruby 1.9 had lots of backward-compatibility breaking changes, there was a long time when lots of things were stuck on 1.8, and much of the other stuff that happened with the Py2 -> Py3 conversion was mirrored. The big difference was that the Ruby community and ecosystem were smaller and less diverse.
Huge chunks of industry and academia run python on windows (usually through (ana/mini)conda). That's a whole 'nother world of encoding hurt.
Especially around the 1.9 transition; Ruby was barely usable beyond the level of toy code on windows. The Ruby on Windows story is actually a lot better now.
https://thoughtbot.com/blog/the-journey-to-ruby-1-9
One key here is that the common subset of Ruby 1.8 and 1.9 was large enough that you could ship a single library for both, and a lot of large ones did, as in the above examples. That wasn't the case initially in the Python transition, and by the time Python3 had started making enough concessions to back compatibility to make this practically possible, people were a lot less willing to try.
(I personally was managing upgrades or a large Rails app at the time; Rails upgrades were, and remain, a bear, but as an accident of code style which just avoided the tricky spots anyway, the changes required to move that code between any two Ruby versions have never been more than a day or two.)
Lots of things; there was for several years a “caniuse” equivalent for Ruby gems tracking if they were on 1.9 yet because it was such a big issue. Bigger issue outside of the Rails-centered ecosystem.)
> I personally was managing upgrades or a large Rails app at the time; Rails upgrades were, and remain, a bear, but the changes required to move that code between any two Ruby versions have never been more than a day or two.
Yeah, a lot of Rails use exercises a fairly narrow slice of Ruby language and core/stdlib directly (that js, outside of library code), while leaning heavily on Rails-provided APIs, so that's not too surprising.
They should have made the unicode handling changes opt-in per package or file. Then, start warning about anything that hasn't opted in. Then, only after a significant migration period, change the default to the new unicode handling.
Upgrading to a new major version should never require changes to your code if you fixed all deprecation warnings in the previous major version. Otherwise, you make incremental upgrades impossible.
The new alternative should always be introduced as opt-in with the old default deprecated, and then made default in the next major version.
str = int
Well python is a duck-typed language, so it will handle the unicode string as if it was a bytestring, and this may or may not work correctly. It's up to the modern code to interface correctly with the legacy code. This could mean manually converting between bytestring and unicode string.
You can override the built-ins by re-defining the symbol in any of: local functions, enclosing functions, or the global scope and it will impact the entirety of that scope.
If I'm understanding your comment correctly this time, it would look something like:
getattr(thing, b'attribute_name')
getattr(thing, u'attribute_with_unicode_characters')
This would work in both python 2.7 and 3.9 (probably works with other 3.X versions but it's the one I actually tried).
Nearly all of computing has made a similar transition in the 2000's and 2010's, and I've certainly debugged my share of encoding issues, but in general it worked for a large majority of use cases.
I think most people would've taken a few encoding issues over the 10+ years long migration.
Python 2 was a code pages world. Not an ASCII world. Also Python’s Unicode isn’t and hasn’t ever been UTF-8 anywhere; rather, it’s sequences of Unicode code points (note that that’s code points, not Unicode scalar values−this and the representation of str are a pair of things I find quite bafflingly bad about Python 3, because it means Python’s Unicode strings aren’t, y’know, Unicode strings, because they can be ill-formed, and they’re stuck with bad fixed-width representations so that any even vaguely interesting string will be stored as (ill-formed) UTF-32).
Python 2’s str type was a mess, used for bytes data, for strings of uncertain encoding, and for Unicode strings. In a vacuum, you might say you could make that work by splitting it up (and indeed the types module was an early endeavour to do that sort of thing back before it became evident it was insufficient). The key is that this confusion and widespread misuse was rife through the standard library and other third-party libraries. If the Python 2 → 3 transition had been just str-as-bytes/str-as-who-knows-what/unicode → bytes/something/str, it might have been manageable. But the libraries were so shot that a straightforward migration was impossible, because str was too many things and it was quite impossible to tell statically what should be done about it. (And impossible dynamically to resolve as well.)
Python would sometimes guess the output encoding on Windows to be ASCII even when UTF-8 was supported by the output device. So you'd end up with encoding-oblivious Python 2 programs that worked perfectly reading and writing UTF-8, and Python 3 programs that failed upon encountering a non-ASCII character.
They've since improved the defaults on Windows, but the changes in Python 3 that caused me a lot of pain were when they made IO to be non-Unicode.
They're still constantly finding bits of code that break cause some customer decided to switch from "delta" to “Δ” in some machine learning log output.
Have an import switch that enables a legacy mode, that will convert your strings into and out of the library? You can even require that the application developers cast their strings into the correct ones if you don't want to calculate it on the compiler.
str is too deeply embedded and used in too many dynamic and unique ways for that approach to work without doing some really ugly or nonperformant or community-fracturing things.
Their point is not that Python3 should not have broken backwards compatibility, it's that Python 3 should not have been entirely incompatible with Python 2. That decision is what hampered migration for years, because it turned out not to be a feasible migration strategy, it would have required the entire ecosystem to migrate in lockstep which when you think about it made no sense.
Python 3 migration picked up steam once the core team backported what could be backported to Python 2(.7), and reintroduced syntactic compatibility features to Python 3.
At that point, and with a few shims for standard library stuff, libraries could create cross-version codebases, which meant they didn't have to try and fork their own codebases. And dependents could migrate at their leisure.
No, my point is precisely that py2 and 3 are not compatible. Even with the backports, py2 and 3 are not fully compatible. Hence the "py23" or "six" dialect, which is the subset of 2 and 3 that overlap, with some shims to facilitate it.
Py2 and Py3 string apis are not compatible. They cannot be made compatible. They can never be compatible. There's no "from __future__" that could make them compatible without a stupendous amount of work.
Not to be pretentious, but I'm developing the opinion that anyone who thinks that the Python team could have just shimmed their way between 2 and 3, does not really understand the Python compute model.
Google literally invented a py2 to go transpiler (grumpy) to avoid converting. Do you think for a second if there was a way to just "from future" their way to a compatible str object, they wouldn't have taken that route?
Which still misses the point, py2 and py3 not being compatible is different than them being completely incompatible, because there is a useful, workable, common subset of Python 2 and Python 3.
The entire issue with Python 3 was that there originally was not, the core team had to be dragged kicking and screaming into carving it out through backports and reintroductions because their original strategy was not workable.
> Py2 and Py3 string apis are not compatible. They cannot be made compatible. They can never be compatible. There's no "from __future__" that could make them compatible without a stupendous amount of work.
So?
> Not to be pretentious, but I'm developing the opinion that anyone who thinks that the Python team could have just shimmed their way between 2 and 3, does not really understand the Python compute model.
How can you claim that when "shimming their way between 2 and 3" is exactly what ended up happening?
> Google literally invented a py2 to go transpiler (grumpy) to avoid converting. Do you think for a second if there was a way to just "from future" their way to a compatible str object, they wouldn't have taken that route?
Have you considered stopping beating up your strawman?
Writing 2/3 python is harder than writing either 2 or 3. Porting 2 code to 2/3 is about as hard or slightly more work than just converting straight to 3. But at the end of it, the benefit is your code can run in both environments.
So let's say instead of pulling the plug, py leadership says "Python '2.8' will only run 2/3 code." How is that any different? You're gonna have the people who already ported, who have no need for 2.8 cause they can run 3, and the holdovers who have yet to port from 2.7.
That's what I mean by there's no shim.
> The entire issue with Python 3 was that there originally was not, the core team had to be dragged kicking and screaming into carving it out through backports and reintroductions because their original strategy was not workable.
Maybe I got into the game too late then, but I don't recall it being this much of a dust-up.
"six" was released on pypi as 0.9.0 on Jun 28, 2010. 1.0 was in Mar 2011. Even if the py core team was dragged kicking and screaming, that's still 4 + 5 years of people who could have been porting, and weren't.
Basically, my primary point is, it was a damn hard problem, and I doubt there's much the core team have done better at the time. All the porting tools in the world won't get people to upgrade if they are resistant for any reason.
E: py3 was released in Dec 2008. So I guess those first two years might have been tough. But to me that seems exactly like the ripe time to work on the translation infrastructure and things like six, where you have public release but no immediate pressure to switch.
3.2 was released early 2011, and P2 compatibility features were reintroduced as late as 3.5 (% on bytes) in 2015, though for my money the last key one was the reintroduction of the `u` prefix in Python 3.3, in 2012. That's the point at which the migration really became feasible without suffering.
Furthermore that's assuming all dependencies were ported, which was not the case: waiting until porting was feasible (and they had the time and manpower to) was also something they had to wait on e.g. Django first added Python 3 support in 2013 (IIRC PyLint only supported 3.4 onwards), difficult to port your own software if the stuff you depend on hasn't been ported over.
And it might well be that the deprecation notice will run for two versions. At the moment no official decision has been made as it is still an active issue.
What's notable about this situation is seeing that their compatibility commitment extends only to the stuff inside the language, and not to the tooling provided to work with the language.
For me, it has caused more confusion than help that the old way of using go get still works, even without any warnings to tell me I should start using the new way. I wanted to upgrade myself to using the latest best practices ASAP, but struggled to learn the proper new way, because all the old ways I do almost automatically still work.
As long as the messages about how to do instead are really clear and helpful, I find it should be better to help people move in the right direction as soon as possible.
This is made worse by the fact that open source is typically developed with a very early public release to drive adoption, rather than that the first decade or so the solution is battle tested in-house and only released when most of the kinks have been worked out.
* do I put in the work to upgrade all my code so that it works with this new version?
* do I jump ship to some other library/language that I've been interested in?
In the Python 3 breaking case, it provided good reason for engineers at major enterprises to start building in Go instead of investing in the Python 3 upgrade.
Another debacle was Angular being completely backwards incompatible, giving an opportunity for React to gain mindshare.
That was an inevitable conclusion from two facts:
1. Python 2 was EXTREMELY well adopted. Like, Java-level, but even deeper in the stack.
2. Python 3 had fundamental changes. (namely byte/string)
The only way for a shorter migration is a change to one of those two facts.
Not for the language itself, but the tooling around it.
Whereas, a breaking language change could impact transitive dependencies of the root project, which a developer doesn't have control over.
This does not infringe on the Go backwards-compatibility promise that attracted me to Go in the first place in any way, and it helps clarify a confusing situation. It's a win-win.
My experience is that a fair number of these utilities are broken anyway, because they rely on "go get" to fetch dependencies, and the dependencies they use have changed too much.
(Yes, that shouldn't happen, but it does happen, and it's one of the major problems that go.mod fixes. You can shame library authors for making backwards-incompatible changes, but that won't fix your tools.)
I kept hearing this about Python 2->3, specifically in the context that people would start writing stuff in Go instead. And it was a little bit true, sure, but there's plenty of stuff that's still in Python that people still figure out how to use.
So I'm deeply amused to hear that people are going to switch away from utilities written in Go and curious what language they're switching to.
Every "update that breaks things" is teaching your users to not update the software, ever. Frankly, it's hurtful to users.
By all means send the deprecation warning, but give a lot more time before removing the functionality. Tools require time to get widely distributed unless there's a huge emergency. In this case, there's no need for the hurry.
> Go 1.17 is deprecating the traditional use of 'go get'
did not link to the original proposal/implementation issues:
initial proposal: https://github.com/golang/go/issues/40276
1.16 deprecation delay: https://github.com/golang/go/issues/42885
1.17 deprecation: https://github.com/golang/go/issues/43684
Seems more than fair to me. Nothing to see here.
1. `go get` will work as expected when run from a directory with a go.mod (or have I misunderstood?)
2. Given the above, the argument about people who include `go get` in their GitHub READMEs makes little sense. Those who blindly copy `go get` from a README will quickly learn to run `go mod init` to set up their projects.
3. There is a deprecation warning, giving everyone plenty of time to adjust to these minimal changes.
Where's the big deal?
I'm reminded of this popular XKCD comic: https://xkcd.com/1172/
Sums up the go philosophy to me. A machine-first language, forces developers to think like a computer.
(The answer of course, is no, but other languages get skewered for less).
Either way, this is a terrible way to do it. I've lived it. It results in libraries that don't work unless you freeze to exact git hashes of dependencies and never evolve because it's too painful.