Go 1.13 Release Notes
golang.org
golang.org
[1] https://go.googlesource.com/proposal/+/master/design/25530-s...
1. https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/
Would be interesting to test the security at some of these companies where this is managements worry. How do these companies worry about dependency leaks but then leave huge S3 / remote exploitable web holes totally open?
If google as an adversary is your worry:
What about all the folks your org emails with that use google in various ways (including perhaps their email stack that they've turned over to google?). How many users are using google single sign on lots of places? How many of their users are using android with telemetry? Browse websites using google chrome? Browse websites that themselves use google analytics.
Google / Apple / AWS / Microsoft are at the root of trust for a lot of what happens online these days.
Many companies have not turned their email over to Google. Many companies are in competition or at least coopertition with Google. Many are trying to get a competitive advantage these days.
All of this was a big enough deal for Mozilla that they got a special contract with Google before using Google Analytics that says Google cannot use the data collected on Mozilla sites for other purposes.
Some won't worry or care about the implications of this. There are a bunch who will. It is not an all or nothing thing. It depends on your business if you are building stuff for a business.
"go env -w GOPRIVATE=*.corp.example.com" was added to make it as easy as possible to configure private modules. If the environment is not set up, just the name of the module will reach the Google services, it will not be published, and an error will be returned.
Anything else would have make it possible to bypass the protections of the new Checksum Database (https://golang.org/design/25530-sumdb), a state-of-the-art authentication system that provides auditable proof that the operator is behaving honestly.
For an example of why the Mirror is valuable, beyond the 10x performance gains (depending on number of dependencies and connection latency) and security gains from not using git on the client, look no further than git.apache.org, which has been down for days breaking a number of builds (https://status.apache.org/incidents/63030p4241xj).
I really think this architecture brings together the best of the decentralized module ecosystem that Go had (with its lack of regularly compromised registry accounts, and no mismatch between code and package origin) and the centralized registries of other languages (with their better performance and reliability).
Particularly proud of how the Checksum Database provides even more security than the centralized registries (because it's cryptographically accountable for its actions, while registries can sign anything they want) without requiring authors to manage keys, register an account, or change their workflow at all.
[Note that I work on the Go team, and co-designed the Checksum Database. But of course I don't speak for the company.]
EDIT to add: I should also mention that the sumdb allows the use of any untrusted proxy, easily configured with "go env -w GOPROXY", for users that can't or don't want to use the Google provided one. The proxy can also opt-in to mirroring the Checksum Database, in which case no user requests at all need to reach the Google services, while still providing all the integrity guarantees that proxy.golang.org users benefit from.
Second, the don't need Git part is a bit of misdirection. Other package manager (Composer for PHP comes to mind but there are others, too) can pull the source at a version right for GitHub. This is much faster than using Git and people have been doing it for some time. Typically it's pulled down as a tarball.
You don't need a proxy to have git out of the loop for downloads or install time stuff.
Third, when it came to making tradeoffs one was made here. I don't know if it was explicit or not. Protections from the global checksum database as a default were put ahead of privacy as a default.
I wonder what other designs could have kept a certain amount of the global checksum protection while keeping privacy. I came up with a few idea while considering my response here.
Fourth, to really speed up performance in things like CI it's much faster to cache the modules locally. This is something used across programming languages for dependencies and most of the major CI systems have this documented. I looked at using GoCenter months ago but didn't opt for it because caching in CI was so much faster.
We were looking at things holistically rather than just with a single tool.
As I said in the blog post, I appreciate the privacy policy being up front. It is the first link on the proxy website. And, I appreciate the configuration to customize the proxy.
A lot of this has to do with the default. Most people won't change it. A lot of people are going to leak proprietary information to Google without realizing it. Everyone here can decide if that is a good or bad thing for themselves. Hopefully, they will learn the config to customize things if they want.
You do with go because go uses git tags for version information and must parse the `go.mod` file at various versions.
Go also supports git repositories hosted in places other than github, where there's no standardized way to fetch a single file at a tag (the go.sum file), or to perform a suitable search for a versoin and its metadata.
> This [pulling github archives] is much faster than using Git
That's only true if you only ever look at a single version of a library. If you have a 20MB git repo at v1.2.3, and ~100KB of files change for the v1.2.4 patch release, using the zip/tar archive means you have to download 40MB total to use it and update it to the new patch. Using the git repository in the first place means the update can only download 20MB since the new `fetch` will be effectively zero data, assuming you cache git stuff well.
The idea that the language I'm programming in now reports anything back to google is distressing.. and I say that as someone who has been programming in Go for about 7 years. What right does google have to collect usage information from modules hosted on github (or elsewhere)?
I checkout the code for the modules, review it, and commit it into my own repository. I don't need you to speed up the rare git checkout of a module. Not that I think some minor speed improvement is worth my privacy.
You talk about decentralization... while centralizing the entire thing around Google. You talk about git.apache.org being down, while you're willing to bring down the entire ecosystem across the board with your own special server.
This "feature" should have an option to disable it.. at the very least.
This is the first language in my 23 years programming that reports my usage back to the language authors. It's unprecedented. And the truth is, if this is ok to you, then there's really no limit what the code I write today, may report back to google tomorrow.
I suppose that means you haven't used any of:
1. nodejs, which reports your usage back to npm Inc with the exact same amount of detail (names of dependencies, ip of caller) (and stores much more, since it publishes 'downloads per month', etc)
2. rust, which does the same with crates.io
3. perl, which does that with cpan
4. python, which does that with pypi
5. ... etc
All of those languages have a central registry of packages that has the same level of detail in the metadata it can potentially collect as the go proxy does.
> This "feature" should have an option to disable it.. at the very least.
It does. Multiple options. You can run your own proxy, disable it entirely, or opt out of the go mod experiment and never run go get, but rather vendor by hand with `git clone` or whatever.
> And the truth is, if this is ok to you, then there's really no limit what the code I write today, may report back to google tomorrow.
Very much a slippery slope. They've communicated clearly how to disable it and what it does, it's in-line with what other language ecosystems do in terms of metadata reported to a central package repository, and I think it's very easy for someone to be okay with this, but to not be okay with any step beyond this line.
What company is using python and building something in competition to the python software foundation? Or something for the others?
This difference is worth taking into account. If Go were part of a software foundation like Python this would be a different story.
This is different. It’s a proxy in front of others repositories. They want info on downloads from GitHub, gitlab, and anywhere else you get code from.
They (google) are a 3rd party to the download.
It's not all that different.
It's perfectly possible to use gems in ruby without ever using the rubygems server by setting every gem to be something like 'gem foo, :source = "https://github.com/foo/bar"'
However, people want to use this centralized repository in the ruby community because it allows for easily updating gems and for convenient caching.
The go proxy is not just a proxy. It also pre-parses out go.mod/go.sum to provide metadata faster and to provide security guarantees.
In a similar way to how centralized repositories are helpful in other languages, the go proxy adds actual legitimate features.
If you look at the public information on its development of this feature, all the motivations cited are about security, performance, and useful features, not information collection.
> This is different
There is a difference. Other repositories, like npm, require someone to opt-in to code being available via it (e.g. type `npm publish`, whether the publisher owns the package or not), while the go proxy default opts-in everyone that publishes a go package publicly and uses the newer go toolchain.
In practice, with how the npm/rubygems/etc communities work, everyone does opt-in if they wish their code to be publicly used, so the difference doesn't seem meaningful to me.
Other than that, the difference seems vanishingly small. In both cases, a 3rd party exists that serves downloads, and it exists to provide performance and discoverability benefits (with the go proxy also does due to supporting version-related operations), and in some cases security benefits.
But why? Who asked for this??
CPAN doesn't do that, because CPAN is a series of mirrors that neither collect nor report aggregate download numbers.
You should be able to. It's not the ideal solution (or recommended for most use cases), but you can by setting GOPROXY=off. GOPROXY=direct will force it into its previous behavior.
The same should be true for the checksum DB with GOSUMDB=off. I don't remember specifically and can't find the page its options were documented on.
Also look at GOPRIVATE and GONOSUMBDB.
But, it's still possible to disable these features, so it's not quite as bad as the OP suggested.
If the proxy goes down, by default the command falls back to directly fetching.
For example, this should in theory work to make the google-run proxy your fourth choice, with three other proxies attempted before the google-run proxy, and with direct (no proxy) access as the fifth and final fallback:
go env -w GOPROXY=gocenter.io,goproxy.io,goproxy.cn,proxy.golang.org,direct
The Go 1.13 Release Notes lead off with documentation links for how to disable and otherwise configure this:
"See https://proxy.golang.org/privacy for privacy information about these services and the go command documentation[1] for configuration details including how to disable the use of these servers or use different ones. If you depend on non-public modules, see the documentation for configuring your environment[2]."
[1] https://golang.org/cmd/go/#hdr-Module_downloading_and_verifi...
[2] https://golang.org/cmd/go/#hdr-Module_configuration_for_non_...
There seems to be an open issue and WIP PR with more details:
Issue: https://github.com/rust-lang/crates.io/issues/955
PR: https://github.com/rust-lang/www.rust-lang.org/pull/919/file...
The cargo manifest also supports things like:
The publish field (optional)
The publish field can be used to prevent a package from
being published to a package registry (like crates.io) by mistake,
for instance to keep a package private in a company.
[package]
# ...
publish = false
The privacy concern is entirely that Google may see the name of some module you attempted to download.
No surprise there, was even called a 'privacy nut' on the neighboring Android 10 thread for highlighting the nature of Google's 'privacy' stance. So essentially they turn on tracking / analytics on by default for module usage without consent or any notice except for the lengthy privacy policy. This reminds me of when Microsoft VC++ embedded analytics in the developer binary when using the compiler. [0]
But then again, Google can do what it pleases since Golang is authored by Googlers.
[0] https://www.reddit.com/r/cpp/comments/4ibauu/visual_studio_a...
There's also there in release notes:
> "The GOPROXY environment variable may now be set to a comma-separated list of proxy URLs or the special token direct, and its default value is now https://proxy.golang.org "
To be abundantly clear, and because a lot of Google employees read this, the we-know-better comment isn't a jab - because often Google engineers DO know better, but it does occasionally mix with i.e. poor communication and create frustration for the many developers who use Google services and tools.
While opt-out for this not ideal, this seems like a minor privacy risk relative to the security risks ameliorated by enforcing checksums by default.
I guess an option which solves would be a mandatory prompt about proxy usage unless there is an explicit environment variable set?
I'm salty about how the proxy, which is enabled by default, makes it so much more difficult to pull private modules by having to set environment variables on everyone's individual machines.
I've got an issue being discussed on GitHub about it https://github.com/golang/go/issues/33985
go env -w GOSUMDB=off
10 * 1000 * 1000 worked pretty well too.
I know we have scientific-notation integer literals, but it's not the same thing, especially when trying to calculate e.g. Mebibytes.
var x int = 1e9 #define foo 1.34E8
10_000_000 can be read, 5 ＊ (1000＊＊4) and 1e7 has to be decoded. By me anyway, I'm just an average developer that never did well in maths - and you have to write code for the average developer, not for yourself and not for the top 10% smartest people.
1e7
12_345_678
1.2345678e7Kind of realising how much of my daily work could use bytes as the main numeric type
Is a float literal.
var u uint32 = 10000000000.0 / 100000
Yes in go. `var u = 1e7` gives a float. That means 1e7 is a float literal. You can coerce it to an integer by explicitly typing the target, but it's still a float literal.
O_o
gofmt -r 'y << uint(x) -> y << x'
Reshaping
Redefining
Removing
Restricting
And he follows up with examples of how they applied those to the code base. If you've not seen it you might enjoy it.
In a different life, I worked deep in C# and the bowels of the CLR. New versions of C# were both exciting (woo, LINQ, anonymous things, lambdas, and a dozen others each release) and frustrating (ugh, visual studio doesn’t have specialized tools to handle this new stuff).
I’ve been writing Go for 8 or 9 years now and I value the tool chain improvements far more than the language features. Race detection was a godsend. Indeed, even taking away capabilities (or just being more strict) has been valuable—like when multi-threaded access to slices became an error.
In this release, I don’t much care about the new literal prefixes. But I’m excited about the improvement to defer performance and the improved range check panic messages.
In a sense, other languages make it easier to to write new code with each release, but every iteration of Go makes my old code better too.
At this point, I think we all have some JavaScript PTSD.
Call me a bit of a luddite but this is one of the reasons why I'm skeptical about generics in Go - it'll be another language construct to keep in mind. Mind you this opinion of mine is baseless because unfortunately I haven't had the opportunity to do any serious go development yet.
Not sure if that's true or not though. Rust is my daily driver for the last year and even coming from Go (after 6 years) I don't feel the pace is hard to keep up with, or that my code breaks frequently.
In fact, I don't think my code has broke as a result of changes once since I switched to Rust. It has happened .. twice maybe, via dependencies, though.
This is why I've been bleating a bit more now than I have before about reducing churn.
N.B. On reflection, perhaps I do have some bias here. My daily work on Go is mostly for $work, and involves one large codebase. My daily work on Rust, however, involves maintenance of dozens of crates. So if there's churn, I feel it repeatedly. So its effect might be magnified for me personally.
* Excluding patch releases, Go has a release twice each year [1], while Rust has a release every six weeks (about 9 times per year) [2].
* For minor (patch) releases (security updates, etc), Rust only cares about the latest version released. Go provides patch releases for the last two releases [3].
* The latest release of the Go compiler (1.13, Sept 2019) is able to bootstrap itself from Go 1.4 [4], released on Dec 10 2014. The latest release of the Rust compiler 1.37.0 (Aug 15 2019) needs Rust 1.36.0 (Jul 04 2019) for bootstrap.
* For the language, I think relative to itself, Rust has considerably slowed down with development compared to the pre-1.0 times. Now it's still evolving more quickly than Go. Rust is getting async_await, const generics, etc while Go is discussing about an error handling operator that Rust already had two revisions of since its 2015 release (try! and ?).
I'm not saying that all of this churn that Rust has is bad. Some of it is good, like the tighter release schedule that prevents half finished stuff to be released. But other things, like the tight version range that the compiler can bootstrap from, could be improved.
[1]: https://github.com/golang/go/wiki/Go-Release-Cycle
[2]: https://blog.rust-lang.org/2014/12/12/1.0-Timeline.html
There are major differences in how easy it is to write compilers in C++ vs C89. There are major differences between C++ and Rust with its ADTs and pattern matching. But are there major differences between, say Rust 1.32 and Rust 1.36? I'd argue no.
There are real use cases like distros wanting to maintain/bootstrap a Rust compiler independently from upstream binaries. The more often these people have to port a compiler, the more troublesome it is for them.
Rust seems like it's the odd one out: Swift is written in C++, Go supports bootstrapping from years old compilers, LLVM does so as well, and GCC has probably some of the best backwards compatibility stories out there.
D has been replacing the C++ code with D, .NET made a major reboot with Rosyln where VB.NET and C# got bootstraped (F# was already bootstrapped), OCaml and Haskell have only the runtime in C due to convinience with everything else bootstraped, FreePascal is bootstraped, OpenJDK has the long term goal of replacing C++ with Java/Graal, Jikes was bootstraped in Java, ...
I also agree an yearly baseline would be much better.
[1] - https://users.rust-lang.org/t/rust-in-large-organizations-me...
In a sense, Go is more like a corporate-internal DSL that the public just happen to be able to use. (Erlang is—or at least, was, for the first ten years of its life—another language that is this way.)
- 2018 roadmap https://blog.rust-lang.org/2018/03/12/roadmap.html
- 2019 roadmap https://blog.rust-lang.org/2019/04/23/roadmap.html
It's a big community, and it has time for a lot of development
Go always had a very narrow and precise ambition and scope (a better C, aimed at server-side data plumbing). Which is the reason why it was able to nail a set of features from the start and keep it that way.
Rust has a much wider ambition : all the modern languages facilities (generics, metaprogramming, etc) with the speed of C, AND breakthrough memory-safety features.
Go doesn't try to innovate, and i suppose the author see innovation as something you should stay away until it's nothing new anymore.
I agree with all what you said but this. Go isn't a better C, it's a better Java.
Not to go off on too much of a tangent, but I think asyncio is a huge mess and was a mistake. I still use gevent for everything in Python. https://glyph.twistedmatrix.com/2014/02/unyielding.html is frequently cited as an argument for why goroutines and gevent and green threads suck and why explicit asyncio-style concurrency is better, but I have a lot of issues with this article and find the former two way easier, simpler, and faster to work with.
Ah ha ha ha ha ha ha. No. Just no.
I say this as someone who writes Go day in and day out and wrote Java for a decade.
Maybe it is if the last time you touched Java was 2001.
Currently available on PTC, Aicas, IBM, OpenJDK AppCDS (originally from BEA J/Rockit), GradleVM native images, Android ART AOT compilation.
Dynamic heap size, since ever. Every JDK vendor had their own specific switches to configure it.
Versioned modules, since Java 9 alongside Maven/Gradle.
Value types, yeah point taken. There were Azul and IBM specific extensions, ObjectLayout and PackedObjects respectively, and the 2nd experimental release for value types was just made recently available.
Maybe you are referring to the "real world" where developers don't pay for their tools.
As for embedding 150MB of JRE / JDK, it is hardly any different than embedding the Go's runtime into every static compiled executable.
"all the modern languages facilities" seems to be something that's come along more recently (last couple of years) as a result of a lot of non web browser developers taking an interest in the language.
Sure, async needs to be shipped and polished, but then Rust needs to tell the world: "We have all you need and as stable as you need."
Oh wonderful! This makes switching back and forth between module projects and 'legacy' projects far easier. (Well, at least for me - since I tend to keep all cloned git repos inside the GOPATH. I don't have a problem, really!)
I'm definitely going to see if Go is acceptable for my next project since I really need the performance for my workload, and I've wanted to put this great language to work in something that isn't a toy project.
This is exactly how I felt moving from a Python codebase to Go. But you realize on the 100th or so Python stacktrace that it's worth the investment up front, if not for type safety alone.
> I really need the performance for my workload
Careful with that assumption. A pure Go application will in many cases be more performant than a pure Python one, but Python has a better C interop story. In the particular cases of parsing XML and JSON, I'd wager Python is much faster, actually.
Yep. Maybe I just suck at programming but I couldn't write enough tests to avoid not being fed up debugging stack traces in production. I started using Python's type hints, since this was the source of most of the bugs which helped but I'd rather just use a strongly typed language next time.
> Careful with that assumption. A pure Go application will in many cases be more performant than a pure Python one, but Python has a better C interop story. In the particular cases of parsing XML and JSON, I'd wager Python is much faster, actually.
I've got a few million records pulled from an API so I'm extremely I/O bound and goroutines are easier to work with than Python concurrency. I've also got reasonably complex data processing beyond just XML/JSON parsing.
I think on the whole Go is still going to be faster, but thanks for the heads up.
Go's not perfect in this respect, either. I had your typical map[string][]interface{}, because the data in the map was an array of records, but different records in different cases, and made an error in an append(), so ended up with nested arrays. That's correct according to the type definitions, but unwanted. Generic could have helped here. So I think the language still needs to grow a bit.
Up until you need to do anything interesting with the deserialized data. :)
Luckily, you don't have to choose between those two outcomes:
You can create sum types (a bit verbose but they are there).
You can introspect them at run time and create things like pydantic[0] and fastapi[1].
Strict null checks with Optional[T] are great!
Now I'm not too familiar with Go's type system, but with protocols, Unions, Optional[T] and other generics offered by MyPy[2] I'd wager the type system is __more__ expressive than Go.
[0]: https://pydantic-docs.helpmanual.io [1]: https://fastapi.tiangolo.com [2]: https://mypy.readthedocs.io/en/stable/protocols.html
Not sure I fully understand the NFA vs DFA/re2 choices -- seem like the issue above conflicts with the comment below. Can someone clarify?
From [1]: "Go has chosen to follow the re2 path (not surprising, since Russ Cox is a major author of both Go and re2). re2 has much better performance characteristics than some other regexp engines, in that it never has an exponential slowdown, but that comes at a cost for other regexps (https://swtch.com/~rsc/regexp/)."
[1] https://www.reddit.com/r/golang/comments/84o986/why_is_gos_r...
Using encoding/json maybe but also using 3rd party libraries? jsoniter is pretty fast, I just ran some tests and for my use cases it outperformed Python.
I am slowly ending up with a library of utils from doing this sort of work. In the end it does end up more stable (and more verbose, which is OK with me) and easier to maintain. I've said for a while Python makes me productive and Go makes my whole team productive.
Have you got any relevant advice I can take? Interested to hear how people feel about the language before I dive in.
He doesn't talk about learning the language at all (he assumes you know all that), instead he fills in all the gaps in a very thorough way that are left out from all the tutorials and such you'll read. I've read dozens and dozens of Go articles while learning and I would say 85% of those 15 hours was NEW information to me, that's how good it is.
https://learning.oreilly.com/videos/ultimate-go-programming/...
There are 2 versions, I watched the older version because he's more lively in it, however I have heard the newer version is better because he does cover a lot more up to date tooling. I went and rewatched the section on debugging and structs on the new version.
For example, its 15 hour course, and in the literal first 5 minutes of the first coding video, he talks about struct field memory alignment on different CPU architectures. That's when I refilled my coffee and said 'dis gon be good'.
Aside from that, I would say pick a good project structure that works well, invest time time into researching the different ways (also covered in that course)
I also use go modules which are new, and make things way easier as far as packages.
Definitely make sure your editor has gofmt and goimports set to auto-run on save, those are lifesavers. Goimports is smart enough to look across packages and in your own package, which is cool
All the other suggestions about actually writing code is basically covered entirely in the ultimate go course (when to choose pointers vs structs, how to structure code, good error handling practices, etc)
- Getting started with go and vscode is kinda bad right now. I installed go, set up GOPATH, installed all kinds of extensions in vscode and then, somewhere down the official tutorial I learn about GO111MODULES, go mod init, vendor folders and the language server... I was completely confused and still am. I removed all extensions except the main one from vscode and the language server was crashing like crazy until I unset GOPATH. Works now, mostly. Intellisense and code formatting seems to be still unstable now and then. But geez, that first experience was horrible.
- The syntax is very weird after writing too much C#. Sometimes I have the feeling, that declarations are reversed just for the sake of it.
- Why does go need := with the colon? I know it's a declaration, but I haven't figured out the reason for the colon.. Seems weird..
- I love gofmt and format-on-save. But it was not the default setting. Still don't know if I should use another one.
- I hate configuring vscode. After hearing only good things about it, the configurations and keybindings are horrible and that I need an extension for each and every small thing. Maybe I'm in the minority, but Visual Studio is miles ahead in this regard, I think.
- Fast compilation times are awesome. I love, that I can create a simple exe without a framework dependency.
- I can parse 2GB of simple JSON files in less than 20s. I'm awed.
- I have no idea how to compile some modules like the sqlite one. Seems to need gcc..? Do I have to fully switch to Mingw/WSL?
- I was losing my mind when I saw my program giving different results on each run... Until I found out about that randomized map iteration.. Funny.
- Finding an item in a slice: Yeah, sometimes generics would be nice but let's not start a holy war...
- Need to get used to error handling.. It can get really verbose and those scoping rules are strange.. I miss exceptions, but I can get used to the go way.
- I love that I can use the official docs locally: godoc -http=localhost:6060
- I had to use google and stackoverflow more than I initially thought for a 'simple' language... But well, I guess that's normal?
Sorry for the rambling. I love the simplicity of the language and hope to get into it more. But the initial experience is still, umm, not streamlined, it seems? But I hope that's just temporary.
It's called a short variable declaration that can be used in place of `var` and the type is implicit.
var s string
s = "foo"
s := "foo" if err := fn(); err != nil {
}
// the result of fn does not affect the identifier err
// in the enclosing scope
You'd have to do this to make the value stick around or affect the enclosing block:
var err error
if err = fn(); err != nil {
}
// err has the result of fn here foo = 3
fooo = foo + 1
foo := 3
fooo = foo + 1 // oops, typo found at compile-time
foo := 3
fooo := foo + 1 // Yep, new var, everything is okvar foo = 1
The weirdness is that ":=" allows redeclaration (within a scope) but only partial, whereas `var` completely forbids redeclaration.
That is:
var a, b = f()
var a, b = f() // NO
var a, b = f()
a, b := f() // NO
// note that the second declaration only partially overlaps with the first
var a, b = f()
var a, c = f() // NO
var a, b = f()
a, c := f() // yesWith ":=" will declare with implicit type and then assign. The type will be that of the right hand side.
Yeah, there are a lot of reasons for this. Things used to be easy when it was just GOPATH, but that didn't support every use case, so modules were added. Now the advice beginners get is fragmented and confusing to navigate. Also, the error messages for modules are terrible. These are tractable problems, but it's definitely a rough time to be a beginner (which is unfortunate because Go is otherwise very beginner-friendly).
VS Code is a different product altogether and its Go integration is pretty buggy in my experience, especially its handling of modules. Go really needs a bullet-proof (free) editor story. If you want a premium IDE (a la Visual Studio), I've heard good things about Goland.
> Need to get used to error handling.. It can get really verbose and those scoping rules are strange.. I miss exceptions, but I can get used to the go way.
The scoping rules are pretty straightforward and standard. I'm guessing "shadowing" is tripping you up. Error handling is verbose, but this is a good thing IMO; it's really clear what is happening and there is no special control flow for certain data that is arbitrarily classified as an error. The real issues with errors are the lack of stack traces and other standard structures for programmatically inspecting / manipulating errors. Some of these issues are being addressed in go1.13 but it will take a while for the best practices to be established and propagated through the community.
> Sorry for the rambling. I love the simplicity of the language and hope to get into it more. But the initial experience is still, umm, not streamlined, it seems? But I hope that's just temporary.
I think it will be. You're entering at a time when there's some churn in the best practices and it makes for a confusing time to be a newbie. Hang in there!
Really hope to use GO more and that those rough edges will be gone after a while.
Currently my advice to them is ditch VS Code and download GoLand, the editor change is jarring but the experience is much more fluid on Windows overall. Also transparent support for modules, per project.
For compiling when your code (or a package you're using, e.g. sqlite) has a cgo dependency - afaik YES you will need to go down the dark road of cygwin/Mingw, if you want your development cycle to be at all normal... and even then, my observation is compilation times are an order of magnitude slower on Windows :(
At GopherCon I met folks who worked for huge, "old guard" Windows companies who said "just run a linux VM" to develop in Go.
If you just need to compile for windows once and a while (e.g. for releases only, not as part of your normal dev cycle), check out xgo [1], which uses docker to deal with all the cgo mess. No cygwin/Mingw to install/worry about.
Best of luck! If anyone else has tips for working in Windows, I'd also love to hear them.
I'll try the WSL Route and see how it goes. (VSCode already made me install that extension.)
Rob Pike explains the reasons behind Go's deceleration syntax in this blog post: https://blog.golang.org/gos-declaration-syntax
It's also funny, that they stuck to the * pointer notation which led to the problem, where they could not tell apart a pointer from a multiplication, so they put it to the left of the variable name. It's the small things...
But I guess * is just too convenient and ingrained.
Without the : it means every assignment might be a declaration so you need a "declaration inference" system, and every language which does declaration inference has odd tradeoffs and corner cases e.g. implicit declaration in the local-most scope (Python), implicit method-wise non-local (ruby), implicit global (javascript), …
Explicit declaration is a very good decision I think. As far as I'm concerned it's one of Python's big annoyances (and javascript but there it's easy to lint away as you also have explicit declarations, you can just ban implicit ones).
It's funny, how many different answers I got. I guess, all those reasons combined justify :=
- Complicates compiler
- Prevents Typos
- Works well with err (also because of partial declarations)
- Variables can be shadowed inside a closure
The compiler part's easy, it's the meatpiler which gets into an odd funk when implicit declarations don't do the expected thing (which will eventually happen). Not having implicit declarations is much simpler and more straightforward for everyone involved.
I used to find implicit declarations NBD, but I've slowly soured on them (starting with CoffeeScript: I found its declaration inference decisions absolutely awful and that started me really thinking about implicit declarations, see http://lucumr.pocoo.org/2011/12/22/implicit-scoping-in-coffe... for more on the subject, or https://donatstudios.com/CoffeeScript-Madness for a more brutal take).
I remember Rob Pike ever said that one if the reasons of needing := is to avoid declaring many error variables in a function, such as err0, err1, err2, ..., and at the same time we don't need declare then as uninitialized variables.
With :=, they can be all called "err".
Personally, I do prefer to try to avoid using :=.
It's really C and languages directly derived from it which are the odd ones out. "postfix type" ordering dates back to at least Pascal, and is very common especially but not solely amongst languages with type inference.
Just look at what more direct C derivatives have to do to back-fill type inference: invent some weird-ass pseudo-type to take the type's place. I also think it maps better to how I read / write / think. "f is an integer", the definition of f is more relevant than its type.
> - I can parse 2GB of simple JSON files in less than 20s. I'm awed.
100MB/s doesn't seem that impressive? rapidjson or serde can do several times that, simdjson reaches 2GB/s.
So far I've found Rust more interesting and pushes the envelope a bit more. But what is the sales pitch for Go?
- with it's M:N model, Go cannot really do FFI efficiently, while both 1:1 async/await have no problem with that in Rust.
- goroutines are cooperatively scheduled, while OS thread are preemptively scheduled. If you have hot loops, Go's model won't be able yo guarantee fairness between goroutines, which may be a problem depending on your usage. OS threads don't have this problem.
- with goroutines you won't achieve the level of performance you can reach with async/await.
Go and Rust fills a different niche of programming and take a different stand on this tradeoff: Rust gives you more power at a complexity cost. Go offers you more simplicity, with less power.
Small stacks (and M:N threading) are needed to efficiently implement tens of thousands of goroutines.
CGO is slow mostly because it needs to switch to a larger stack when calling a C function.
It's a trade-off.
A different Go implementation could make Goroutines map 1:1 to threads and have fast cgo calls at the expense of slow goroutines.
Rust is not exempt from those trade-offs. They chose fast FFI and smaller runtime. They paid with slow threads.
Async/await promises to be the best of both worlds but it comes at a cost of great complexity, both for the programmer and the implementor.
At the end of the day under the covers it's just threads that need to be managed in a complex and often invisible way plus a complex rewrite of your straightforward code into a mess of a state machine.
I can confidently say that learning to use goroutines took 10x less time than learning async/await in C#.
I understand goroutines better than I ever understood async/away.
Tens of thousands of threads are no problem with 1:1 threading on Linux.
> They paid with slow threads.
I would not call 1:1 threads "slow threads". If they were slow, then the 1:1 NPTL would have not defeated the M:N NGPT back in the day when this was being debated in the open source OS community.
> complex rewrite of your straightforward code into a mess of a state machine.
The entire point of async/await is that you don't have to do a complex rewrite of your straightforward code. It remains straightforward, and the compiler does the transformation for you.
Yes it's a problem, even on modern kernel, memory, context switch ... there is a reason why high performance code uses small thread pool.
Yes, That's exactly my point.
> I can confidently say that learning to use goroutines took 10x less time than learning async/await in C#.
It's a matter of preferences, I've always preferred async/await to threads because you don't need to juggle with channels and you're way less likely to cause a deadlock.
> it comes at a cost of great complexity, both for the programmer and the implementor.
Is that really harder to implement than a M:N runtime like Go's?
> The reason cgo is (relatively) slow is small stacks, not M:N threading.
You're picking nits here. Without small stacks there would be no point using M:N threading in Go…
And Cgo is not only relatively slow: most of the time it is slow enough to destroy all performance benefit of calling a FFI function. It's a pretty big deal.
- async/await is available in the nightly version of Rust (1.39), but not in the stable version (1.37).
- 1:1 threading is not strictly equivalent to M:N threading (the main practical difference being that the stack size per OS thread is larger than the stack size per goroutine).
Writing that "Rust has broadly the same concurrency support" is misleading right now. But it could be true in a near future.
Rust I/O are strictly similar to Go when using 1:1 OS threading, but become conceptually different when using async/await.
> Rust has more features for safe concurrency--Rust prevents data races statically, while Go is not even memory safe in the presence of such races
Agreed. That's a big advantage of Rust.
> What is different is the performance characteristics.
Performance is a feature. It's significant enough to justify using one language or the other depending on the project at hand.
> In some cases, M:N will be more efficient; in some cases, 1:1 will be.
The main advantage of the M:N model, compared to the 1:1 model, is the memory usage, because each goroutine starts with a small stack (a few kB). It makes possible to start a larger number of M:N goroutines than 1:1 threads.
> But just as I wouldn't say Go is lacking FFI features because M:N makes cgo slow, I wouldn't say Rust is lacking concurrency features.
Go's FFI works but is slow. It's a well known fact. Rust's concurrency story is not stabilized yet (areweasyncyet.rs). It's a fact too. I don't see a problem with acknowledging both :)
Rust's async-fns compile into state machines, so they only allocate exactly the number of bytes that they need.
Go's concurrency shines best in short-lived single-purpose processes, which is a near perfect fit for CLI tools. When you don't care if something gets abandoned or fails to make progress and can just ctrl-c, the relative simplicity is 100% worth it. Other parts of Go work well here too, especially the very low startup overhead (unlike, say, python). For long lived processes though, like most services, it feels very error-prone or very boilerplatey and manual.
It helps if you follow certain principles like making your concurrent tasks idempotent etc., but Go doesn’t force you to write things that way, so it takes a bit of remembering each time.
If the task is relatively simple, you end up writing a lot of concurrency infra for a little bit of concurrency, which can become frustrating if you’re doing it for the Nth time.
It sounds like a problem that’s ripe for abstraction, but it turns out that it’s quite hard to build abstractions that get all the trade offs just right for each case. There’s some worker pool style libraries out there, but if you care about performance you can usually eke more out by writing a custom solution.
As for alternatives, I have heard that erlang has a better story for this, because of the inherent restrictions of the language, but I haven’t used it much.
In my experience, Go gives you enough rope to make this task “merely annoying” (rather than impossible or astoundingly challenging) without giving you enough to truly hang yourself with. You can get really tangled up though :)
Go's concurrency is cheap[0] multithreading (shared-memory concurrency). It has a tagline that you should not share memory ("Do not communicate by sharing memory; instead, share memory by communicating"), however nothing prevents sharing memory (so it will eventually leak in) and channels are pretty slow[1] so it's not uncommon to go back to regular shared-memory primitives (with no safety net).
You've got no idea what your application state really is (even less so once things start falling over), and there are no built-in tools for reliability e.g. you can't get notified that a goroutine has keeled over or anything unless you implement that yourself.
> And perhaps give an example of a language that makes it easy/easier?
Erlang / elixir was built specifically with that in mind. It uses lightweight shared-nothing concurrency (non-OS processes) and the system builds-in ways for processes to have oversight over others (e.g. monitors & supervisors) or get killed alongside them (links) and provides behaviours which bundle known patterns, etc…
[0] as in "not expensive", as in uses quite a bit less memory & especially kernel resources than OS threads or processes[1]
[1] though this point should not be too overstated, people commonly misuse "the C stack is XMB" with "creating a thread consumes XMB of physical memory", a C stack is mostly a regular allocation and thus usually just VSS until actually exercised
[2] and not necessarily a huge gain in bug-freedom: https://songlh.github.io/paper/go-study.pdf
I haven’t really benchmarked channels since the bottlenecks in code I write tend to be I/O, but I’ll play around a bit to understand that better.
This is not reasonable. Go's concurrency shines for its flexibility. It has nothing related to whether or not the written programs are short/long-lived. It is the only language which gives me a happy and fun experience in concurrent programming.
> For long lived processes though, like most services, it feels very error-prone or very boilerplatey and manual.
This is not reasonable too. Whether or not a written program is robust depends on the experience of the authors. Rust may reduce many mistakes at compile time, but it can't prevent all mistakes. Programmers still need experience to avoid many mistakes.
Which is true to some degree, you can build anything in anything if you're careful enough. But programming history is full of repeated, preventable errors, and languages where X error is impossible or much less likely if you're less than perfect.
Go's rather far down "unsafe" in my books, especially in regards to concurrency and error handling. For simple code, it's pleasantly simple. For larger, robust code, it's several times more verbose than other langs I've used, and often requires continually re-implementing common "wheels" (that should not usually be written by hand, as they're finnicky to get right, which is also easy to miss in code review) to get around a lack of generics.
Rust does prevent some errors at compile time, but achieving this has its own cost, such as sacrificing flexibility and simplicity. For many use cases, the sacrifices are not worth it.
> it's several times more verbose than other langs I've used, and often requires continually re-implementing common "wheels"
If the common wheels need generics to implement, I have nothing to defend. I can only say generics will come to Go eventually.
Something different than that? Yes.
Sarcasm apart, I wasn't talking about features. But cargo is a really well-designed package manager. I've been using Rust full-time for 2 years now and it had never annoyed even once.
[1] the sarcasm was referring to the Minimum Version Selection algorithm used by go's package manager, which is a really bad case of NIH from Go's teams who decided to throw away a thriving community work to do their own stuff, ending up with something different from what everybody else has been for more than 20 years. And unsurprisingly, it's really bad…
You can replace a dependency with a local modified copy of the dependency to do this.
Minimum Version Selection is for security reason. It might has some disadvantages, but personally I like it. It gives programmers more control.
On my bubble everything cloud is either Java or .NET, alongside VM images, and native OS containers.
Azure and AWS also have hardly any major tech done in Go.
1. Cross-platform 2. Fairly mature ecosystem 3. Pretty easy to pick up coming from either a dynamic or static language background 4. Somewhat simpler approach to concurrency (although I've created my share of race conditions) 5. Backed and used by Google (if that is positive for you as a developer) 6. Is a pretty good multi-purpose language
I think in comparison to (my limited understanding of) Rust, Go is beginner friendly. Also, I think Rust has most of the above covered as well, but it seems like there are lower level primitives with functional aspects built-in. Haven't needed these in Go, so haven't investigated them there.
Perhaps I’m old and grumpy but I feel that at this point I’ve had every language discussion before and if people start to moan about lack of metaprogramming support I just leave.
I have yet to see a project that failed due to lack of high order abstractions
I’ve also abandoned a few paths during projects because of libraries that were overly abstract, didn’t work and were thus hard to understand and maintain.
A product at a company I used to work for had to reimplement a huge chunk of code because the original designer left and the metaprogramming behemoth he wrote was too hard to understand, extend and debug. That one chunk of code cost about $500k and 6 months to replace. I’m unsure of the total cost of that effort as it delayed the next release at a critical time.
Abstraction levels need to be appropriate. Enough to make things simple - not so much that it makes life harder.
I’m not quite sure this is what I meant from the description of the problem. This looks more like a problem of trying to program Java in Go. What I was thinking of was more along the lines of modeling things in ways that either disconnect you from the problem you are trying to solve or solve it with too much abstract cruft grafted onto the model so just dealing with the abstractions themselves becomes more work than a more direct and naive approach.
More accurately speaking, Rust forces people writing software in a specified way. You need to think about how to write software much less in Rust than in Go. In Go, you really need to think more to choose a better solution from many potentials.
And mind you: I manage quite a few projects where Rust would be a great fit because it would replace C/C++, but for embedded projects Rust just isn’t getting the traction it needs to convince key industry players.
Some more details: https://github.com/golang/go/issues/30116
That's pretty cool
Edit: :)
First, a string can point to an unlimited amount of data (i.e., it can be of arbitrary length) and all of its characters are taken into account when deciding string equality.
Now, an array type in Go is not like that: the length of the array is part of the type. There is (or at least there wasn't when I last checked) no type of all arrays of ints, say, but there is a type of array of 5 ints, a type of arrays of 8 ints, etc. Each of those is of limited size.
A slice is like a string, it can point to an arbitrary amoun of data, but slices are not allowed as map keys.
A struct has a fixed number of members and is only allowed as a map key if all of its members are of types allowed as maps keys.
So a struct type that is allowed as a map key type can only point to unlimited data all used for equality if it contains directly or indirectly a string somewhere. Among types allowed as keys, the only source or arbitrary-sizedness are strings.
For example, if you want to represent multivariate polynomials sparsely as a map from exponent vector to coefficients (i.e., the polynomial 3x^2y^3-7xy^4 corresponds to the map {(2,3)=>3, (1,4)=>-7}) you can only do it in Go with the built-in maps awkwardly. You could pick a maximum number of variables and use arrays of exponents as keys. You can leave the number of variables arbitrary and convert exponent vectors to strings (i.e., use {"2,3"=>3, "1,4"=>-7}). I find it frustrating that those are my options.
[1]: https://go.googlesource.com/proposal/+/58566e5309365c84587ae...
I guess that if immutable slices became a thing, one would probably be able to use them as map keys. But so far the Go Team is very reluctant about adding immutability to the language.
package main
import "fmt"
func main() {
x := [2]int{1, 2}
m := make(map[[2]int]int)
m[x] = 5
fmt.Println(m[x])
x[1] = 3
fmt.Println(m[x], m[[2]int{1, 2}], m[[2]int{1, 3}])
for k, v := range m {
fmt.Println(k, v)
}
}
5
0 5 0
[1 2] 5
It's true that you can't use a slice as a map key, though, even though slices are also fixed-size (pointer, length, and capacity). As I recall, the argument is that the equality rules for slices are unclear. (maps and funcs can't be used as keys either, for the same reason: they're not comparable.) Is a 0-length slice equal to a nil slice? Is a slice with len 3, cap 3 equal to a slice with len 3, cap 4?
In practice, this is rarely a problem. It most commonly comes up when you want to use []byte as a key, in which case you can convert the []byte to a string. This overhead of this conversion (specifically when it occurs in the context of a map key) has been optimized away for a few versions now.
If it is, it means Go is like Lua, where if you want to use any compound data structure as a key in a map, you either marshall it to a string or you implement your own hashtable.
In Python you are allowed to use tuples of immutable objects as keys, for example. Much nicer than serializing everything to strings.
As for "in practice this is rarely a problem", I agree, but prefer languages where the standard library doesn't have these arbitrary limitations. For example I might want to implement multivariate sparse polynomials as maps from exponent tuples to coefficients (the polynomial 3 x^2 y^3 z + 7 x y z^4 corresponds to the map {(2,3,1): 3, (1,1,4): 7}). That's straightforward in Python but awkward in Go (in Go I might be tempted to just set a maximum number of variables and use arrays or exponents as keys).
You can do this in Go too with structs.
In Python, you can use (1, 2), (3, 4, 5) and (6, 7, 8, 9, 10) all as keys in the same map.
package main
import "fmt"
func main() {
var m = map[interface{}]int {
[2]int{1, 2}: 12,
[3]int{1, 2, 4}: 123,
[4]int{1, 2, 3, 4}: 1234,
}
fmt.Println(m)
}Can I trademark that? ;)
EDIT: I understand that there are environment variables you can set, but how do I go about deploying my own proxy/checksum server? Is the source code behind https://sum.golang.org/ published somewhere?
> "...the mirror aims to cache content in order to avoid breaking builds for people that depend on your package, so this bad release may still be available in the mirror even if it is not available at the origin. The same situation applies if you delete your entire repository. We suggest creating a new version and encouraging people to use that one instead."
I disagree. The quicker that the personal data can be taken down, the less chance there is for someone to discover that personal data. I can't imagine any large company being comfortable with developers choosing technology that makes this harder to deal with than it should be.
(I am not at all suggesting this scenario is ok, simply asking how one would deal with it if the scenario should occur.)
On the other hand, proxy.golang.org re-hosts your module, and you can know that the proxy isn't serving you maliciously or differently than anyone else by verifying it via the public checksum database at sum.golang.org.
I don't believe proxy.golang.org, or any proxy for that matter, is required to host all of the modules and versions listed. It's a dumb caching proxy, and must deal with all of the real world complexities of hosting user content, including taking it down if deemed necessary. So if they need to take something down from the proxy they can, and the checksum database will only permanently retain the hash of the content that makes up the module.
Oh my god why?
Here is a link to Pike fucking it up 10 years ago.
https://github.com/golang/go/issues/151#issuecomment-6604830...
You would break so much code if you just threw that away.
Nice example of something else that should have been depreciated about 35 years ago.
... and honestly, is the ONLY place where I have ever seen octal been used.
> Go 1.13 now requires macOS 10.11 El Capitan or later; support for previous versions has been discontinued.
> Go 1.13 now requires FreeBSD 11.2 or later; support for previous versions has been discontinued. FreeBSD 12.0 or later requires a kernel with the COMPAT_FREEBSD11 option set (this is the default).
Sounds like they're still using syscalls directly on those platforms. Is Google ever going to admit that it's broken by design, because userspace apps on those platforms (outside of the base system, which is tightly coupled to the kernel) are not meant to be calling into the kernel directly directly, and should do everything via libc and other userspace libs that wrap syscalls?
On macOS and iOS, the runtime now uses libSystem.dylib
instead of calling the kernel directly. This should make
Go binaries more compatible with future versions of
macOS and iOS. The syscall package still makes direct
system calls; fixing this is planned for a
future release.
FreeBSD looks deprecated not because of syscall difficulties, but because missing functionality and upstream EOL.[1]
[0]: https://golang.org/doc/go1.11#runtime [1]: https://github.com/golang/go/issues/27619
That they don't run into issues with this constantly on BSD is not surprising - it didn't always break in macOS, either. Syscalls just don't change that much, usually, so you can get away with it more often than not. The problem is that when they do change eventually, every precompiled Go app that used one stops working.