typical182

> Do you know if that was the case when the decision to switch was made?

It’s a good question.

The timing is a bit confusing because the blog was apparently published a bit after the fact, but I think I saw that they said they made the decision mid 2019.

Go 1.12 seemed to address their latency issue, which was available as GA in Feb 2019.

That’s based on some retroactive benchmarks on a set of old Go versions starting with 1.9 and targeting what they described as the problem and symptoms.

Things seemed to line up, but I can’t be sure.

I don’t know if Go 1.11 would have addressed their issue.

(In general, Go GC tail latencies including for large heaps have improved a bunch since the last version they said they tried, which I think was Go 1.10).

In any event, they saw a problem, and made a rationale decision for multiple rationale reasons.

True, and it was an interesting case study.

I had looked at the Discord GC blog issue [1] fairly carefully, and it certainly seemed that specific GC latency issue they hit and blogged about would have been solved by upgrading to a modern Go version....

Also, after publishing that blog, they also said they had wanted to try Rust for other reasons:

wanted to try out rust as well for services like this, due to adoption elsewhere in the company. Also, after upgrading between 4 golang versions on this service and noticing it didn't materially change performance, we decided to just spend our time on the rewrite (for fun, and latency) and to get a head start into the asynchronous rust ecosystem. [2]

And nothing wrong with that decision from my point of view (Rust is great!), but I believe they could have solved it without dropping Go for that particular service that they blogged about, at least as far as I was able to understand.

[1] https://blog.discord.com/why-discord-is-switching-from-go-to...

[2] https://news.ycombinator.com/item?id=22239707

FWIW, “comparable” and “ordered” are well defined terms in the Go language specification:

https://golang.org/ref/spec#Comparison_operators

In terms of where it is used, here is a related tweet by Filippo Valsorda:

https://twitter.com/FiloSottile/status/1270115515378384897

For scale, this GnuTLS vulnerability is considerably worse than Heartbleed. If you use Linux distributions with GNU tendencies, you might want to check your dependency trees.

(He is a cryptographer on the Go project, who also happened to win the CloudFlare Heartbleed Challenge).

As he says, that thread is a good starting point for potentially vulnerable uses.

Some sample quotes from that thread:

——

The good news is that this is a server-side issue

——

For obvious reasons, systemd ships a custom http server with client auth via GnuTLS.

——

On Fedora "dnf repoquery --whatrequires gnutls" lists: Samba NetworkManager pacemaker qemu / libvirt wget rdesktop tigervnc gnupg

Apache mod_gnutls ...

It looks like it has been released now.

If you haven't already, it sounds like it would probably be worth your time to open an issue. If things like workspace-scoped references and renaming aren't working for you when they are working for other people, perhaps you have a non-standard setting, or perhaps there is a need to adjust how you open vscode, or perhaps you are hitting bug(s). My experience has been the people working on gopls are very responsive.

Are you using an old version of gopls?

From the v0.3.0 release notes[1]:

Workspace-scoped references, rename, and go to implementation. These features use your workspace root as the search scope, so behavior will vary based on the directory you open in your editor

[1] https://github.com/golang/go/issues/33030#issuecomment-51015...

> Godoc is broken because of it

That was resolved a while ago. Godoc supports modules[1]:

For those who are following this issue because you are waiting for module support in the godoc command (golang.org/x/tools/cmd/godoc), please note that it has been implemented in the latest version. See issue #33655.

[1] https://github.com/golang/go/issues/26827#issuecomment-55194...

As far as I understand, I don't think it is officially released as of this moment. (It is getting close, though).

For example, as of now, the standard release note url for 1.14 currently 404s: https://golang.org/doc/go1.14

Do you have any load tests or synthetic benchmarks that are still capable of producing this?

It would be interesting to see what a more modern Go would do given there have been a bunch of tail latency GC improvements since your older 1.9 Go version... and in an ideal world, it would be nice to file an issue on the tracker if you were still seeing this.

(Maybe that ends up later helping another one of your Go services, or maybe it just helps the community, or maybe it’s a topic for another interesting blog...).

In any event, thanks for taking the time to write up and share this one.

Maybe that is what they hit... but it seems there is a pretty healthy chance they could have resolved this by upgrading to a more modern runtime.

Go 1.9 is fairly old (1.14 is about to pop out), and there have been large improvements on tail latency for the Go GC over that period.

One of the Go 1. 12 improvements in particular seems to at least symptomatically line up with what they described, at least at the level of detail covered in the blog post:

https://golang.org/doc/go1.12#runtime

“Go 1.12 significantly improves the performance of sweeping when a large fraction of the heap remains live.“

Where does it say that?

It says things like:

“We were not creating a lot of garbage.”

... but that statement there doesn’t say anything about the heap size, including the size and count of live objects (i.e., not garbage).

It also says:

“There are millions of Users in each cache. There are tens of millions of Read States in each cache.”

Large is often in the eye of the beholder, but I missed it if it said anything specifically about not having a large heap size.

Probably a dumb question, but how do you envision connecting something like a networked printer? Front it with a cheap device?

For context, this is a post from bradfitz, the creator of LiveJournal, memcached, OpenID, been on the core Go team for last 10 years or so.

There was a recent thread on him leaving Google: https://news.ycombinator.com/item?id=22161383

As far as I have followed, I think it is more that while a fair amount of work had gone into prior designs for generics in Go, the general feeling of the core Go team was the prior designs did mesh well with the rest of the language, whereas the more recent design seems like it could work.

From this blog post[1] from last year from the core Go team:

We’ve been thinking about generics since work on Go began, and we wrote and rejected our first concrete design in 2010. We wrote and rejected three more designs by the end of 2013. Four abandoned experiments, but not failed experiments, We learned from them, like we learned from check and try. Each time, we learned that the path to Go 2 is not in that exact direction, and we noticed other directions that might be interesting to explore. But by 2013 we had decided that we needed to focus on other concerns, so we put the entire topic aside for a few years.

Last year we started exploring and experimenting again, and we presented a new design, based on the idea of a contract, at Gophercon last summer. We’ve continued to experiment and simplify, and we’ve been working with programming language theory experts to understand the design better.

Overall, I am hopeful that we’re headed in a good direction

[1] https://blog.golang.org/experiment

For Go on Darwin:

libSystem is now used when making syscalls on Darwin, ensuring forward-compatibility with future versions of macOS and iOS.

From https://golang.org/doc/go1.12#darwin

Probably worth filing an issue, if you haven’t already?

Some chance you might be hitting some pathological behavior that could be fixed or tweaked.

> The mainline kernel, as released by the Linux core team is up to date with security. Hold to account people that decided to skip patches as a matter of course

To me, the particulars of this exact case are not as interesting as the fact that the entire Linux patching and backporting of security issues seems _very_ fragile, with things frequently getting "lost" for mundane reasons, and a key part of the "why" it is so fragile is due to many of the core Linux development processes.

This particular CVE is apparently one small example that happened to catch some headlines out of _thousands_ of similar problems.

That talk linked above by Dmitry Vyukov is worthwhile for getting a sense of the magnitude of the problem.

To me, the biggest part of this story is:

1. Over two years ago, this was apparently detected automatically by the syzkaller kernel fuzzer, and automatically reported on its public mailing list. [1]

2. Over a year and a half ago, it was apparently fixed in the upstream kernel. [2]

3. It was apparently never merged back to various "stable" kernels, leading to the recent CVE. [3]

So you might read that and think "Ok, probably a rare mistake"...

...but instead:

4. This is apparently a _super_ common sequence of events, with kernel vulnerabilities getting lost in the shuffle, or otherwise not backported to "stable" kernels for a variety of reasons like the patch no cleanly longer applies.

Dmitry Vyukov (original author of syzkaller fuzzer that found this 2 years ago) gave a very interesting talk on how frequently this happens a couple weeks ago at the Linux Maintainer's Summit, along with some discussion of how to change kernel dev processes to try to dramatically improve things:

slides: https://linuxplumbersconf.org/event/4/contributions/554/atta...

video: https://youtu.be/a2Nv-KJyqPk?t=5239

---

[1] https://twitter.com/dvyukov/status/1180195777680986113

[2] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux...

[3] https://mobile.twitter.com/grsecurity/status/118005953923380...

Yes, fixed upstream near start of 2018, but apparently never merged back to various “stable” kernels.

See for example:

https://mobile.twitter.com/grsecurity/status/118005953923380...

Well, one thing is it was apparently already publicly reported over 2 years ago by syzkaller:

https://twitter.com/dvyukov/status/1180195777680986113

As far as I understand, performance is not a valid concern.

For example, from dvyukov's KTSAN wiki [1]:

  Given a sufficiently expressive atomic API and a good
  implementation, you pay only for what you really need (if
  you pay just a bit less, generated code becomes incorrect).
  So performance is not an argument here. 

[1] https://github.com/google/ktsan/wiki/READ_ONCE-and-WRITE_ONC...

You can also pick your own proxy primary and fallbacks.

For example, this should in theory work to make the google-run proxy your fourth choice, with three other proxies attempted before the google-run proxy, and with direct (no proxy) access as the fifth and final fallback:

  go env -w GOPROXY=gocenter.io,goproxy.io,goproxy.cn,proxy.golang.org,direct

There is some geographic diversity in that example as well -- I think two mirrors in that list are primarily run in China.

> This "feature" should have an option to disable it.. at the very least.

The Go 1.13 Release Notes lead off with documentation links for how to disable and otherwise configure this:

"See https://proxy.golang.org/privacy for privacy information about these services and the go command documentation[1] for configuration details including how to disable the use of these servers or use different ones. If you depend on non-public modules, see the documentation for configuring your environment[2]."

[1] https://golang.org/cmd/go/#hdr-Module_downloading_and_verifi...

[2] https://golang.org/cmd/go/#hdr-Module_configuration_for_non_...

I've used Rust a bit, but not enough to know the details of how cargo and crates.io approach managing collected information.

There seems to be an open issue and WIP PR with more details:

Issue: https://github.com/rust-lang/crates.io/issues/955

PR: https://github.com/rust-lang/www.rust-lang.org/pull/919/file...

The cargo manifest also supports things like:

  The publish field (optional)

  The publish field can be used to prevent a package from 
  being published to a package registry (like crates.io) by mistake, 
  for instance to keep a package private in a company.

  [package]
  # ...
  publish = false

But I would be curious if someone more knowledgeable than myself would be interested in giving a quick summary of the approach?

I think the concern was that it would be too expensive, but then later they were able to figure out a way to do it.

Some more details: https://github.com/golang/go/issues/30116