typical182

I just submitted shfmt separately:

https://news.ycombinator.com/item?id=34754116

It’s nice, and it probably deserves to be more widely known...

shfmt is like gofmt, rustfmt, ..., but for shell programs.

Supports bash, posix, mksh, bats.

For open source projects in general, and the Go project in particular, I think it is easy to underestimate how much maintainer time is consumed discussing & considering whether a change should be made.

As I understand it, written in Go.

There's a mildly humorous "How do you know" exchange where someone on HN quizzes the very person most likely to know:

https://news.ycombinator.com/item?id=9920020

> the ban prevents me from discussing the matter further

Hi ddevault, FWIW, in May 2022 on that #44577 issue [0] you had opened, it looks like someone on the core Go team commented there [1] recommending that you email the golang-dev mailing list or email them directly.

Separately, it looks like in July 2022, in one of the issues tracking the new friendlier -reuse flag, there was a mention [2] of the #44577 issue you had opened. In the normal course, that would have triggered an automatic update on your #44577 issue... but I suspect because that #44577 issue had been locked by one of the community gardeners as "too heated", that automatic update didn't happen. (Edit: It looks like it was locked due to a series of rapid comments from people unrelated to Sourcehut, including about “scummy behavior”).

Of course, communication on large / sprawling open source projects is never quite perfect, but that's a little extra color...

[0] https://github.com/golang/go/issues/44577

[1] https://github.com/golang/go/issues/44577#issuecomment-11378...

[2] https://github.com/golang/go/issues/53644#issuecomment-11751...

Some people today raised concerns about disabling background refreshes (the temporary workaround originally suggested by the Go team) as having possibly unacceptable resulting performance for end users...

...but it sounds like disabling background refreshes would have strictly better end-user performance than what the Sourcehut team had been planning as described in their blog post today (GOPRIVATE and whatnot)?

Semi-related question: did Latacora or @tqbf ever open source their Go-based SAML IDP: https://twitter.com/tqbf/status/938501701526487040

(That tweet I think was a teaser saying it was coming. I subsequently looked for it a few times and never found it, but maybe plans changed, or maybe I just failed to find it).

Hi Steve, first, thanks for weighing in here!

I might have misunderstood your comment, but in my GP comment I was indeed attempting to contrast 'go install foo@latest' with 'cargo install foo', which both install binaries. (I wasn't talking about 'go get bar@latest', which now is just for updating or adding dependencies to a project).

Also, I'm contrasting what happens by default at the moment either binary install command is run. My understanding is Cargo's (non-default) 'cargo install --locked foo' behavior is similar to the default behavior of 'go install foo@latest'. In other words, the default behavior is fairly different between 'cargo install foo' (without --locked) vs. 'go install foo@latest'.

I edited my GP comment to simplify the example to use 'foo' in both cases. Maybe that helps?

In that same section, the blog describes the behavior of 'go install foo@latest' and contrasts it to how the default install "in some ecosystems bypass pinning."

That is also a difference in default behavior between Go and Cargo.

To install a 'foo' binary, 'go install foo@latest' gives you the latest version of foo, but the direct and indirect dependencies used are the versions listed in foo's go.mod or a dependency’s go.mod file (and not whatever the latest versions of those direct and indirect dependencies might be at the moment the install is invoked).

'cargo install foo' supports the optional --locked flag, but its not the default behavior: [1]

> By default, the Cargo.lock file that is included with the package will be ignored. This means that Cargo will recompute which versions of dependencies to use, possibly using newer versions that have been released since the package was published. The --locked flag can be used to force Cargo to use the packaged Cargo.lock file if it is available.

There are definitely pros and cons here, but to my knowledge it is not "just NPM" that is being contrasted in the blog.

Finally, I'm no world-class Rust expert, but I like using Cargo. I think Cargo is a fantastic tool that set the bar for package mangers, and it has done great things for the Rust community. But it is easier for communities to learn from each other with a base understanding of where & why different choices have been made, which is part of what is behind some of my comments around Go's behavior. ;-)

[1]: https://doc.rust-lang.org/cargo/commands/cargo-install.html

The default way Go handles go.mod is fairly different than the default way Cargo handles Cargo.lock files, including for example with libraries.

Also, when the blog says:

> Moreover, when a dependency is added with go get, its transitive dependencies are added at the version specified in the dependency’s go.mod file, not at their latest versions, thanks to Minimal version selection.

I believe that is significantly different than default Cargo behavior and for example default 'pub' behavior for Flutter (though I know approximately nothing about Flutter package management beyond a cursory search just now ;-)

To my knowledge, both Cargo and Flutter 'pub' prefer the most recent / highest allowed version by default when asked to solve constraints, whereas Go does not.

Cargo: [1]

> When multiple packages specify a dependency for a common package, the resolver attempts to ensure that they use the same version of that common package, as long as they are within a SemVer compatibility range. It also attempts to use the greatest version currently available within that compatibility range.

Flutter 'pub': [2]

> For each package in the graph, pub looks at everything that depends on it. It gathers together all of their version constraints and tries to simultaneously solve them. (Basically, it intersects their ranges.) Then it looks at the actual versions that have been released for that package and selects the best (most recent) one that meets all of those constraints.

[1]: https://doc.rust-lang.org/cargo/reference/resolver.html

[2]: https://dart.dev/tools/pub/versioning#constraint-solving

That's a nice & concise way to describe it.

> Okay but if I depend on A and A depends on C and has it pinned at 1.5, but I also depend on B and B has C pinned on 1.8, then I get 1.5?

No, you end up with the highest explicitly required version. So 1.8 in that scenario, if I followed. (Requiring 1.5 is declaring support for "1.5 or higher". Requiring 1.8 is declaring support for "1.8 or higher". 1.8 satisfies both of those requirements).

> if the constraints on C across various transitive deps are > 1.5 and > 1.8, and 1.9 or 1.10 exists, I probably want the last version of those.

By default, you get 1.8 (for reasons outlined upthread and in the blog post & related links), but you have the option of getting the latest version of C at any time of your choosing (e.g., 'go get C@latest', or 'go get -u ./...' to get latest versions of all dependencies, and so on).

Also, you are using the word "pin". The way it works is that the top-level module in a build has the option to force a particular version of any direct or indirect dependency, but intermediate modules in a build cannot. So as the author of the top-level module, you could force a version of C if you needed to, but for example your dependency B cannot "pin" C in your build.

I think you might have a small typo here:

> by choosing the lowest version specified in any package that depends on a given package

It picks the highest version specified in any of the requirements. (That's the minimal version that simultaneously satisfies each individual requirement, where each individual requirement is saying "I require vX.Y.Z or higher". So if A requires Foo v1.2.3 and B requires Foo v1.2.4, v1.2.4 is selected as the minimal version that satisfies both A and B, and that's true even if v1.2.5 exists).

That's not correct. You can unilaterally decide to update the version of E without waiting for anyone. Alternatively, if only C for example decides to update their required version of E, you would get that version of E if you updated your version of C (directly or indirectly), without needing to directly do anything with E yourself.

Slightly longer explanation of the mechanics here: https://news.ycombinator.com/item?id=30871730

The best complete explanation is probably here: https://research.swtch.com/vgo-principles

I suspect the primary purpose of the word "may" in that sentence is that you can choose to disable checking the hash against the Certificate Transparency style https://sum.golang.org. In other words, you can opt out. If you do, you fall back to your local go.sum file, which is more-or-less a "TOFU" security model: https://en.wikipedia.org/wiki/Trust_on_first_use

More on sum.golang.org: https://go.googlesource.com/proposal/+/master/design/25530-s...

Continuing that example -- in Go you end up with v1.0.1 of X by default even if v1.0.2 is the latest version of X.

That is a difference with many other package managers that can default to using the latest v1.0.2 of X (even if v1.0.2 was just published) when doing something like installing a command line tool. That default behavior is part of how people installing the 'aws-sdk' tool on a Saturday started immediately experiencing bad behavior due to the deliberate 'colors' npm package sabotage that happened that same Saturday.

In any event, it's certainly reasonable to debate pros and cons of different approaches. I'm mainly trying to clarify the actual behavior & differences.

FWIW, that's not what Go does. In your scenario, a Go binary ends up with a single copy of library X -- the 1.0.1 version. That's because library A is stating "I require at least v1.0.0 of X", and library B is stating "I require at least v1.0.1 of X". The minimal version that satisfies both of those requirements is v1.0.1, and that's what ends up in the binary.

That behavior is Go's "Minimal Version Selection" or "MVS". There are many longer descriptions out there, but a concise graphical description I saw recently and like is:

https://encore.dev/guide/go.mod

That's the default behavior, but a human can ask for other versions. For example, a consumer of A and B could do 'go get X@latest', or edit their own go.mod file to require X v1.2.3, or do 'go get -u ./...' to update all their direct and indirect dependencies, which would include X in this case, etc.

FWIW, there are two machine-formatted sections of go.mod -- the first for direct dependencies, the second section for indirect dependencies.

(That's as of Go 1.17. Previously, that information was communicated via machine-generated comments in a single section).

There are some subtleties here, but go.mod files are not lockfiles, including go.mod files don't follow a traditional constraint/lockfile split, which I think is part of the point in that snippet you quoted.

Another way they differ is when installing a top-level tool by default npm does not use the exact version from a library's lockfile to pick the selected version of another library as far I as understand, whereas Go does by default use the exact version required by a library's go.mod file in that scenario.

In other words, go.mod files play a bigger role for libraries than a traditional lockfile does by default for libraries in most other ecosystems.

Here's a good analysis on the contrast between go.mod and the default behavior of more traditional lockfiles (using the npm 'colors' incident as a motivating example):

https://research.swtch.com/npm-colors

That link also includes some comments on 'npm ci' and 'shrinkwrap' that I won't repeat here.

All that said, go.mod files do record precise dependency requirements and provide reproducible builds, so it's possible to draw some analogies between go.mod & lockfiles if you want. I just wouldn't say "go.mod files are lockfiles". ;-)

> tradeoffs against an alternative security model, where transitive dependencies are automatically updated to pick up security fixes.

One thing to keep in mind is that Go doesn't stop you from updating.

For example, its common to do 'go get -u ./...' or 'go get -u=patch ./...' from your project root to update all of your direct and indirect dependencies.

The built-in tooling & language server give you nudges, and if desired it can be automated via things like dependabot or otherwise.

In practice, it means it is often a slightly slower cadence for typical projects in the Go ecosystem compared to say the Node.js ecosystem, but the upgrades still happen. That slightly slower pace I think has worked out so far, and was a conscious choice[1]:

> Many developers recoil at the idea that adding the latest B would not automatically also add the latest C, but if C was just released, there's no guarantee it works in this build. The more conservative position is to avoid using it until the user asks. For comparison, the Go 1.9 go command does not automatically start using Go 1.10 the day Go 1.10 is released. Instead, users are expected to update on their own schedule, so that they can control when they take on the risk of things breaking.

[1] https://go.googlesource.com/proposal/+/master/design/24301-v...

There is also a new set of FAQs on generics:

https://go.dev/doc/faq#Type_Parameters

I think they meant “type information is not available at parse time” (rather than “compile time”).

There is a longer and more precise description of the problem as part of this FAQ of the generics proposal:

https://go.googlesource.com/proposal/+/refs/heads/master/des...

…which includes an example and the statement “It is a key design decision of Go that parsing be possible without type information”.

FWIW, the best intro to CUE I’ve found so far is:

https://bitfieldconsulting.com/golang/cuelang-exciting

…which does a nice job of starting from the problem, and building up from "how could we improve JSON for use as a config language" to arriving at CUE.

(Those types of explanations happen to resonate with me, including understanding the "why" first at least for me helps the mechanics and details later make more sense and feel more intuitive…)

This is a great list of influences on the design (from the article comments where the prototype author Sam Gross responded to someone wishing for more cross pollination across language communities):

—————

"… but I'll give a few more examples specific to this project of ideas (or code) taken from other communities:

- Biased reference counting (originally implemented for Swift)

- mimalloc (originally developed for Koka and Lean)

- The design of the internal locks is taken from WebKit (https://webkit.org/blog/6161/locking-in-webkit/)

- The collection thread-safety adapts some code from FreeBSD (https://github.com/colesbury/nogil/blob/nogil/Python/qsbr.c)

- The interpreter took ideas from LuaJIT and V8's ignition interpreter (the register-accumulator model from ignition, fast function calls and other perf ideas from LuaJIT)

- The stop-the-world implementation is influenced by Go's design (https://github.com/golang/go/blob/fad4a16fd43f6a72b6917eff65... )"

Very nice write up.

Go’s focus on simplicity means that there is only a single parameter, SetGCPercent, which controls how much larger the heap is than the live objects within it.

FWIW, there is a new proposal from a member of the core Go team to add a second GC knob in the form of a soft limit on total memory:

https://github.com/golang/proposal/blob/master/design/48409-...

It includes some provisions to make sure that the application can keep making progress and avoid death spirals (part of the reason why it is a "soft" limit), and also includes some new GC-related telemetry.

From the blog write up, a second GC knob with a soft limit might have only been a minor help here, with the bigger wins coming from the code changes they described in the blog.

There is also a WIP Go proposal for adding a set to the standard library using the upcoming generics:

https://github.com/golang/go/discussions/47331

I think though it is recommending against strncpy as well…

linters and code reviewers commonly recommend alternatives such as strncpy (difficult to use correctly; mismatched semantics) […] Besides their individual shortcomings, these answers are incorrect. strcpy and friends are, at best, incredibly niche, and the correct replacement is memcpy.

People can have different definitions and still communicate usefully, and I think there is not 100% agreement on the exact boundaries between the two.

That said, for me: they are distinct but related, and that distinction is useful.

For example, Hypothesis[1] is a popular property testing framework. The authors have more recently created HypoFuzz[2], which includes this sentence in the introduction:

“HypoFuzz runs your property-based test suite, using cutting-edge fuzzing techniques and coverage instrumentation to find even the rarest inputs which trigger an error.”

Being able to talk about fuzzing and property testing as distinct things seems useful — saying something like “We added fuzzing techniques to our property testing framework” is more meaningful than “We added property testing techniques to our property testing framework” ;-)

My personal hope is there will be more convergence, and work to add convenient first-class fuzzing support in a popular language like Go will hopefully help move the primary use case for fuzzing to be about correctness, with security moving to an important but secondary use case.

[1] https://hypothesis.works

[2] https://hypofuzz.com

There is a good LWN article that gives a useful overview of the current proposal as well as briefly hits on some of the history:

https://lwn.net/Articles/829242/

Based on reading through various posts from Discord people after that blog was posted, my understanding was there was a material gap in time between the last time they tried in earnest to solve the latency issue with Go (with Go 1.10, as in your quote) vs. when they later did the re-write in Rust.

It seems during that gap in time, the Go runtime team happened to solve their problem, which was GA in Go 1.12 in Feb 2019, which was prior to them doing the re-write in Rust, at least as far as I was able to follow.

One imprecise quote on timing of the re-write: [1]

This blog post perhaps is a bit "after the fact" we had made the switch over mid 2019

It's not crazy for someone to put aside a problem for a while, and then upon returning to the problem some time later decide to go a different route without re-exploring prior solutions, if that is what happened.

All that said, I might have misunderstood the timing, and I'm trying to avoid going back over all the various forums they commented in to find a better quote on timing ;-)

[1] https://news.ycombinator.com/item?id=22239707