Go 1.20 released
go.dev
go.dev
The math/rand package now automatically seeds the global random number
generator (used by top-level functions like Float64 and Int) with a random
value, and the top-level Seed function has been deprecated. Programs that
need a reproducible sequence of random numbers should prefer to allocate
their own random source, using rand.New(rand.NewSource(seed)).
It's the kind of "bug" that doesn't manifest until your data is thoroughly boned.
How do I make sure I'm passing a random seed to my RNG?
(I must be missing something here)
math/rand provides global functions using a global RNG, as well as the ability to instantiate your own RNG and call functions (methods) on that RNG.
Previously, the global functions all used a seed of 1, which made them generate identical sequences. Now, they use random seed, which makes them less-easily predictable (though still predictable).
There is no change to the self-instantiated RNGs.
> How do I make sure I'm passing a random seed to my RNG?
With the change, using the global functions in Go 1.20+ is the same as instantiating your own RNG with a random seed.
https://en.m.wikipedia.org/wiki/Hardware_random_number_gener...
Maybe we're in the first batch of simulations, and the tester came along and asks why they're all identical. The cosmic coder then realises that they forgot to call the function to seed the prng.
*BSD, newer macOS, and GNU libc: int getentropy(void *buffer, size_t length);
Linux 3.19+: ssize_t getrandom(void *buf, size_t buflen, unsigned int flags);
iOS and macOS 10.7+: int SecRandomCopyBytes(SecRandomRef, size_t, uint8_t *);
Fuchsia: void zx_cprng_draw(void* buffer, size_t buffer_size);
Any *nix: Read from "/dev/random" (or "/dev/urandom" under the assumption that your system has already enough entropy).
Windows: There are different functions/libraries depending on the Windows version and some of them are a complicated multi-step mess.
Some time ago I wrote a C library that abstracts that away just for fun: https://github.com/panzi/portable_get_random*
This change only affects the global RNG. Creating a new RNG has always required setting a seed
Amusingly, I submitted a patch for this some years ago, and it was rejected on the grounds that callers might depend on the specific sequence of numbers outputted by the global RNG using the default seed, which could break the compatibility promise.
Now that that's been deemed a non-issue, I might resubmit.
Given randomness requires a seed and UNIX's ISO standard defines the default seed as 0, it's rational from a comp sci perspective to expect to supply a seed to produce a random number.
However, due to how ergonomic and safe high level languages are today, you don't need to be a computer scientist to make a highly available, beautiful and robust application. As such, the inhabitants of the application development world are considerably less concerned with such details and want, when they call a thing, it gives an intuitive result.
It's important that tool makers consider their users when creating APIs. Supplying a seed to a function called "getRandomNumber" might be obvious to a C programmer, but not obvious to a JavaScript programmer looking to use Go to improve the performance of their back end service.
Agreed, but worth nothing that this does not make it cryptographically secure. The output can still be predicted. For cryptographic security, you need crypto/rand: https://pkg.go.dev/crypto/rand
In general, my advice is to use the cryptographically secure RNG unless you specifically know you need reproducibility (e.g. scientific simulation, or map generation in video games). With non-secure RNGs, it's very easy to accidentally expose yourself to problems without realizing it.
(Also, FYI, the default seed was 1, not 0).
Sure, and that's even been in the documentation since before Go 1.0, yet people still make mistakes with it in practice. I've found it worth making the point explicit. Particularly in a case like this, where a casual reader might not notice the distinction between "non-reproducible" and "secure".
The change here is specific to the global RNG, which users often used without explicitly seeding - e.g. calling rand.Int() without first calling the now-deprecated rand.Seed(int64).
The distinction is obvious to people who have domain expertise here, but I've found many people make mistakes with it in practice, because it's easy to do.
As examples of secure cryptographic RNGs that can be seeded:
- Fortuna (has a seedable generator without the entropy pool parts)
- HMAC-SHA256 DRBG
- ChaCha20 and Salsa20
- AES-CTR
- Keccak
If you're seeding an RNG, you're almost certainly working with a userland RNG, which is, almost always, a mistake.
If you, as a developer who doesn't know much about random numbers or cryptography, think you need a random value, and you don't know if it needs to be cryptographically secure or not, you may as well just use a cryptographic RNG interface (unless you're using so much that performance or entropy becomes an issue.)
I think in most cases, it's pretty benign if you use cryptographic randomness even when it's not necessary. But, if you use math/rand when you wanted cryptographic randomness, to generate IDs or some such, that would be a much worse outcome.
Maybe it's bad for someone to use an RNG without understanding this well enough, though, and they should instead use a higher level abstraction if at all possible. But I can get behind the general idea.
This is more or less what I was getting at. The main two downsides to using crypto/rand are:
- ergonomics (crypto/rand is a less user-friendly interface than math/rand)
- concurrent performance
The first one can be easily solved with a wrapper[0]. The second is particularly relevant here, because the main distinguishing feature of the global RNG in math/rand is that it is safe for concurrent use, whereas user-instantiated RNGs in math/rand are not. The big downside to this is that it's very easy to end up with performance issues due to mutex contention when multiple packages all use the global RNG (which is common in practice).
I actually submitted a CL (patch) to fix the mutex contention issue in the global RNG about five years ago, but it was rejected on the grounds that callers might depend on the specific sequence of numbers with the default seed, which would arguably break the compatibility promise. That apparently is no longer a concern (this change breaks the same thing, and the notes in the CL justify it), so I might resubmit it now.
crypto/rand is a little less performant in the single-threaded case, but not much - I think it'd be rare for that to be the bottleneck in real-life workloads at scale. The mutex, on the other hand, is a common bottleneck - I've run into this multiple times in multiple different codebases, one of which is what motivated the aforementioned CL.
So I generally advise people to use crypto/rand unless they are certain they need reproducibility, because the potential downside of accidentally using a non-secure RNG when you actually need one is quite high[1], but the downside of using a threadsafe cryptographically-secure one when you needed a threadsafe non-secure one is quite low: you're already taking much of the performance hit because of the mandated mutex, so the number of use cases that actually require the global RNG is quite small.
[0] e.g. https://pkg.go.dev/github.com/andrew-d/csmrand
[1] there are a number of places where RNGs end up being used that don't obviously result in exploits but nevertheless result in exploits in practice. For the average developer, it's easiest just to avoid that quagmire altogether, rather than try to reason about the potential adversaries (and potentially get that wrong).
https://cs.opensource.google/go/go/+/refs/tags/go1.20:src/ma...
https://cs.opensource.google/go/go/+/release-branch.go1.20:s...
https://cs.opensource.google/go/go/+/release-branch.go1.20:s...
https://cs.opensource.google/go/go/+/master:src/runtime/asm_...
I feel strangely vindicated seeing this change.
EDIT: Okay, C and a bunch of other languages I don't use. Keep the roasts coming, guys. :D
(BTW, I'm deliberately avoiding making any distinction between so-called "true random" and PRNG above, because the difference isn't actually meaningful here!)
Not a Go programmer though but that was my understanding.
Personally I've wrote a whole bunch of unit tests in the past that relies on that repeatability property "math/rand" has over "crypt/rand"
I don't really see an issue with either default, so long as it's documented and you can use a fixed seed if you'd like. I personally like needing to set a random seed explicitly. But then again, I learned about using random number generators a long time ago when this was always required, so what I think is obvious is probably a surprise to many (as shown by this stdlib change).
The only downside about this new change was that in the old way, if you needed a cryptographically secure random number, you had to explicitly call those functions from the stdlib. The choice should have been deliberate, but people don't like to read documentation...
POSIX (or at least ISO C) defines rand as being seeded with 0 by default.
Stay safe everyone.
But of course, rather than make people study up, let's just dumb shit down
I wish I had found this little throwback (whether accidental or not) before it got fixed - though I definitely agree with the change.
F
Projects who have been seeding the random generator like they should suddenly think “oh I don’t need to do that anymore” and get rid of their manual seeding.
Then a compromised or rogue library decides to seed the global generator themselves to a hard coded value in an `init()`, thus meaning merely importing the library re-statics the seed.
It would look pretty innocuous and non-obvious in code AND be potentially pretty difficult to notice it happening in a lot of use cases. For bonus points/making it slightly harder to detect points they could even have a random set of seeds they use.
The right answer, probably just generally anyway, is to never use the global generator, and always create your own instance. Global state is a danger once again
I don't think people would use the "global" random functions for deterministic randomness. At least I hope they didn't...
The library contains generics based helper functions for working with maps.
https://github.com/golang/go/issues/57436#issuecomment-14125...
I've been using this library for ages now...
https://github.com/cornelk/hashmap
Always entertains me to see developers write all the lock/unlock code when they could just use that.
Yes, https://pkg.go.dev/golang.org/x/exp/slices gives some of that, but likely the rest should be added as well.
This is absolutely awesome! I'm looking forward to trying this on some server binaries!
go test -coverpkg=./... -c . -o main.test
I’m not exactly sure what the change in Go 1.20 is about, maybe it’s easier somehow, I’ll have to try it.
> The vet tool now reports use of the time format 2006-02-01 (yyyy-dd-mm) with Time.Format and time.Parse. This format does not appear in common date standards, but is frequently used by mistake when attempting to use the ISO 8601 date format (yyyy-mm-dd).
It's frequently used by mistake because Go doesn't allow datetime layouts to use the standard YYYY,MM,DD,HH,MM,etc which they ironically used for clarity in their release notes.
I don't understand why Go still forces datetime formats to be specified using "magic numbers" from some time in 2006.
01/02 03:04:05PM '06 -0700
So that's why it's in 2006, since you asked.
Just have to remember that the year comes before the time zone but after the seconds :clown:
No idea where that came from. Probably the time format from the punchcard machine at bell labs.
If it was iso 8601 based, it would almost make sense.
This is interesting. I wonder what Go 1.21 will depend on that requires at least Windows 10?
Nothing concrete as it seems. It means that new releases are no longer tested with the old versions of Windows on their builders, and if you open a bug report about a problem with an unsupported version of Windows, nobody will care.
I imagine there's some enterprise customer somewhere on an old version of Windows that will throw money at vendors to make a similar effort.
Aside from that, there are a broad swath of flags to LoadLibraryEx that are only supported on earlier platforms with a KB [1] from over a decade ago installed. My suspicion is that Go has decided that requiring a security KB (while good hygiene) isn't a supportable situation.
[1] https://support.microsoft.com/en-us/topic/microsoft-security...
This is interesting because in certain cases it can be a performance hit when comparing structs which have been declared in alignment order to save memory. Simple example:
type t struct {
a int64
b int32
}
t1 := t{a: 1, b: 2}
t2 := t{a: 1, b: 3}
same := t1 == t2
Of course this is a trivial example, in real-world cases structs can have many more fields with much larger contents. The point is that the optimal ordering for alignment, and the optimal ordering for comparison, seem to be different.
This is more of a constraint if the struct contains a comparison that can panic. The panic must happen in order or not at all depending how the fields are listed.
type t struct {
a int64
b any
}
Overall, I think adding generics to Go was a big mistake. It brings the following drawbacks:
- Slower compile times, even if generics aren't used. This slows down development pace in Go.
- The reduced code readability if generics are used. This slows down development pace in Go.
- The increased complexity of Go compiler. This slows down Go compiler development and increases chances for bugs.
- Very low adoption of generics in practice, since they aren't useful in most Go code bases. The generics are actively used in some freaky packages only after the year since they were released in Go1.18.
The only useful thing from Go generics is a syntactic sugar, which allows replacing `interface{}` with `any`.
It pops up on /r/golang sometimes. I don’t think it gets taken super seriously but there’s usually at least someone bringing it up.
Go showed that useful software could be written without user-level generics. I don't think any other language today would dare to do that. In fact most languages seem to be converging into the same thing.
That's not the same as it being a good idea
Of course, there are other brilliant features in Go ecosystem, which simplify writing and maintaining non-trivial codebases in Go - tooling, standard library, fast compile times, statically linked binaries, etc.
Is it a surprise that the low uptake is there with a discouragement like that?
- Simple syntax
- Fast compile times
- Great tooling (go fmt, go vet, go too pprof, go tool cover, race detector, ect.)
- Useful standard library
Generics do not improve any of these features :( They complicate syntax, they slow down compile times and they complicate internals of go compiler and tools.
- Performance optimizations (profile-guided optimization, GC optimizations, etc.)
- Reducing compile times
- Reducing binary sizes (improved linker, which can throw away unused code more aggressively)
(1) Make it work, (2) make it right, (3) make it fast
They did 1 and 2 in the last two releases, and in Go 1.20 they did (3). So what's left to complain about?
Are there more bugs in the compiler? Is readability reduced, and having an effect on pace? Especially if adoption is so low to begin with? Is adoption actually so low, or just rising?
The irony is that vocal minority still do not use Go, since they have other excuses now - "bad error handling", "missing functional features", etc. But if these harmful features will be implemented in Go, the vocal minority will find another reason why they do not use Go.
Speak for yourself: I prefer generics to the mess if copy-pasta or generated stuff that one had to use before.
[1] https://github.com/VictoriaMetrics/VictoriaMetrics/blob/mast...
I effectively used it in this little experimental CLI library: https://github.com/cpuguy83/go-cli/blob/main/command.go
It's pretty simple, but the nice thing is it can use any flag library you want (stdlib flag package, pflag, whatever).
In fact, the generic-free implementation based on the FlagSet interface is more flexible, since it allows storing multiple different FlagSet implementations in the same Cmd.
The main point is that generics come in handy when building libraries so that you aren't forcing callers of your library into specific types or loosing some type safety.
What priority would you assign a bug that migrates the existing, working code that other people are doing god knows what with downstream? What would the theoretical benefits "better code" be to your downstream, and how do they weigh against the cost of "I ran go mod tidy, and our build broke?"
I can think of a use where it would lead to better code, and that's in k8s custom resources, where each resource type also has an associated list type that people create with code generation. It'd be much neater for k8s lists to be
type List[T runtime.Object] struct {
...
Items []T
}
It is.
> Additionally, it may slow down the resulting code.
FUD.
- Go compiler may fail to inline the callback passed to the maps.DeleteFunc(). This will result into an additional overhead for callback calls per each item in the map.
- Go compiler may move some variables inside the callback from stack to a heap. This will result in an additional memory allocations comparing to a simple loop, leading to an additional load on garbage collector.
The strings.Split() implementation is non-trivial because of performance optimizations.
The bytes.Equal() is actually written in highly tuned and optimized assembly in order to achieve high performance for inputs of various lengths.
Now compare this to trivial implementations behind generic-based functions for maps. And do not forget that these implementations may hurt performance because of excess memory allocations in Keys() and Values() functions or because the compiler may fail inlining the callback passed to DeleteFunc().
Though part of why they’re optimized and in the stdlib in the first place is because they’re such common patterns. So without them people would end up writing trivial, unperformant, custom versions. So now that more routines can be moved into the stdlib, they can benefit from optimization later.
(I’m not sure how much the maps routines specifically can be optimized, but stdlib routines routines can generally be more aggressive with unsafe or asm or being coupled to the runtime and its quirks, like bytes.Clone, strings.Builder, etc.)
And there are still plenty of ubiquitous patterns that have been worth including in the stdlib even if they’re usually just simple loops that aren’t very optimizable. Like strings.Index is an easy loop to write, but it comes up so often. Or strings.Cut is basically just an if-statement. But it makes code clearer about its intentions; and optimizations to these down the road benefit everyone.
It’s also true that maps.Keys and maps.Values allocate slices, and that you could avoid this with a loop, but strings.Split, bytes.Split, regexp.FindAll, os.ReadDir return slices and are still worthwhile as opposed to specialized iterators for each one. As with any code, you’re conscious of memory allocations where it counts, and optimize as needed.
In fact, now that generics make it possible, the Go team has discussed using iterators (https://github.com/golang/go/discussions/54245), which would benefit strings.Split even further in addition to all the other slice-returning functions.
So generally you have three options for those slice-returning functions:
- Custom inline loop for some of them. More verbose, will probably be naive and not benefit from stdlib optimizations. - Return a slice and iterate over it with a for loop. Creates allocations that could probably be avoided. - Create a customized iterator for that type. Unfortunately, you can’t really use an ordinary for loop, and extra custom iterators for each type. - Use generic iterators to benefit from the optimized functions and also avoid allocation overhead.
So part of the motivation is that now with generics there’s a variety of further optimizations available even to old functions like strings.Split and regexp.FindAll, in addition to opening up common patterns and optimizations for maps/slices/etc. to be included in the stdlib.
A few remarks:
> Like strings.Index is an easy loop to write, but it comes up so often
Actually, strings.Index() is very non-trivial function partially written in assembly in order to achieve high performance [1]. This function is used in Go projects *much more frequently* than functions from the golang.org/x/exp/maps package.
> strings.Cut is basically just an if-statement
No, strings.Cut() has non-trivial code when comparing to a trivial loop for map copy or for map delete [2].
> It’s also true that maps.Keys and maps.Values allocate slices, and that you could avoid this with a loop, but strings.Split, bytes.Split, regexp.FindAll, os.ReadDir return slices and are still worthwhile as opposed to specialized iterators for each one.
The *key* difference between maps.{Key,Value} and the mentioned functions from the standard library is that it is trivial to write the `for k, v := range m` instead of maps.{Key,Value} and avoid memory allocations, while it isn't trivial to write the corresponding code without memory allocations, which substitutes strings.Split() or other mentioned functions from the standard library.
[1] https://github.com/golang/go/blob/86c4b0a6ec70b07ab49d3813a5...
[2] https://github.com/golang/go/blob/86c4b0a6ec70b07ab49d3813a5...
This optimization then could speed up other similar interface-based algorithms.
Of course I want sync.Map to use generics instead of interface{}. How could I not? And it’s less complex-looking than type-asserting everywhere.
The container/heap is more useful, but it could benefit more from adding an optimization for inlining interface method calls when Go compiler knows the underlying implementation behind the interface.
The golang.org/x/exp/maps is useless and may be harmful [1].
The golang.org/x/exp/slices is mostly useless, except of Sort*() functions. But it would be better to apply the optimization mentioned above to standard sort.* functions instead of forcing users to switch to different Sort*() implementations in other packages.
They aren't widely used because the ergonomics suck, because they aren't generic yet.
This is exactly what generics do. With e.g. a heap.Heap[uint32] the compiler knows the implementation and there’s no interface method call overhead.
In order for the compiler to do this optimization, it has to know that you don’t e.g. pass a *heap.Heap[uint32] to a function expecting *heap.Heap[uint64], so the type system is what allows it to optimize.
And on top of that, now the user also gets assurance at compile time that heap.Heap[uint32].Pop returns a uint32, preventing bugs from type confusion and also so you don’t have to add type assertions everywhere you use the heap.
So now heap, sort, etc. can benefit from this improved performance; users don’t have to write wrapper types and interface implementations just so their type can be sorted; and bugs are prevented at compile time.
For [1] I posted a reply. It’s true that there are overheads with some slice-returning routines but I explained how in the reply how I viewed the tradeoffs.
type customStruct struct { ... }
func (cs *customStruct) Less(i, j int) bool { ... }
func (cs *customStruct) Swap(i, j int) { ... }
func (cs *customStruct) Len() int { ... }
func sortMyCustomStruct(cs *customStruct) {
sort.Sort(cs)
}
In fact, it can improve readability and maintainability in some cases instead of having multiple structs copy pasted everywhere.
While true that generics are not necessarily going to make programs harder to read, it's also not very interesting to talk about theory.
The question which is answerable is, "on average, were go programs easier to read before the introduction of generics". The interesting point is what actually happens, not what could.
I have not seen a lot of comments that complained about the slower compile times. In my own experience it didn't really had an impact. But I agree the compiler should not become slower over time, so I appreciate the effort of the Go team to bring the compiler speed back.
I don't think that code readability is so much impacted. The square brackets work well. I find the angle brackets from C++ harder to read and there is the problem that >> is a token and cannot be used for two closing template angle brackets.
The increased complexity of the compiler is an issue, but cannot be avoided if you want to support Generics. But they took the time to make it right and as I stated it works for me.
I don't think that there is low adaption. Using type parameters visibly in a public API, breaks the API, which is the reason there are not a lot of uses in the standard library and with popular packages now. But this will change when maps and slices will be integrated in the standard library, which provide completely new APIs. Yesterday I found a library writing and reading parquet files, which used it quite extensively. But since I simply checked what libraries existed to assess how well the file format is supported, I cannot say much whether the use of type parameters by the library is useful.
This was fixed years ago in C++11.
The chevrons also confuse all my editors, because they can't figure out whether `<` is an opening bracket pair, or if its just a less-than operator. I'm a bit miffed that Cpp2 doesn't try to replace the chevrons with something else, like D did. I'm also miffed that Rust uses the chevrons too.
Does anyone still actively use Usenet?
Last time I checked Usenet, it seemed like many of the users were pirates. The pirate releases would then get reuploaded to bittorrent sites.
Wait, isn't that the whole point of the constraint in the first place, to keep you from using it with things that aren't comparable? Wouldn't it make more sense to have the constraint be a requirement in the interface itself, so that you can't create an interface value from a type that isn't comparable?
Do you read the docs of every basic feature you use? When you call a rand function, you usually expect it to be seeded. In any language, I'll only read the docs of a rand function to know how it handles bounds. You still can seed it manually in your tests if you want a deterministic behavior.
It's like if you used `time.Now()` only to discover that you first have to call `time.StartClock()` for it to work as expected, it can make sense but is not the commonly expected behavior.
Yes? You don't?
"The directory $GOROOT/pkg no longer stores pre-compiled package archives for the standard library: go install no longer writes them, the go build no longer checks for them, and the Go distribution no longer ships them. Instead, packages in the standard library are built as needed and cached in the build cache, just like packages outside GOROOT. This change reduces the size of the Go distribution and also avoids C toolchain skew for packages that use cgo."
https://tomaszs2.medium.com/%EF%B8%8F-go-1-20-released-its-l...