Secure Randomness in Go 1.22
go.dev
go.dev
> Go aims to help developers write code that is secure by default. When we observe a common mistake with security consequences, we look for ways to reduce the risk of that mistake or eliminate it entirely. In this case, math/rand’s global generator was far too predictable, leading to serious problems in a variety of contexts.
> For example, when Go 1.20 deprecated math/rand’s Read, we heard from developers who discovered (thanks to tooling pointing out use of deprecated functionality) they had been using it in places where crypto/rand’s Read was definitely needed, like generating key material.
I made exactly this mistake in rclone. I refactored some code which was using the Read function from crypt/rand and during the process the import got automatically changed (probably by goimports when mixing code which did use math/rand) to math/rand. So it changed from using a secure random number generator to a deterministic one rclone seeded with the time of day. I didn't notice in the diffs :-( Hence
https://www.cvedetails.com/cve/CVE-2020-28924/
So this change gets a big :+1: from me.
Also has corresponding analyzer rule if you want to enforce this project-wide[1].
[0] https://learn.microsoft.com/en-us/dotnet/api/system.security...
[1] https://learn.microsoft.com/en-us/dotnet/fundamentals/code-a...
text/template and html/template are similar. In hindsight this package name shadowing was a bad idea.
* It is syntactically identical to `variable.Member` so even at a glance it's ambiguous. After shadowing occurs, diagnostics get really confused. Go could at least include more error information when this is a possible cause.
* The best package names have a lot of overlap with the best local variable names and in some cases good unexported type names [1]. Single words, almost exclusively nouns, usually singular form (prior to `slices`/`maps`), etc. so if you follow these idioms for both packages and variables you risk a lot of shadows. Who among us hasn't wanted to shadow the name `url` or `path`? To this day I feel I waste a lot of time trying to choose good package names without burning the namespace of good variable names.
Outside the standard library, the official gRPC library is an arguably even better example of terrible package naming: `codes`, `peer`, `status`, and `resolver` are all likely variable names too. The only properly namespaced package is `grpclog`.
* Even if you accept that lowercase is unexported and uppercase is exported, the lowercase side of this makes package-level unexported type names more likely to shadow package names. You can nest those type names inside functions, unless they're generic functions, and these nested types can't have methods so they're very rarely useful. If intra-package namespacing and privacy were more refined, at least we'd have more workarounds to avoid type names shadowing package names.
* Semi-related, with packages being the only way to enforce privacy, it seems like we're encouraged to create a lot of packages if we want to enforce privacy a lot. But the more you create, the more globally unique names you have to choose, creating more pressure on that namespace shared with variables and unexported types.
* You can't nest `pkg1.pkg2.Symbol` even though you can nest `var1.var2.Member`[2]. This could get ugly fast and I don't actually want this in the language, but if it had existed then it would have been a useful tool to resolve ambiguity and separate namespaces without aliases. In any case it's another inconsistency between syntax and semantics.
* Import aliases can solve a lot of problems on the spot, but trades them for other problems. When used inconsistently, humans can get confused jumping around between files, but there's no tooling to enforce consistent use, not even a linter. Even when used consistently, tooling can still get confused, e.g. when you move code to another file that doesn't have the import yet then your tooling has to make its best guess and even the state of the art makes a lot of mistakes. In general, having even a single import alias makes code snippets less "portable" even within a project and even more so across projects, so in practice they should be avoided.
* Package names are tied to file organization while still also being tied to privacy. When you're perfectly happy with a project structure, this is very elegant and you forget all about it. When you want to refactor a bit, especially to avoid a circular dependency or adjust privacy control, you have to update all callers at best or are permanently limited by your exported API at worst [3]. I observe that most libraries out there prefer to have just a couple of huge exported packages, giving up privacy on their end in exchange for simplicity for users.
* Dot imports were designed in a way that ensures they never get used. You can only dot-import an entire package, not individual symbols from it, so any dot-import is a semver hazard and discouraged. It didn't have to be this way, importing individual symbols (like C++, Python, Java, Rust, and probably many more [4]) would have still reduced clutter without trading it for a semver hazard. It's a strange oversight in a language otherwise so well suited to writing future-proof code. Of my gripes in this comment, I think this is the main one I feel could be resolved by backwards-compatible language extensions, but it's also the least relevant to the original issue of name shadowing.
These are all manageable issues when you carefully structure your own projects and pick all of your own package names. Though, I'm sure anyone who has worked in Go long enough has had to contribute to (or outright inherit) a project written with a very different approach that has a lot more shadowing hazards, showing how these individually simple rules can combine to serious quality of life issues on a project. Exported package names often become unfixable, so when you inherit a situation like that, you're going to get routine papercuts with no real way to avoid them.
[1] If the separator wasn't the same `.` then this wouldn't be a problem, though I admit any other separator in this position is automatically uglier.
[2] Assume the type is in the same package so the middle token can be lowercase.
[3] Rust goes a bit far in the other direction by always requiring an explicit module structure, but once you have it, it's actually decoupled from file organization and privacy. That said, my overall experience has been that managing `mod` and `use` hierarchies in Rust is still far more clunky and manual than packages in Go. The good parts are fine privacy control, separate namespaces, and consistent nesting, the bad part is needing boilerplate even in what should be simple cases.
[4] I respect that Go doesn't copy other languages and does its own thing, but when a feature like importing individual symbols is common to languages as different as Python and Rust, there might be real value in solving the same problem in Go too.
Good, unqualified imported names are horrible for readability.
I've always loved the way Go's packages work. With care, I can organize large codebases that end up feeling so clean and nice to deal with.
I usually get to pick all my own package names, so the issues you bring up are manageable for me. But you explained them well, and I can imagine the frustration of inheriting public API packages that were not carefully designed.
And this one is a perennial little naming hiccup:
> Who among us hasn't wanted to shadow the name `url` or `path`?
I still prefer Go's packages to other languages' ways of code organization, but I think you convinced me that giving packages their own separator (and not sharing '.') might have been a better design. Though strangely, as you mentioned, any other separator is uglier.
Anyway, I'm glad we're cleaning all this up!
So not goimports problem, my problem for expecting goimports to magically do the right thing like it usually does!
Go keeps getting better. Thanks for your hard work!
So goimports rocks and my code review skills suck!
This was discovered after I found in our code someone used math rand and was curious where they copied it from :)
The reason is that I'm using the VSCode extension that automatically adds imports. In all the source files requiring secure randomness, I carefully imported `crypto/rand` manually, but I forgot to do it in one of the files. Everything compiled and worked file, and I didn't notice that in that specific file, the extension silently added a `math/rand` import.
Since then, I import `crypto/rand` as `cryptorand` to avoid the wrong `rand` to be automatically imported.
By the way, Zig also uses a ChaCha8-based random number generator, and for cryptographic operations, people can't supply their own generator; the secure one is always used. For testing, some functions accept an explicit seed. For constrained environments, the standard library also includes a smaller one based on the Ascon permutation and the Reverie construction.
And then it happened again with every IDE. And with other packages that haven't been special cased yet. And with… I dunno, I never could convince anyone but it just seems like a terrible idea to me. Please just write out your imports, it doesn't take that long. :(
I have to disagree with that; add debug fmt.Println() → add fmt import. Okay, found it, so remove debug → remove fmt import. Okay so the problem was that we need to use filepath.EvalSymlinks() → add path/filepath import. Wait, that still didn't work; let's add back that debug Println()... etc. etc.
Other people may have other dev cycles, but I hugely miss it when it's not available (I sometimes do some work on VMs for cross-platform work where this is the case).
Same with gofmt really; these days much of the time I just write:
if foo==bar{fun()}else{panic("oh noes")}
And let gofmt sort it out.
Of course I can write all the spaces and whatnot but why bother?
I do agree the whole package shadow thing is a right pain.
The tools need to get better though. I'd rather they fall back to asking me than guessing when the import is ambiguous. I also think they should only ever autoimport from the stdlib or what's directly referenced in go.mod.
Imports should rarely be manually managed directly in source. Doing that is the epitome of tedium and most imports are just boilerplate, so your eyes will glaze over and you'll miss the finer points unless a specific file warrants deeper scrutiny.
That doesn't mean imports should never be reviewed, and again, I think the tools need a lot of improvement in this regard.
The problem here is that there were multiple possible imports for the same name, and in such situation, the extension picks an arbitrary one.
It feels rather like a bug (or a design issue) in the extension, that can be fixed, than a conceptual issue in automatic imports.
Go will refuse to compile if you have unused imports.
When I'd be playing around trying to get things to work, the import management thing was a lifesaver, I'd delete a line of code that was only a temporary exploration and its imports would disappear, I'd write something similar and the tool would add the imports back again. The annoyance of having to add them back constantly or always make sure there was some dummy "use" somewhere or manually comment them out and back in every time would be incredible to me.
It seems like it would be a lot better to simply warn on used imports and maybe refuse in release mode, but Go apparently thinks religious purity in one aspect is worth the price of making standard tooling remarkably lax in another?
Now that math/rand.Read is marked deprecated, at least if it does get selected, you get a nice strikethrough rendering as well.
It seems to be better to err on the side of 'people dont know if they want a PRNG or a CSPRNG' and switch the default to the latter with an explicit choice for the former for people that know what they need :)
I think theres another little force that pushes people towards the PRNG even when they don't need seeding: CSPRNG api always includes an error you need to handle; in case the sys call fails or you run out of entropy.
I'm curious how often crpyto.Rand read fails? How much random do I have to read to exhaust the entropy of a modern system? I've never seen it fail over billions of requests (dd does fine too). Perhaps a Must/panic style API default makes sense for most use-cases?
Edit to add: I took a look at the secrets package in python (https://docs.python.org/3/library/secrets.html) not a single mention of how it can throw. Just doesn't happen in practice?
> Perhaps a Must/panic style API makes sense?
Yes, CSPRNGs APIs should be infallible.
A user-side CSPRNG — which is the point of adding a ChaCha8 PRNG to math/rand — performs no syscall outside of seeding (unless it supports reseeding).
> you run out of entropy.
Running out of entropy has never been a thing except in the fevered minds of linux kernel developers.
Linux used user input and network jitter to generate random numbers, not a pure pseudo random number generator. For a perfectly deterministic pseudo random number generator entropy is only required for seeding and even then you can avoid it if you have no problem with others reproducing the same chain of numbers.
Linux used/uses user input and network jitter as the seed to a deterministic CSPRNG. It continuously mixes in more unpredictable bits so that the CSPRNG can recover if somehow the kernel's memory gets exposed to an attacker, but this is not required if the kernel's memory remains secure.
To reiterate, running out of entropy is not a thing.
Attack on the RNG state or entropy is much more of a risk. The entropy required is not a function of how much randomness you need to generate but how much time the system has been running.
In addition to math/rand being faster, I was worried about exhausting the system's entropy pool for no good reason: in this case, the only possible reason to have the ID's be random would be to serialize and de-serialize the data structure, then add more components later; which I had no intention of doing.
Not sure exactly how the timing of the changes mentioned in this blog compare to my experience -- possibly I was using an older version of the library, and this would make crypto/rand basically indistinguishable from math/rand, in which case, sure, why not. :-)
It's closer to the other way around. crypto/rand was not modified in any way, its purpose is to expose the OS's randomness source, and it does that just fine.
math/rand was modified to be harder to confuse with crypto/rand (and thus used inappropriately), as well as to provide a stronger, safer randomness source by default (the default RNG source has much larger state and should be practically impossible to predict from a sample in adversarial contexts).
> I was worried about exhausting the system's entropy pool for no good reason
No good reason indeed: there's no such thing as "exhausting the system's entropy pool", it's a linux myth which even the linux kernel developers have finally abandoned.
Also, make sure that your patch does not introduce a secure vulnerability as math/rand output is not suitable for anything security related.
That’s exactly what we did in PHP 8.2 [1] with the new object-oriented randomness API: If you as the developer don’t make an explicit choice for the random engine to use, you’ll get the CSPRNG.
Now unfortunately the hard part is convincing folks to migrate to the new API - or even from the global Mt19937 instance using mt_rand() to the CSPRNG using random_int() which is already available since 7.0.
[1] https://www.php.net/releases/8.2/en.php#random_extension
There's a lot of stuff like that still floating around. One of my favorite examples is that the *at family of file handling functions really ought to be the default (e.g., openat [1]), and the conventional functions really ought to be pushed back into a corner. The *at functions are more secure and dodge a lot of traps that the conventional functions will push you right into. But *at functions are slightly more complicated and not what everyone is used to, so instead they are the ones pushed into the background, even though the *at functions are much more suited to 2024. I'm still waiting to see a language's standard library present them as the default API and diminish (even if it is probably impossible to "remove") the standard functions. Links to any such language I'm not aware of welcome, since of course I do not know all language standard libraries.
[1]: https://linux.die.net/man/2/openat , see also all the other "See Also" at the bottom with functions that end in "at"
https://github.com/securego/gosec/blob/d3b2359ae29fe344f4df5...
>... but it should not be used for security-sensitive work ... This package's outputs might be easily predictable regardless of how it's seeded. For random numbers suitable for security-sensitive work, see the crypto/rand package.
If that's the case, then why hint at using math/rand/v2 "for secrets" in the blog post? Is the short version that we should all still use "crypto/rand" for anything sensitive, and all of the improvements described here are a safety net should someone inappropriately use math/rand/v2?
> It’s still better to use crypto/rand, because the operating system kernel can do a better job keeping the random values secret from various kinds of prying eyes, the kernel is continually adding new entropy to its generator, and the kernel has had more scrutiny. But accidentally using math/rand is no longer a security catastrophe.
Go is doing the right balance between safety and performance for the standard library (and for the apps built on top of it). Hopefully other ecosystems follow suit.
If an application needs fast insecure random numbers - they should implement an app internal generator themselves. Having insecure randoms within easy reach is a footgun we can put away.
Encouraging people to assume the "random" primitive is cryptographically secure is just encouraging bad practice. Making math/rand/v2 cryptographically secure might solve a problem, but it's now making something which doesn't look like it's promises security "okay".
`math/rand` functions in general though do not have the convention of being cryptographically secure - changing them so they are is just making a change to make bad code do the right thing, potentially masking the fact that if we're making that obvious mistake, what others are we also making?
> Using Go 1.20, that mistake is a serious security problem that merits a detailed investigation to understand the damage. Where were the keys used? How were the keys exposed? Were other random outputs exposed that might allow an attacker to derive the keys? And so on. Using Go 1.22, that mistake is just a mistake.
It is not masking a mistake - you should be using a CSPRNG in security sensitive contexts, however if you screw up the collateral damage is much lower. You should be detecting this mistake with static analysis and code audits rather then observing the random number generator on prod.
Whereas `crypto/rand` is very obviously "the CSPRNG function".
I understand the motivation, but if we're trying to find bad code why not go further and just require people to put //go:I AM NOT DOING CRYPTOGRAPHY at the top of files which use math/rand or something?
It's unloading the gun but not worrying that people are still pointing it around and clicking the trigger.
Rob Pike and I exchanged mail with Don Mitchell (who wrote the original C version of the Go 1 generator) a few months back to see how he would describe the algorithm, and he said "As I recall Jim and I implemented Marsaglia's LFSR-like generator."
I think both descriptions (lagged Fibonacci and LFSR-like) are accurate in different ways, so either would be fine, but for the post I decided to use the original author's description.
One nit.
I think statistical randomness seems to be conflated with pseudo-random number generators here. (PRNGs)
The wiki definition of statistical randomness is: "A numeric sequence is said to be statistically random when it contains no recognizable patterns or regularities"
Does this apply to true random number generators (TRNGs)?
You better hope it does. At least in the long-run or "in the limit". Otherwise, it is not a TRNG.
A TRNG should generate, in the long-run, "A number sequence containing no recognizable patters or regularities.
So, I think we can say: statistical randomness does not imply PRNGs, but can apply to TRNGs also.
I think the issue comes from the fact there are a large number of statistical randomness TESTs for PRNGs to qualify PRNGs with a qualified form of statistical randomness.
So, I think psuedo-random number generators would have been more appropriate than "statistical randomness" to identify PRNGs.
But, it's really a small nit.
Again, great article!
This is such a developer-friendly take especially for all of us who have had unfortunate run-ins with java.util.Random
Call me paranoid but my mind immediately jumps to the question of whether this paper can be trusted or if it has been planted by a TLA to intentionally weaken crypto.
I don't know Jean-Philippe, or much about them, but they seem to be both an experienced cryptographer and someone who has founded a company that is close to many government-adjacent organisations (UNHCR, banks, payment services, defence contractors—and that's just from the home page [0]) and therefore could easily have been exposed to persuasive state actors.
Does anyone know more about the security of the 8-round form and whether we should be concerned?
> “But what if your adversary is NSA or Mossad? Won’t they have the computing capabilities to run a 280 attack?” Such a question is irrelevant. If your problem is to protect against such adversaries, the answer is probably not cryptography.
Handwaving away better cryptographic security on the basis that they'll probably get what they want some other way does not work for me. This is likely and often true, but those other methods may be more expensive, be unusable without revealing their hand, or be politically or diplomatically sensitive.
We should not give up on our security being as resistant as possible to these agencies on such a basis.
> Does anyone know more about the security of the 8-round form and whether we should be concerned?
This is the latest cryptanalysis I could find (see Table 2 and 3 for an overview):
https://ieeexplore.ieee.org/document/10410840
We don't even have an attack against ChaCha8. While it is likely one will appear as cryptanalysis improves, it is far less likely such an attack will ever become practical.
But obviously, not everyone from within the cryptographic community would agree with JP Aumasson either. For example, DJB had this to say 1 year and 5 months before "Too Much Crypto" first appeared on the IACR ePrint archive: https://twitter.com/hashbreaker/status/1023969586696388613.
So in conclusion; somewhat inconclusive? Going by the results so far, ChaCha8 is probably fine.
You're being too paranoid. If you have a substantive disagreement with the content of the "Too Much Crypto" paper then we can talk about it, but to posit that Aumasson was compromised by a TLA(with no evidence) and that this paper is the result is pure conspiracy thinking.
Aumasson designed BLAKE[0], as well SipHash[1] and SPHINCS+[2](both of which he designed with DJB, btw).
[0]: https://www.blake2.net/#co [1]: https://en.wikipedia.org/wiki/SipHash [2]: https://sphincs.org/
Except we have some evidence that the NSA has compromised processes in exactly this way before. The OP was just asking a question and suggesting a likely and known mechanism for perfidy, he didn't actually posit that it was true.
Regardless of whether there’s a third party with an ulterior motive or it’s (more likely) simply the author’s genuine opinion, the paper “Too Much Crypto” seems ok with limiting the security of cryptography to levels that may not be secure against the most advanced and well-resourced adversaries:
> “But what if your adversary is NSA or Mossad? Won’t they have the computing capabilities to run a 280 attack?” Such a question is irrelevant. If your problem is to protect against such adversaries, the answer is probably not cryptography.”
You may agree with that, too. But it’s quite an opinionated stance and one that I’d expect to see clearly signposted and explained in API docs, and for the more expensive and secure alternative to also be available.
So that you could annotate a variable as needing to be cryptographically secure and the language could check that, somewhere along the way, its value depends on an adequate RNG function?
ChaCha8 is still very fast even without direct hardware acceleration. The 32-bit benchmarks at the end of the post are running with no assembly at all and still running within 2X of the 64-bit SSE2-based assembly. AES-GCM with hardware is pretty fast, but AES-GCM without hardware is quite slow.
Just now I tried benchmarking ChaCha8 in 256-byte chunks compared to AES-GCM in 256-byte chunks. With HW acceleration, AES-GCM is maybe 10% faster on my Apple M3 but 20% slower on my AMD Ryzen. Same ballpark as ChaCha8 though.
On the other hand, if I disable AES hardware acceleration, that same benchmark drops by about 20X. So using AES would not have been a good idea for systems without AES hardware.
Overall, not much win to AES in the best case, and quite a loss in the worst case.
I'm kind of surprised that it's slower on AMD Ryzen - it looks like only the Pro series have a an actual co-processor. Weird decision on AMD's part to implement AES-NI without HW acceleration on some CPUs instead of just not implementing the AES-NI instruction set. That being said, AES-CBC would be even better for this purpose since the authentication guarantees aren't needed.
On my Intel machine, it's 5.7 GiB/s for AES-GCM. I don't know how you benchmarked the chacha8 version so I can't run the equivalent on my machine.
go test -bench=Block internal/chacha8rand
go test -bench=GCM/-128-256 crypto/cipher
GODEBUG=cpu.aes=off go test -bench=GCM/-128-256 crypto/cipher
I meant that AES-GCM is 20% slower than ChaCha8 on that system, not that HW-accelerated AES-GCM is 20% slower than a software implementation. On the contrary, the HW-accelerated AES-GCM is 20X faster than software on that system.
It was unfortunately a bit slower: ~27 ns per 64-bit value (6 round permutation) vs ~4 ns for the included ChaCha8. I suspect it could be optimized, and run at the higher output rate (8 rounds per 128-bit output). One nice thing is that it does have a smaller state of only 40 bytes.
But - for the performance, this ChaCha8 implementation is awesome!
All in all, I leave my thanks for the excellent work done by the Go team here!
I’m furious at C++ for removing the easy to use random_shuffle, with no easy to use replacement.
I’ve seen several programs use C++s new random generators wrong, and end up breaking programs while removing random_shuffle.