On Linux's Random Number Generation | Hacker News Reader

On Linux's Random Number Generation

research.nccgroup.com

176 pointsraesene983 comments

This is Thomas Pornin, by the way, from whom I and many others have learned a lot, and whose original Stack Exchange answer† about random vs. urandom set me and presumably Thomas Huhn (there's something about urandom and Thomases) tilting against this Linux windmill with our respective urandom advocacy pages, both of which pop up on HN every couple of months.

This article is authoritative and also covers some recent updates, like Jason Donenfeld's work on the Linux RNG, is well worth a read, and hopefully will show up on HN at least as often as me and Thomas Huhn's.

https://security.stackexchange.com/questions/3936/is-a-rand-...

That, and the article, seems like sort of a tangent though. In practice, /dev/urandom advocacy has won for almost all apps. That's what everything uses, and it meets the requirements desired by you and Pornin AFAICT. People who want crypto-backed non-blocking kernel RNGs have a very high quality one in linux. Most of this is just an argument about defaults, whether "/dev/random" should be non-blocking by default and whether it should be renamed to something else.

The more interesting question, and one I think is very poorly served by this article, is whether or not Linux's entropy pool engine is a useful feature or not (and tangentially whether or not we should trust RDRAND). And... I mean, in a world where we have known backdoors in both hardware and crypto algorithms, it seems like having a system that defaults to not trusting them might not be the worst thing...

Basically, this argument seems really fragile to me. All it takes is one paper showing a likely backdoor in RDRAND and everyone is going to be laughing like crazy at what-in-hindsight-would-seem the ridiculous naivete in this article.

The point of this article is not that we should exclusively trust RDRAND.

No, but that's the focal argument that IMHO makes it fragile. Again I repeat the point: if tomorrow someone published a paper showing evidence of a backdoor in RDRAND, would you still cite this article as good advice?

The point of the article is that the Linux entropy architecture is bad, mostly because it's over-engineered and unnecessary (and that it's the default, I guess). Which only makes sense in a world where we can trust our PRNG entropy sources and the algorithms used to generate the stream. And there is an uncomfortable amount of actual evidence that that is not the world we live in.

A Cryptographically Secure Random Number Generator relies on cryptography being correct and not breakable in a useful amount of time.

If that doesn’t hold what in the world are you using the random numbers for??

If it does hold then the whole idea of an “entropy pool” is useless cruft.

Think of it this way: AES has to look like random noise. If there is any correlation to the input detectable by someone without the key it would be useless for crypto. A CSRNG isn’t quite the same thing but you can think of it the same way: it only needs some initial entropy, just like AES needs a key. Thereafter it must be able to generate random bytes faster than they could possibly be consumed (TB/s or more) without disclosing any information about its internal state or it is a useless CSRNG whether the entropy pool exists or not.

> AES has to look like random noise. If there is any correlation to the input detectable by someone without the key it would be useless for crypto.

Uh... AES (to pick your particular example) is a symmetric algorithm. By definition there is a 1:1 correlation to the input. If you have the key and know the input you can compute the output, and vice versa. The question at hand is not whether the algorithm is breakable but whether someone can find the key (or more generally the PRNG state).

And that's what the entropy pool does: it seeds the PRNG state with values that are derived at runtime by the kernel and not under the control of an attacker, even one embedded in the system.

There are side arguments about whether the entropy pool's estimates are good ones, about whether it should block by default when empty, etc... But I don't see any reasonable arguments that this isn't a useful feature for an OS kernel to provide.

> By definition there is a 1:1 correlation to the input

But it is still not detectable without the key.

You still need some way to securely seed a CSRNG. That's what we use the entropy pool for in Linux. The idea is that we mix multiple sources, and hopefully at least one can't be guessed by a particular attacker.

For example, you might use three hardware random number generators; one might be compromised by the Chinese MSS, and one might be compromised by the KGB, and the third might be compromised by the NSA --- but hopefully they won't be all compromised, and even if they are, hopefully they aren't working together.

Or maybe we're mixing randomness from the UEFI and the TPM. Neither can be audited, and we know that many firmware engieners are incompentent. But hopefully, if the UEFI BIOS has some terrible bug, or can be remotely compromised because Intel snuck in a Minux OS with a web server without telling us in their CPU's, maybe the TPM provider didn't make a similar mistake.

> In practice, /dev/urandom advocacy has won for almost all apps.

Which is a shame, because the thing that should've won is arc4random_buf or something similar that always works (empty chroot with no /dev/ under it, isolated user with no read access to anything on the filesystem, and no free file descriptors? still works!) and can't give an error.

getrandom should be used:

https://lwn.net/Articles/606141/

for most user code it's already good enough, the breakage happened in some early use cases:

https://lwn.net/Articles/800509/

For clarity: I'm using "/dev/urandom" as shorthand for "the non-blocking interface to the kernel entropy pool" and not the literal device node interface. As others have pointed out there is a syscall to get that for you if you can't use the device.

To be completely correct: On modern kernels, /dev/urandom is not an interface to the kernel entropy pool. It is rather a ChaCha20 Cryptographic Random Number Generator which is periodically seeded from the kernel entropy pool.

getrandom(2) uses the same ChaCha20-based CRNG. The main advantage is that it doesn't require that you be able to open a file descriptor. (There were some obscure attacks on some userspace where the attacker would exhaust the number of file descriptors, and the sloppy userspace code wasn't checking error returns, and so it wasn't actually reading from /dev/urandom. Getrandom(2) is supposed to be foolproof --- which is hard, given how ingenious fools can be, but at least it's harder to screw up using getrandom(2). It also works in the completely empty chroot scenario referenced in the grandparent of this post.

There are a number of things that the blog post gets wrong. First of all, it was not Jason A. Donenfeld, the author of Wiregard, which added the ChaCha20-based cryptographic random number to Linux. It was me, as the maintainer of Linux's random number generator. The specific git commit in question can be found at [1].

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

Secondly, I think a lot of people have forgotten what things were like in the early 90's. Back then encryption software was still export controlled, so we couldn't put DES (AES wasn't released until 2001) into the kernel without triggering all sorts of very onerous US government restrictions (and Europe was guilty of this too, thanks to the Wassenaar Agreement); you couldn't just put things up on an FTP. Also, back then, there was much less public understanding of cryptography; the NSA definitely knew a lot more about cryptoanalysis that in the public world, and while it was known by 1996 that MD5 had Problems, there wasn't huge trust that the NSA hadn't put a back door into SHA-1 which was designed by them with zero explanation about its design principles.

This is why the original PGP implementation, as well as the Linux Kernel random number generator, was very much focused on entropy estimation. We knew that it was potentially problematic, but then again, so was relying on cryptographic algorithms that were poitentially suspect. There was a good reason why in the 90's, it was generally considered a very good idea to be algorithm agile; there simply wasn't a lot of trust in crypto design, and people wnated to be able to swap out cryptographic algorithms if it was found that some algorithm (e.g., like MD4, and later MD5) was found to be insecure. So the snide comments about people not trusting algorithms seems to miss the point that even amongst the experts in the field --- for example, at the Security Area Directorate at the IETF, of which I was a member during that time --- there was a lot of thinking about how we could deploy upgrades if it were found that some crypto algorithm had a fatal weakness, and we would need to swap out crypto suites with minimal interoperability issues.

Unfortunately, being able to negotiate crypto suites leads to downgrade attacks, such as we've seen with TLS --- but what people forget is that when the original SSL/TLS algorithm suites were designed, people thought they were good! It was only later that some crypto suites were found to be insecure, leading to the downgrade attack issues. But it also shows that people were right to be skeptical about crpyto algorithms in that era.

Since then, we've learned a lot more about cryptographic algorithm design, and so people are a lot more confident that algorithms can be relied upon to be secure --- or, at least, other issues are much more likely to be weak link. That's why Wireguard is designed without any ability to negotiate algorithms, and as a result, it makes it much simpler than IPSEC. And it's probably the right choice for 2019. (At least, until Quantuum Computing wipes out most of our existing crypto algorithms; but that's a rant for another day.)

As far as monitoring entropy levels in Linux, in general, the primary reason why we need it is because even if we are willing to invest a lot of faith into the ChaCha20 CRNG, we still need to provide a secure random number seed from somewhere. And that can be tricky. If you fully trust a hardware random number generator, then sure, no worries. Or if you are using a cloud provider, so you have to trust the hypervisor anyway, then using virtio-rng to get randomness from the cloud provider is fine. (If you cloud provider wants to screw you, they can just reach into the guest memory or intercept network or disk traffic at boot time, so if you don't trust not to backdoor virtio-rng, you shouldn't be using the cloud provider at all.)

As far as whether or not to trust RDRAND, the blog post seems to assume that it's absurd to trust that NSA couldn't possibly have backdoored the CPU instruction. On the author hand, there are those who remember DUAL-EC-DRBG, where most people do now believe the NSA did put in a backdoor. And Snowden revelations did show that NSA teams were putting backdoors into Cisco routers by intercepting them between when they are shipped and when they were delivered. So given that you can't audit the Intel CPU's RDRAND, and Intel is a US company, it's not that insane to perhaps have some qualms about RDRAND. After all, if you were using a chip provided from a Chinese company (where the owner of said company miight also have been a high ranking general in the PLA), or a CPU provided by a Russian company controlled by a Russian Oligarch who is good friends with Putin and who also had a background from the KGB --- is it really insane to be worried about those CPU's? Let's not even talk about concerns over China and 5G telephony equipment. Why is it then completely absurd for some people to be considered about the complete inauditability of RDRAND, and the fact that no functional or statistical test can determine whether or not there is a backdoor or not?

Of course, if you really don't trust a CPU, you should simply not use it. But creating a CPU from scratch, using only 74XX TTL chips really isn't a practical solution. (When I was an undergraduate at MIT, we did it as part of an intro CS class; but MIT doesn't make its CS students do that any more.) So the best we can try to do is to try to spread out the entropy sources; that way, even if source 1 might be compromised, if it is being mixed with source 2 and source 3, hopefully at least one of them is secure. (Or maybe source 1 is backdoored by the NSA, and the source 2 is backdoored by the Chinese MSS, but if we hash it all together, hopefully the result will only be vulnerability if the NSA and MSS work together, which hopefully is highly improbable.)

The bottom line is that it's complicated. Of course I agree that we should use a CRNG for most purposes. But we still have to figure out good ways of seeding a CRNG. And in case the kernel memory gets compromised and read by an attacker, or if there is a theoretical vulnerability in the CRNG, it's good practice to periodically reseed the CRNG. And so that means you still need to have an entropy pool and some way of measuring how much entropy you think you have accumulated, and how much has been possibly revealed ("used") for reseeding purposes. In a perfect world, of course, assuming that we had perfectly trustworthy hardware to get an initial seed, and in a world where we are 100% sure that algorithms are bug free(tm), then a lot of this isn't necessary. But we in real world engineering, we have safety margins because sometimes theory breaks down in the face of reality....

> There are a number of things that the blog post gets wrong. First of all, it was not Jason A. Donenfeld, the author of Wiregard, which added the ChaCha20-based cryptographic random number to Linux. It was me[1], as the maintainer of Linux's random number generator.

For the avoidance of doubt: I can confirm that I didn't add chacha20 to the kernel or have anything to do with that. I did move get_random_int() and get_random_long(), which are used for things like aslr and wireguard handshake ids, from some weird md5 horror to using Ted's chacha20-based rng (when rdrand is unavailable) a few years ago, but that's different from /dev/urandom. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... Ted is the one who added chacha20 for /dev/urandom.

Question: why did most OS’s move to chacha when they could have moved to AES and benefited from hardware support? Asking due to a friend who seemingly has to use a userland PRNG because getrandom is too slow

Linux has to support multiple architectures, and it's a pain in the tuckus to add conditional support for different CPU architectures which might or might have AES acceleration, using different CPU instructions.

Linux does have support for it, but you have to drag in the crypto system, which is optional, and it's a super-heavyweight and complex interface. Jason tried to simplify it for Wireguard, but ran into a lot of resistance, and he's now adding Wiregard with an interface layer to the crypto subsystem. He's still going to work on trying to add a simpler crypto interface, but a core principle of Linux's RNG is that it must always be present; I didn't want to make it an optional component that could was enbled or disabled at compile time. That means I couldn't rely on the crypto subsystem, even if I was willing to put up with its rather horrific interface. (There are some reasons for its complexity, but it adds no value to the random driver or Wireguard.)

In any case, if you really need more speed than the current ChaCha20 CRNG, you're doing something wrong. It's almost certainly not for cryptographic purposes. So if you do want that kind of speed, number one, you probably really don't need the security, and so a PRNG will always be faster. Or if you do need the security, grab a seed value using getrandom(2), and then implement a userspace CSRNG. (But I bet you really don't.)

Thanks for the answer! My friend’s response:

> Let’s take this comment at face value and say Linux rng throughput is 180MB/sec: https://www.reddit.com/r/crypto/comments/ednj0x/comment/fbjr...

> Let’s take one of the more moderate types of AWS instances and suppose our max network speed is 10gbps = 1250MB/sec.

> Supposing we have an application that does not need to do any slow operations like reading from a database and is only performing very simple operations in memory (like, let’s say, generating a symmetric key), we see that the Linux rng is roughly an order of magnitude slower than our network throughout.

In what situation do you need to saturate your network pipes with cryptographically secure random data?

What's your response to Pornin's criticism of 5.3 changes?

> Linux 5.3 will turn back getrandom() into /dev/urandom with its never-blocking behavior, because, quite frankly, Linus’s opinions on his own mastery of RNG theory exceed his actual abilities. The reasoning seems to be that if there is “not enough entropy”, then the application should make an interpretative dance of some kind to promote the appearance of new hardware events from which the entropy can be gathered. How the application does that, or why the kernel should not do it despite being much closer to the hardware, is not said

I wasn't a real fan of the 5.3 change, but it was better than some of the alternatives that people were proposing. At the end of the day, the problem is that userspace just shouldn't be trying to get randomness during early boot. But if they do, and we make the kernel more efficient things can break, and Linus believes that if there is a user-visible regression we Have to Fix it, even if it the root cause is broken user space.

In this particular case, what triggered this was an optimization in ext4 in how we did directory readahead, which reduced the number of I/O's done during the boot sequence. Some user space program in early boot tried to call getrandom(2), and it blocked because with a smaller number of I/O's, for some hardware platforms, it would stop the boot sequence in its tracks, and this then resulted interrupt events, leading to no further activity, leading to an indefinite hang.

So what do we do? We could revert the ext4 optimization, but what it did was draw attention to the fact that in the absence of a hardware random number generator which everyone trusted, what we had in terms of CRNG initialization was fragile.

Now the blog posting is inaccurate here as well, it is not the application which needs to make an interpretative dance of some kind. We actually are doing it in the kernel, and there is at least some hope that on an x86, making some assumptions about how the caches and clocks work, it probably is secure. I am actually worried that it won't be secure enough on simpler RISC cores, such as ARM, RISC-V, MIPS, etc.

The real right answer is that we should be pulling from as many hardware random number generators (that are designed for crypto purposes) that are available, such as from the UEFI, or the TPM, and mix that in as well. We'll probably continue to have config options so that the person building the kernel can decide whether or not those will be trusted. That hasn't happened yet, but I've been trying to recruit some volunteers to implement this in UEFI boot, or using NERF or Coreboot, etc. Util we do, or for hardware that doesn't have trusted hwrng's, starting in 5.3, we now have an in-kernel interpretative dance which is the fallback, for better or worse.

I'm not super-fond of that, but it was better than the alternative, which was no interpretive dance, and simply having getrandom(2) return "randomness" regardless of whether or not we thought it was random or not. On modern x86 processors, we will be mixing in RDRAND, so if you trust RDRAND, you'll probably be OK, interpretative dance or not. But the big worry is going to be on simpler CPU's such as RISC-V.

Ultimately, there are no easy solutions here. Arguably, just gathering timing events during the boot was also an "interpretive dance", since how much uncertainty there really is from SSD operations, and whether the SSD is using a oscillator different from the one used by the CPU, etc., involves a certain amount of hand-waving. So the only real solution is real, carefully designed, hardware RNG's. But then the question is how an you be sure they are trustworthy? This conundrum has always been there.

For myself, I use a ChaosKey[1] and make sure it is contributing to the entropy pool before I generate long-term public keys. Of course, can I be sure that the NSA hasn't intercepted my ChaosKey shipment and trojaned it, the way they did with Cisco Routers? Nope. I can only hope that I'm not important enough so that they wouldn't have bothered. :-)

[1] https://keithp.com/blogs/chaoskey/

> Now the blog posting is inaccurate here as well, it is not the application which needs to make an interpretative dance of some kind. We actually are doing it in the kernel,

Yeah, I was thinking the article looked inaccurate there. Thanks for confirming.

Thanks for the detailed answer!

>We'll probably continue to have config options so that the person building the kernel can decide whether or not those will be trusted.

My understanding is that XORing a trusted seed source with any number of untrusted seed sources results in a trusted seed source. What would be the point of such kernel options?

The problem is "trusted" is as much a social construct as it is a technical one.

For example, if you are an NSA employee, you might be utterly confident that the NSA didn't twist Intel's arms to put in a backdoor into RDRAND --- e.g., that it isn't AES(NSA_KEY, SEQ++) --- and even if it were, you would be sure that as a US citizen, you would be safe from intrusive attacks to spy on your communications without a FISA warrent, which of course would only be done with a super scrupulous attention to legal process, the recent IG report of the Carter Page FISA warrant to the contrary.

In 2019, if you are a Republican member of the House of Representatives, such as Devin Nunes, you might be sure that the FBI is playing fast and louse with all FISA warrants, and so if so no one is safe from politically motivated investigations, especially if you are working for the Trump campaign, such as Carter Page.

See? Two different people might have very different opinions about whether a particular source should be trusted or not.

My point is that you don't have to trust any particular source of randomness. NSA can backdoor RDRAND all they want. As long as there is a single legit source of randomness XORed in then the NSA is just wasting their time.

So having options to remove some sources is pointless and can only make things worse, never better.

Sure, and that's why we still have the entropy pool in Linux. It's where we do the mixing.

My apologies if I didn't understand the point you were making.

And yes, I would love to see us adding more entropy sources in the future. The problem is I don't have a lot of time, and a lot of other projects to work on, but I've been trying to recruit people to add support to pass entropy from UEFI into the kernel (which requires changes to Grub, NERF, Coreboot, etc.), being able to pass entropy from one kernel to the next when using kexec, etc. I can't do it all myself; this is something that requires a lot of people to make things better.

Check out https://blog.cr.yp.to/20140205-entropy.html Yes, xorring RDRAND with your existing secret can hurt security in certain cases.

I think that's a pretty bad misrepresentation of the situation. The root problem is that security-first people have a hard time trusting anything for real entropy to seed the RNG at first boot.

RDRAND is used, but not trusted by default (and there was a fault in AMD RDRAND with old un-patched bios). systemd applies a seed file generated during last boot, but again disables the option to credit the entropy pool, because it does not trust that this wasn't an improperly prepared and distributed VM image with the same seed file used over and over. There are lots of things that very likely contribute usable entropy, but the kernel can't know with absolute certainty that any particular one does not have some flaw. The final straw was ext4 optimizations and modern SSDs resulting in very few storage controller interrupts, and some programs using getrandom() during early boot, and boot locking up indefinitely for some users running the latest linux kernel, ultimately due to paranoia about the RNG entropy sources.

So if you have to choose between failing to boot up, or just giving the best random numbers you can despite lack of certainty/guarantee of sources of entropy, Linus prefers to not fail to boot up. I know that OpenBSD trusts that seed file, which Linux+systemd uses but does not trust, and I'm not sure what macOS and Windows do, but they probably trust RDRAND, which Linux also uses but does not trust. (My personal fix is to trust rdrand, there's a kernel cmdline option for it.)

Part of the problem is that Linux supports a much larger set of architectures and boot loader than OpenBSD. So trying to use a seed file which is read by the bootloader at boot time is hard.

Could systemd choose to read a seed file after it mounts the root file system? Sure, but then it's on systemd to believe that the seed file is always secure, even on IOT devices where the seed file might be imaged onto millions of devices with an identical value out of the box. Using a seed file means you also have to trust how the OS is installed; security is a holistic property involving the entire system's design and implementation.

Ultimately, it's not going to be up to kernel or systemd to tell users and system administrators what they should or shouldn't trust. If you trust RDRAND, and you're on x86, you can enable the config option or provide the boot command line flag, and you're all set. But I'm not going to tell you one way or another whether or not you should trust RDRAND. And even if I did, you could just reject my advice, either way. As I said in another reply, "trust" is as much a social construct as it is a technical one.

I'm sure Pornin can speak for himself better than I can, but it seems like this response might not so much engage with the substance of his argument than with a simplified subset of it.

It's not, so far as I can tell, Pornin's claim that RNG design is mooted by RDRAND. I would be surprised to see Pornin argue for a system design that replaced a standard RNG design, with some kind of secret-unpredictable-event secure seed, with calls to RDRAND.

But your argument that it's sensible to be cautious about RDRAND because of Dual EC might itself be misleading. Dual EC is a complete design for a CSPRNG. RDRAND is a component of a CSPRNG. In every proposed system I've seen that uses RDRAND, RDRAND is one of several secret inputs to the RNG. I've never seen a design that chains Dual EC; it was surprising to see Dual EC even used, anywhere, because it is so comically expensive. Most of the BULLRUN revelations (and the subsequent Juniper fiasco) weren't new discoveries about Dual EC, but rather the discovery of systems not thought to be relying on Dual EC that were.

Even Bernstein doesn't really argue that RDRAND hurts security in the design context you're talking about; for the same reason that you say cloud users should trust virtio, a "snooping" RDRAND is simply out of the threat model.

But of course, RDRAND isn't really the point. Clearly Pornin doesn't think that haveged makes sense in systems without RDRAND. So what are you really saying here? That he's not charitable enough about the historical context you were working in?

> Even Bernstein doesn't really argue that RDRAND hurts security in the design context you're talking about

Would you mind explaining https://blog.cr.yp.to/20140205-entropy.html then?

OT but the chacha20 code IIRC spills badly with either gcc or clang.

Off topic or no, this is very disturbing. Are we talking about power noise, or timing noise? The former is generally hard to counter, but we should be able to get our timing right. And doesn't DJB understand all this well enough to get the code right?

Your parent is talking about register spills to memory. Using memory instead of registers may be somewhat inefficient, but it's still constant-time (i.e. no dependence on the key etc.)

If the attacker has Meltdown/Specter-style access to your stack, register spills (especially but exclusively if not overwritten when you're done) can compromise your keys, but you need to have a rather weird threat model to avoid having them just read the actual keys out whatever storage you got them from in the first place.

This is kind of hard to avoid without hand-coded assembly or careful use of vector intrinsics. The core of chacha20 is a 4x4 matrix, which would need 16 registers to hold without spilling (plus a few more for temporary use during the calculations). Both 64-bit x86 and 32-bit ARM have only 15 or 16 general-purpose registers.

FWIW x64 effectively has 32 GPRs worth of 32-bit registers, which is what chacha20 needs. You'd need to access them as something like:

  add eax r8d
  rol r8 32
  add edx r8d

though, so several are useless due to clobbering, and you'd need to schedule things rather weirdly. Also, where did you get the RNG state from, if you don't trust memory to be read-what-where-proof?

You could also use the SSE/AVX registers.

Probably true (I think all x64 hardware includes SSE at least), but they did imply they were trying to use just the scalar instructions. (They also said no hand-assembly, but something like:

  register uint64_t ra,rd,...
  ra += 0xFFFFFFFF&r8
  rd += r8 >> 32

should accomplish much the same thing given a resonably well-developed compiler.)

Specifically on

> it's good practice to periodically reseed the CRNG. And so that means you still need to have an entropy pool and some way of measuring how much entropy you think you have accumulated, and how much has been possibly revealed ("used") for reseeding purposes

Periodically re-seeding the PRNG can be argued to make sense, but doesn't require entropy estimation: consider https://en.wikipedia.org/wiki/Fortuna_(PRNG) (and the paper mentioned under "Analysis".) Not needing to estimate entropy does come at the cost of a longer time-to-reseed, but that still seems a good trade-off to me.

(Just in case you weren't already aware of that design.)

> I think a lot of people have forgotten what things were like in the early 90's.

Thank you for posting this.

“Those who do not know history's mistakes are doomed to repeat them.”

A few problems with the reasoning in this article.

It assumes that either "cryptography works" (and you only need to seed the CSRNG once) or "cryptography does not work" (and the entropy pool is necessary). In actuality, "cryptography" covers a wide range of algorithms, some of which seem to work, and some of which (like RC4) are known not to work. It's possible that (say) ECDSA works but Linux's CSRNG isn't as good as we think it is.

The article also makes fun of the kernel for not trusting RDRAND. Well, here's a sequence of random numbers from RDRAND on the AMD Ryzen 3000: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff. Do you see a pattern? Was it indeed a foolish choice not to trust RDRAND?

Nothing malicious in this RDRAND output btw if you use through /dev/urandom or getrandom

People were initially scared about it cancelling other entropy sources, which still hasn’t been proven.

Funnily enough, the kernel does trust RDRAND (for fetching a single-integer "random enough" value[1]). That's why WireGuard had a DoS bug on those chips until they patched around it to use the proper CSPRNG after a few failed attempts.

[1]: https://elixir.bootlin.com/linux/v5.4.5/source/drivers/char/...

> Was it indeed a foolish choice not to trust RDRAND?

From that output? No, not at all. It's not backdoored, it's the same as any other microcode bug. What would be more insidious was if it gave back 0xD82C07CD, 0x6BAA9455, 0x82E2E662, 0x7A024204: seemingly random but secretly predictable. (Hint: Python, random.seed(0))

You can always check with Random Sanity https://www.randomsanity.org/ In this case https://rest.randomsanity.org/v1/q/D82C07CD6BAA945582E2E6627... It is not 100% sure, it was "true" until a few seconds ago.

I strongly advise against trusting statistical entropy tests in the application layer.

There are reasons to do statistical tests (for example, the USB key Ted mentioned needs to make sure the circuit is still generally functioning), but when you're in the application, there has already been so much de-biasing and whitening going on, that you will never really catch a problem. You will only obsess about a random (ha!) fluke where a tests sporadically fails.

It was a semi joke. It's an interesting project, but note that:

It is not so popular, so they didn't have this string before. (Perhaps they can preload some common cases, but perhaps is too much memory.)

Once you send your secret random stuff to a random site in the internet, it's no longer secret. You can send some samples as a test and discard them, and use other part of the random stream, but from a security point of view I'm not sure it is a good idea.

I agree that flukes are a problem if you call something like this too much. I'm not sure about the implementation details to be sure how strict it is.

I don't understand why nobody ever mentions CCD noise as a source of quality randomness. Billions of devices, now, including all phones and most laptops, have an extremely productive seed source with many millions of pixels, each pixel supplying at least a quarter bit of entropy. It works equally well with a sticker over the lens.

Microphones are also good sources, down in the low bits, because true silence does not exist. Like CCDs, even if you don't think much of the entropy in any one sample, there are one hell of a lot of samples to draw on.

If you have an accelerometer, that's another six sources. If the device is on the table, jiggling the vibration motor provides enough activity.

The RDRAND behavior in that shipment of Zen2 chips suggests that the hardware RNG can be turned off by the System Management backdoor device, and that a code path to do it is active and (a little bit too) available. People messing with CCD pixel readings seems less likely.

> I don't understand why nobody ever mentions...

Most of these things aren't present in the context of a server, let alone a virtualised one.

Nobody's there to literally physically jiggle your virtualised server while it boots.

A virtualized server has no reliable source of randomness at all, besides whatever the host provides. They are out of the picture entirely.

> They are out of the picture entirely.

They’re a central topic of this article and where the problem we're discussing is usually felt.

> Most of these things aren't present in the context of a server

That's a good reason why it cannot be the main source of randomness, but I find parent's point extremely good.

Of course, there might be hesitation about activating a camera for "extraneous" purposes.

I'm not sure if this is just a typo/mistake, and perhaps what was originally intended, but CCDs are almost no longer present on most consumer devices. The image sensors used now are most commonly CMOS-based, not CCD.

What, if any, change to your original assertion that makes, I don't know. That depends on if the quality of the randomness you cited is significantly different between CCD-based sensors and CMOS ones.

It is true that I missed the shift from CCD to CMOS cell tech, but it doesn't change anything. It might be that the camera chip filters out much of the noise, which would be too bad, but there is plenty left.

The problem that lead to the Linux 5.3 changes was early boot. Even if your system has a camera early in boot it is not likely to be initialized. I fear it might be difficult if not impossible to have a generic kernel that gets entropy from an unknown model of camera early in boot.

Good point. Where to get randomness when nothing is initialized yet is a tricky matter. I wonder if RAM could be mistuned to generate enough randomness. Certainly not always and everywhere, but probably usually, if you are running on bare metal.

If not, whoever is has plenty of randomness to hand over.

This was a good article. Bits of entropy are not lost as the CSPRNG spits out bits 1:1. Nobody thinks that a 64 bit key is compromised after a CSPRNG outputs 64 bits.

However RDRAND should not be trusted because we know for certain that motherboard BIOSes have the capability to force the output of RDRAND to -1 every time it is called. Hardware vendors have demonstrated this to us as a "bug" multiple times, but I consider it a "wink". They have fixed it with BIOS patches. The fact that this is patchable at all indicates RDRAND is not secure. The BIOS can control this behavior; therefore it is IMHO very highly suspect and not worthy of trust. There is no legitimate reason why a ring oscilator circuit with it's whitening/etc should ever output a predictable sequence of any kind.

Don't BIOS patches contain microcode updates?

Yes, an earlier version of my comment made a distinction between BIOS vs microcode, but I realized it wasn't relevant and edited it. Apparently my edit and your comment experienced a race condition.

>In a sense, whether a given mechanism provides entropy is a matter of “this or that expert said that it does”; impossibility of accurate simulation comes from physics, specifically quantum mechanics, so the entropy pool estimator is based on a fair amount of trust in physicists such as Feynman or Bohr (but not Einstein, for that matter). But the entropy depletion is an assertion that cryptographers such as Shamir cannot be equally trusted.

Well yeah, we have much more reason to believe our physical laws represent something about reality than to think our cryptographic primitives are unbreakable. Where's the analogue of Bell's theorem for cryptography?

FYI Andrew Ayer said this change didn't go through: https://twitter.com/__agwa/status/1208066598407483393

It's not clear to me what the current state is

That would explain why I wasn't able to find "CONFIG_RANDOM_BLOCK" in my kernel configs.

On a related note, I've been running jitterentropy-rngd on most of my machines for a long while now, including the ones that have internal and/or external HWRNGs (even though it's probably not really doing much of anything on most of them).

I'm far from an expert on any of this but jitterentropy-rngd seems much, much better than haveged (although, FWIW, I think pretty much anything is better than haveged). I was kinda surprised that Debian decided to use haveged in the installer, although (to their credit) they did say, in effect, "we need something now, this works for now, we can find something better later".

According to the commit log, the kernel adds a timer loop to generate some entropy. So getrandom() still blocks, but kernel is adding jitter entropy to seed the pool so the wait time should be minimal.

I find it strange that the article says that Java uses /dev/urandom by default. When I've been having to configure Java to use /dev/./urandom for years to deal with the fact that it defaults to /dev/random and setting it to /dev/urandom doesn't work because the Java code treats the two as the same.

Slightly tangential, but why doesn't Linux symlink /dev/random to /dev/urandom like OpenBSD does?

Wouldn't that stop issues like these (what's the benefit in keeping /dev/random the same)?

Extreme backwards compatibility. An application might depend on the blocking behaviour (be it sensible or not). And since it's a change visible from user space, it won't be done (except in extreme circumstances).

FWIW, Virtio RNG is natively supported by Linux, NetBSD, and OpenBSD as guests, and hypervisors like Linux KVM/QEMU and OpenBSD VMM/VMD. I use libvirt and virt-manager on Linux which makes it easy to add a Virtio RNG device, but it should be the default, as it is with OpenBSD VMM/VMD.

Ah, I just found --rng in another read of the virt-install manual. This works a lot better than aborting the install and editing the xml and restarting.

There is little more disappointing to see a position you embrace argued with weak or misleading arguments.

The entropy estimation used in Linux is indeed essentially pointless, but a number of the arguments used in this page are not great.

The main reasons I'd give is that the applications we use randomness for have only computational security at best so the effort to provide information theoretic security is overkill. The actual mechanisms that have been used in linux both for estimating random inputs and maintaining the pool are entirely adhoc and there is no particular reason to believe they'd actually provide information theoretic randomness even if you did have some fringe usage where it could be a theoretical benefit. And finally, the weird behavior like blocking RNGs added for purely conjectural benefits results in real vulnerabilities.

Not much more is needed to be said: It's a common story, security theater resulting in insecurity.

> so the entropy pool estimator is based on a fair amount of trust in physicists such as Feynman or Bohr (but not Einstein, for that matter). But the entropy depletion is an assertion that cryptographers such as Shamir cannot be equally trusted

Our expectations about the security of cryptographic constructs are largely unlike our expectations of physical systems.

Thomas is making a cheap and misleading argument here, and I very much expect that Shamir would disagree with it... (heck, one of the famous 'Shamir' named things is shamir secret sharing which comes straight from the realm of information theoretic security.)

A case can be made for cryptography with information theoretic security... but what the kernel provided in the past was at best a cargo-cult imitation of it, taking many of its costs without providing its assurances.

> The Linux kernel uses rdrand. It does not trust rdrand, because NSA (I’m not exaggerating! The kernel source code explicitly calls out the NSA),

Imagine this in a world where the kernel always blindly trusted rdrand: https://arstechnica.com/gadgets/2019/10/how-a-months-old-amd...

CPU makers can't manage to achieve really strong reliability against obvious _accidental_ fault in rdrand. Gonsidering this, some caution against backdoors in a totally black box magic source of random numbers seems eminently prudent. NSA is just an existence proof of intelligence operations with the intellectual and financial capabilities that would make backdooring a CPU possible, if not likely.

Relying on seed-based algorithms makes the seed very attractive for leaking by some side channel, how does CSRNG prevent that?

I'm not sure what the seed has to do with anything. Every CSPRNG has a state which can be leaked, and while nominally larger than a typical seed, if a side-channel leaks 32 bytes then it likely can leak 512 bytes, 1Kb, etc. (The way the article distinguishes the "RNG" entropy pool from the "CSRNG" seems rather confusing and maybe a reflection of Linux' convoluted framework. Better to think of the whole framework as a giant, overwrought CSPRNG function.)

A proper CSPRNG offers forward security, which means if the state is leaked in the future then past outputs are still secure. This is usually accomplished by cycling the PRNG after output.

There'a also backward security (perhaps what you were getting at), which means future outputs are protected from past leaks. Of course, you need new randomness to achieve backward security. AFAIK, typical kernel CSPRNGs mix in new entropy on a regular basis (notwithstanding the implication in the article that on Linux this can be unnecessarily delayed). However, that's not necessarily a better thing. Daniel Bernstein has argued that maybe, depending on your threat model, you're better off with a purely deterministic CSPRNG after initial seeding: https://blog.cr.yp.to/20140205-entropy.html

Basically PRNGs often gets re-seeded periodically, preventing a leak/compromise of one seed to lead to a long-lived compromise. (This is a property called backward secrecy or post-compromise security)

For comparison, here‘s a sane interface: https://fuchsia.dev/fuchsia-src/reference/syscalls/cprng_dra...

How is this different from read(open("/dev/random", O_RDONLY), buffer, sizeof(buffer))?

- It won't fail if there's no /dev/random device mounted (e.g. in chroot)

- It won't fail if there are no file descriptors available

- No error handling needed: the call always succeeds and random bytes are always returned

So like the standard arc4random_buf(3) which everybody has except for Linux.

Don't worry the name is historic it does not us the arc4 cipher any longer, at least on OpenBSD. They switched to chacha20 as stream cipher.

The interface is the same, but arc4random_buf is a user-space CSPRNG.

The analog of zx_cprng_draw is getentropy(2): http://man.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man2/... except zx_cprng_draw kills the process, while getentropy returns an error.

Linux has had getrandom(2) for quite a few years, and there are glibc wrappers which use it.

> Or not. Linux 5.3 will turn back getrandom() into /dev/urandom with its never-blocking behavior, ...

This can be disabled by passing the

  random.getrandom_block=on

parameter to the kernel.

---

EDIT: It appears that this functionality may have not actually made it into the kernel (see other comments) so this parameter would have no effect.

> The whole premise of entropy depletion is that cryptography does not work (the CSRNG does not prevent the leak),...

Nice post but had to stop there. I don't get why some cryptologists don't grasp redundancy. In systems design this is a very sane assumption, in other words The premise is not that "cryptography does not work" but rather "one piece of a cryptography systema may have been broken or weakened".

You don't tell a system engineer "The whole premise of HA is that servers fail" right? Are elementary components of a crypto system expected to be unbreakable or uncompromisable? I mean to me this makes me want to ask the writer of this post: I thought security of CSRNG is estimated to be infeasible to crack based on existing computational resources (brute force) and existing research to break and weaken the RNG ? If so, how is it improbable enough for a bug or backdoor to exist in the CSRNG or for some algorithmic breakthrough with PQC?

In my opinion, it is entirely possible(as history has proven) for elementary components of a crypto system to be weakened and a general purpose OS should presume some level of redundancy where possible. I mean, most users should not be concerned of such rare attacks(except maybe they should,given how billions of devices depend on Linux's rng and there are many extremely resourced attackers that would be interested in dragnet attacks) but you have to also keep in mind there are people who are of enough high value that they will be targeted individually. And I can envision not just the NSA and other spy agencies but plenty of private organizations and exploit brokers that will capitalize big time on discovery of csrng flaws that are not reported for a bug fix or publicized.

I think urandom is fine for most cases but on rare situations like pgp keys, TLS certs, disk encryption keys,etc... The paranoia of /dev/random might come in handy at least for a handful of people somewhere.

Just use rdrand. If your cpu is compromised at the level of individual hardware instructions, you are likely toast anyway, and all your extra steps are just false reassurances.

This article seems to continually conflate "sources of randomness" with pRNGs. The linux entropy pool is not merely some estimation of randomness by the kernel -- it's a collection of bits accumulated from a variety of sources (including rdrand, btw) -- which are munged in such a way that any single bit of input causes a cascade in the states of all output bits -- then fed into a pRNG. Not trusting rdrand (or any other single source, including input devices, internal sensors, IRQ timings, internal clock, etc.) is a feature and does not somehow make the degree of entropy weaker. Having a broad collection of entropy sources feeding into your entropy pool simultaneously helps make the random device more secure by ensuring that if an attacker gains access to any number of these inputs, they still cannot predict outputs as long as they lack even a single one.

This is exactly the article I would have published if I were a state attacker trying to make it easier for myself to create side-channel vulnerabilities in applications relying on randomness.

If tpornin's primary concern is blocking IO (and not unpredictable, cryptographically-secure RNG) then his or her advice makes sense. However, if secure randomness is a concern, then the advice should simply be to encourage application developers to use the minimum amount of entropy needed from the entropy pool. Many, if not most applications are guilty of consuming far more randomness than they truly require.

Reply on news.ycombinator.com