OpenSSH taking minutes to become available, booting takes half an hour (2018)
daniel-lange.com
daniel-lange.com
Linux has an opportunity to learn from OpenBSD.
https://www.openbsd.org/papers/hackfest2014-arc4random/index...
OpenBSD also controls its own boot loader, which also feeds in entropy that the kernel then leverages. Perhaps GRUB(2) should do something similar?
Seems like the man page sort of notes the problem in the article:
> Writing to /dev/random or /dev/urandom will update the entropy pool with the data written, but this will not result in a higher entropy count.
But, why not, man page, why not? I feel like if I, as root, write to /dev/random, I am saying that this is acceptable input for seeding the generator, and that it is on me to not, say, seed it with the exact same stuff every boot, or seed it with straight nuls, no?
How does OpenBSD guarantee that this file exists, though? Perhaps making it a system default is wise, but what about brand new VMs/installs? Does the install just take the time to do the first generation of that file?
The OpenBSD folks also appear to trust RDRANG, which would help not blocking. But I disagree with that trust decision.
[1]: start reading here: https://www.openbsd.org/papers/hackfest2014-arc4random/mgp00...
It feeds the entropy pool, mixed in with many sources. OpenBSD also feeds in data from the AMD CCP as well. The point is it doesn't matter.
And anyone running "echo badapples > /dev/random" in a loop, it still doesn't matter.
https://man.openbsd.org/random.4
https://man.openbsd.org/arc4random.9
> How does OpenBSD guarantee that this file exists, though?
If you keep reading, OpenBSD's random subsystem always mixes old with new data from many sources. If the random.seed file is not available (for example, boot media), it is not fatal. There is a warning displayed. But the system rc(8) scripts handle creating the seed file at both boot and shutdown.
https://www.openbsd.org/papers/hackfest2014-arc4random/mgp00...
Sure. But say we don't trust RDRAND. It's the first boot, so you haven't got a seed file yet. Assuming you haven't got a HW generator at hand, what other source are there that don't take significant time to collect from? (I.e., I know you can get it from network timings, keyboard/mouse if they exist, etc., but that takes time, and would cause a call not wanting to return prior to there being sufficient data to block. If you're only mixing in unready or untrusted sources, you might get lucky and make it hard for an attacker, or worse, if your other sources are blocking b/c they're collecting data… you might not, but regardless it hardly seems principled?)
It'd be willing to write off the first boot, except that I feel like a lot of things do potentially get initialized then; e.g., in a VM in the cloud, I think that's when host keys are initialized, for example, and certainly whatever application that VM might be hosting could have further requirements. Not blocking and issuing a warning would be to supply data prior to having sufficiently initialized a generator, no?
Why not? The install media should be collecting entropy as it does the install, xoring it with a pool of random data built into it, and writing that to disk as part of the install.
Take a look at https://github.com/openbsd/src/blob/master/distrib/miniroot/..., specifically, feed_random and store_random
What about virtual machines / machines booting off an image? We can't put a seed in the image, or it'll get distributed to all downstream consumers.
Also, there's the case posed in response to my first comment, about IOT devices which would ship with some factory-installed image.
(I suppose there are some novel ways one could work around this such as somehow keeping a pool of one-shot images ready w/ just the seed added to them, but I don't feel like this is how real-world systems work. E.g., an AMI in AWS?)
Mitigations within mitigations.
Having one random seed per VM image that gets installed is better than having nothing at all -- now, an attacker needs to have your install image. Then the hardware RNG and the virtio random drivers help mix into that seed.
Given that presumably you've already booted the image for testing, you've probably already generated the SSH keys that you need to be concerned about -- so you'd probably need to take care to regenerate them securely in any case, from a running system that has started to gather entropy from all the sources it can get its hands on, including the boot loader.
And, ideally part of the imaging process would write some random data to the image. But I agree, most people wouldn't even think of doing that.
and during first reboot you image the disk and flash it into millions of routers, resulting in something like https://devcraft.io/posts/2017/07/21/tp-link-archer-c9-admin...
But, at some point nobody can save you from yourself entirely. Get a broken enough system, and every security feature will fail.
I'm assuming a bootloader can't collect more in three seconds than an OS can collect in multiple minutes. Is that assumption wrong?
If it's the design of the code that lets it collect entropy faster, then it's that code that needs to be ported, and it doesn't matter if it's in a bootloader or not except to save a second or two.
There is a way for a root process, like systemd, to contribute trusted entropy to the entropy pool, using an ioctl. Of course, then it is systemd which will be taking on the responsibility if it turns out that /var/lib/random_seed turns out to be replicated onto a zillion static images on some IOT device, so it turns out /var/lib/random_seed wasn't actually all that random to begin with
Mixing more data into a random source cannot make it worse. OpenBSD does not use rdrand alone.
"Basically as of now the entropy file saved as /var/lib/systemd/random-seed will not - drumroll - add entropy to the random pool when played back during boot. Actually it will. It will just not be accounted for. So Linux doesn't know. And continues blocking getrandom(). This is obviously different from SysVinit times2 when /var/lib/urandom/random-seed (that you still have lying around on updated systems) made sure the system carried enough entropy over reboot to continue working right after enough of the system was booted."
Thing was you need the Haveged daemon to generate entropy, something that should come within the kernel itself.
edit: but i guess that's on purpose to handle disk imaging
Per some of the comments in the Debian bug(s): if you tell users and devs simply "you deal with it", you will get a multitude of ad hoc, half-baked solutions that may nor may not be secure. You will have dozens of people trying to re-invent the wheel.
The whole point of things like /dev/urandom and getentropy() is that the problem is solved once properly, and then you allow everyone to benefit. By having the above solutions not-work, we are basically going back to the days of where they did not exist--in which case what was the point of creating them?
The BSDs seem to not have a problem with this, so I have no idea why the Linux folks can't seem to get their act together.
> Key generation is already slow anyway.
Define "slow". Key generation was slow in the early 1990s when I first started using PGP (nee GPG) back in the day. It is generally not-slow nowadays IMHO--as in, it takes less than ten minutes to find p and q for RSA 1024 (nevermind 2048+).
To a first approximation:
* SHA256( cat /var/lib/randomseed || ifconfig || date -u ) > /dev/urandom
will get any decent stream-cipher-based PRNG going.
Each machine will have a unique MAC address, so that's 48 bits right off the bat, even if randomseed is identical and every machine is booted at the exact same second.
If the system is not communicating with other systems... what attack tree are you actually worried about if it is inaccessible?
I would also be curious to know which virtualization environments generate duplicate MAC addresses over (say) dozens of systems?
qemu uses the same MAC address by default.
Are you telling me there is not even, say, 8 bits' worth of entropy credit in there? (Especially when hashed with other data?)
Remember: the context of this discussion is VMs (and not embedded systems without a battery-backed RTC).
Persistent state is hard. For example, some systems run with read-only filesystem.
Edit: Replaced "host keys" with "cryptographic state".
https://fedoraproject.org/wiki/Features/Virtio_RNG
For embedded systems, many have inbuilt RNGs, have a look at your SoC docs.
Quality entropy isn't a hard ask, at least not for anything typically running OpenSSH or other server software. Intel has RDRAND, and even where AMD's RDRAND is broken their PSP coprocessors provide an entropy function. Similarly, NICs and other controllers also often come with RNGs. RNGs abound on modern embedded systems, actually, it's just that nobody has the full-time job of plugging them into the kernel's PRNG pool. It's not a full-time job for any OpenBSD developer, either, but they do seem to do a better job of this than on Linux; often times the only feature supported of a miscellaneous system component is the RNG.
It's the APIs that are broken. getrandom shouldn't block, period. A system only needs 16-32 bytes of pure hardware randomness for strong security. That's it! You either have it shortly upon boot, or you don't. If you don't, you're screwed anyhow, so why block? If you have 32 bytes of good entropy, all the entropy accounting mumbo-jumbo is pointless.
I do take issue with the author's complaint that systemd shouldn't have its own user-space PRNG. BSD systems have arc4random() as part of their libc, which is seeded from the kernel pool. This is very convenient; developers shouldn't have to think twice about calling into a PRNG for a 32-bit number, but if acquiring that requires a syscall they do think twice and often screw things up. Not to mention that Linux getrandom's default block semantics is broken by design.[1] Until glibc, musl libc, and other Linux runtimes wisen up and add arc4random, it's hard to blame projects like systemd for including their own PRNG.
[1] I realize that getrandom has an option to not block, but it's only function is to cast suspicion on itself when in fact the only thing that deserves suspicion are entropy guesstimators.
128 bits is enough to feed into a stream cipher that will generate a lot of random bits:
> The original version of this random number generator used the RC4 (also known as ARC4) algorithm. In OpenBSD 5.5 it was replaced with the ChaCha20 cipher, and it may be replaced again in the future as cryptographic techniques advance.
* https://man.openbsd.org/arc4random.3
* https://security.stackexchange.com/questions/85601/
Re-feed/-stir every so often so that past entropy state can't be used to compromise things in the future.
Correct.
> You either have it shortly upon boot, or you don't.
Not correct. On virtualized systems, with minimal interrupts, there is frequently not enough entropy available. This comment seems to reflect a poor understanding of the getrandom(2) system call and its history; part of the reason for implementing getrandom was in order to provide this new behavior of no longer blocking once enough entropy has been collected for a secure CSPRNG initialization. Linux needs this "entropy accounting mumbo jumbo" because it doesn't have a standard mechanism for persisting random seeds across boots; OpenBSD doesn't need it because the bootloader and kernel are tightly integrated, so they can easily implement this feature.
Entropy accounting can't fix the underling problems here, all they do is obscure and confuse. Hypothetically and in a very technical sense, they can be useful and even necessary. But in practice they simply have no place outside of the actual hardware-based entropy generating devices. If you can't quickly seed yourself with 32[1] bytes of cryptographically strong entropy, then you're screwed, period. Blocking doesn't improve security, it just induces people and developers to implement awkward workarounds with the net effect of drastically reducing security.
When Linux added the getrandom syscall they should have dropped support for blocking. It was patterned after OpenBSD getentropy, which doesn't block; neither did Linux' long deprecated sysctl() random UUID mechanism that many programs once relied upon (like Tor). But they, and Ts'o in particular, seem unable to resist the siren call of entropy guesstimation.
[1] Even 16 bytes is enough to seed the system pool for an indefinite period, at least relative to a system without a strong hardware RNG. And that's the point. There are reasons for why a component might need ongoing sources of strong entropy, but if a system can't even provide 16-32 bytes at boot then those arguments are purely hypothetical because there clearly aren't sources of strong entropy available, anyhow. But if those sources are available, then it's ridiculous to think they can be "depleted" as a practical matter. If the CSPRNG pooling functions are broken, then all modern cryptography is broken, so you gain nothing with the convoluted semantics of Linux's traditional /dev/random machinations.
I'm not a fan of trusting closed hardware for randomness.
Fortunately in practice the situation isn't so dire. There are usually multiple hardware RNGs on a system. They may be all untrusted, but I'll trust them together before I'll ever trust the strength (and persisting correctness) of guesstimators. Ironically, entropy guesstimators are predicated on the very notion that the hardware is benign. Malicious hardware could implement subtle timing patterns in interrupts, much like what they might do for an actual RNG (e.g. use an AES encryption function to "randomize" their visible behavior). And in any event, you can still mix various system timing events into your pool without pretending you can quantitatively and reliably know their entropic contribution.
For example, the server could have an RSA private key, and the clients have the corresponding public key. A client could generate a full RSA block of random data, and encrypt that with the public key, and send the result to the server. The server could recover the random data and use that to initialize a CSPRNG.
This CSPRNG could be user mode code running in the ssh system, and only used for randomness needed for that particular connection, so that if a client supplies poor random data it only weakens that client's connection.
don't blame the linux kernel, we're just waiting on the dbus interface, gnome3 applet, and systemd binary + unit files before we integrate /etc/grub.conf.d/entropy.d/randomseed.d/grub-entropy-randomseed.conf and it's good buddy systemd-enable-grub-conf-d-entropy-d-randomseed-d.target into the Loobuntitis GNU/Linux Kubernetes CloudPaaSOS 2019.07.18 'doofy nerdguy' LTS Server release.
OpenBSD guys are clearly antiquated and using old modes of thinking with such a simplistic and neanderthal like design of having a system wide library installed in a single location without a dynamic json configurable runtime rest api endpoint.
Read and parse the config, fork, and lazily build an entropy pool.
Let the ssh connections or key generations hang if there isn’t enough entropy yet, but don’t get in the way of booting.
This is two bugs: one in systemd, and one in OpenSSH.
It is very frustrating.
Edit: looks like this is supported to at least back to debian lenny, released in 2009.
... not to mention it is the entire point of /dev/urandom .. if you want something to block for entropy, you use /dev/random. if you are using /dev/urandom, you are saying I want something 'randomly random' and I'm okay if I can't get something random enough..
2. If haveged isn't perfect enough, setup a hw TRNG (i.e., EntropyKey, BitBabbler, InfiniteNoise) and EGD or EntropyBroker.
https://everipedia.org/wiki/lang_en/Comparison_of_hardware_r...
https://www.vanheusden.com/entropybroker
3. It maybe something else.
Is not sufficient by itself to never have any part of the ssh connection and authentication not use DNS.
See https://unix.stackexchange.com/questions/56941/what-is-the-p... and https://serverfault.com/questions/576293/sshd-tries-reverse-...
...tl;dr quick change to Java property so that SecureRandom uses /dev/urandom instead of /dev/random and problem solved.
And of course it's systemd, what else. (not sarcasm)
There are posts as lately as this year.
I can't even count these any more, systemd closed it as "not a bug". I have angry words in my head.
I'd rather not be relying on Intel hardware to do something so important correctly.
In the end it's a defense in depth approach - I simply think the original and default Linux kernel policy of demanding sufficient entropy from other means is preferable from a security perspective.
Theodore Tso has some good posts on this on the kernel mailing lists. Due to the way these sources get mixed, it would only improve security to turn this option off.
Don’t worry about it.