Ruby Bug: SecureRandom should try /dev/urandom first
bugs.ruby-lang.org
bugs.ruby-lang.org
How is one supposed to know who wrote those blog posts? Just because it is linuxexpert.com does not mean they are linux experts. It's funny, if people changed it randomly following blogs then people will claim this is some NSA conspiracy :-) How does one verify the person behind the blogs?
node has similar https://github.com/nodejs/node/issues/5798
I think Ruby devs' position is more than this. By keeping the faulty man page, Linux maintainers are implicitly communicating that they intended `/dev/urandom` to be a limited and less recommended way of doing things. The intention is important: even though `/dev/urandom` is actually better in the current kernel it may not in the future. It is not the only answer, as other languages did another choice, but it is perfectly reasonable to be conservative like this.
We badly need to change that intention, really.
No, that is not a legitimate concern.
Clearly document system behavior or code is essentially useless.
The random char device code isn't that hard to understand, and if you're not a strong C programmer (the Ruby-core people are good C programmers, I suppose) - there's a paper explaining how it works: https://eprint.iacr.org/2012/251.pdf
Aaron
According to this documentation:
http://ruby-doc.org/core-2.1.2/Float.html#method-i-round
How do you explain?
2.1.2 :011 > 15.round(-1) => 20
Basically everything else. If you see how manpages are written, or just how often they are out of date, you quickly arrive at the conclusion that they are not authoritative source at all. Add in the political plays done in this area. Being stubborn over a wrong manpage is very frightening, especially in a security context. This thread shakes my believe in ruby as a language.
But I put all relevant information, academic and engineering-wise in the thread to try to convince Ruby-core to change their opinion. I replied to false assumptions and comments as best as I could.
I'm also only a human and since this bug has been open for 2 years, I've used SecureRandom extensively in the past, this was a very frustrating experience for myself and all the commenters involved. I certainly do not have the most "diplomatic" approach (as a friend put it). I know that. But I'm not really sorry about that either, it's just who I am. I'm a nice guy IRL people tell me, but I can get obnoxious when people don't listen to severe security issues and always refer to upstream, have been so in quite a few projects and standards processes.
I'll work on that, promise ;)
Aaron
I understand that Japanese culture is really different and that as a country they've really had horrible things done to them over the last hundred years that are inexcusable to say the absolute very least.
But just like any country, there are people with great people skills and people with no people skills. The Ruby core team lacks people skills. Whether it is a result of cultural problems that are a result of terrible things happening there, I can't say, but it really does Ruby a disservice.
All they need to do is listen and consider, but they don't, really.
If you look at replies I got from Ruby-core: some people would consider them to be rude as well; I'm constantly told I do not understand what I'm doing, and I've been in engineering for more than 12 years, into crypto for more than five (and been reading cypherpunk lists since I was 15). I'm certainly not an academic cryptographer nor among the best engineers in the field, but I think I know a fair bit about the topic by now. I've contributed to many security projects, academic publications and standards processes -- this was certainly among the worst experiences I've had so far (you'd think IETF is worse, no. heated discussions all the time, but people stay focused and technical, listen to comments made by domain experts et cetera and act on them).
The Ruby community even has their own acronym for being nice to other developers: MINASWAN (https://en.wikipedia.org/wiki/Yukihiro_Matsumoto). I'm puzzled by the outcome of this discussion, but am assured by other security engineers and cryptographers that bugs they opened were treated equally badly, often ignored, even if they were non-disclosed, heavy security issues.
No idea. I'm not part of Ruby-core, neither Japanese. In my travels I've encountered many cultures and peoples, Japanese are amongst the most polite and friendly people I've met. Often very shy in that regard, like many asians (this is indeed a culture thing & certainly not a bad one). Some are xenophobic, but I wouldn't say that they all are, that's just false, I've met so many open-minded Japanese that I'd never generalise in that regard.
Aaron
If you have organization with strict good manners policy and problems arise, this kind of passive-aggressive behavior is exactly what you should expect. "company policy" and PC rules can't make people better persons.
NOTE: This is not my opinion about Ruby-core team. I just want to point out that being polite does not mean that you interact well with others.
"The Ruby community even has their own acronym for being nice to other developers: MINASWAN"
No, that is not true. That was coined, if I remember correctly, by the pickaxe.
Matz is nice, but how does this translate to any other bad suggestion out there? I don't understand that logic.
It's also not as if it is ... impossible to make suggestions to ruby core that are accepted?
Like hundreds other people manage? Why does the dude above fail?
Here is the issue tracker:
https://bugs.ruby-lang.org/projects/ruby-trunk/issues?set_fi...
You'll see a lot of assignee's done to matz, nobu, koichi etc... I mean they don't have 50 arms each and infinite time so they have to prioritize on what they work.
"Some are xenophobic"
That is so totally rubbish.
Just go to the japanese bboy scene. They are not xenophobic AT ALL.
https://www.youtube.com/watch?v=f5Y75Rjl6UU
They are people like YOU AND ME. Assuming that there is a huge, insurmountable cultural difference is just c-r-a-p.
Or do you think that every japanese loves video games? Or loves ninjas and samurai? Or knows karate?
I replied to a post made above, and expressed deepest sympathy for the Japanese people. While I was born in central Europe, I'd rather not be a citizen of any nation. I spend more time abroad than in the country I was born in. Speaking of which: central Europe currently has a huge resurgence of facist ideologies and xenophobia due to migrants from war-torn countries. Something which is utterly inconceivable to most people able to read a history book, given Europe's not so distant past with genocide. Unfortunately, xenophobia and right-wing sentiment is something that every democracy and thus nation faces, Japan isn't exempt from that [0] [1] [2].
(I currently live in Asia and spend a lot of time in Arabic countries, you may reconsider educating me on the subject with references to the bboy scene.)
Aaron
[0] https://en.wikipedia.org/wiki/Category:Far-right_politics_in...
[1] https://en.wikipedia.org/wiki/Political_extremism_in_Japan#R...
Absolute deference to man pages and insistence on getting things fixed upstream are textbook examples of, well, following the textbook. And they won't accept blog posts and presentations as authoritative because they are not familiar with the authors and presenters. Had they been even a casual reader of HN, they wouldn't dismiss names like tptacek so easily. They simply have no idea who the heck he is, so they stick to TFM as they were taught to do.
So it's neither malice nor xenophobia. I think they're just following rules, and maybe a little annoyed that everyone is telling them to ignore their rules.
Also, why are the Ruby devs so dead set on the manual page?
The reason this is a problem is simply that the Linux maintainer is obstinately wrong. Theodore Ts'o was on HN a year or so ago, and in defending the current design (and man pages), made some incoherent arguments, such as that urandom might be OK for nonces, but not key generation (a head-scratcher for anyone familiar with cryptography engineering).
As for "Also, why are the Ruby devs so dead set on the manual page?" - because it makes sense. If I didn't know much about random number generation, I'd rely on manuals and most widely adopted software. That means: urandom says it's not good for that usage, while openssl is used by almost every system. If you used a dependency that documents it does X and someone tells you that actually it does Y, but you have no ability to prove it either way, would you listen to the docs or random-internet-person.
would you listen to the docs or random-internet-person.
Which people? Could you please name some of them? Honest question - the only name I can think of right now is tptacek's, but I'm sure there are more.
We don't expect person writing a Ruby binding for sqlite to know the theory behind database indexes and who are the acknowledged experts in that field. Yet we seem to expect that from a person writing binding for RNGs.
We don't? More to the point, do we expect them to close a bug related to those things without educating themselves on the subject enough to make a rational decision on it?
It's not the best solution, but if I ignored everything I know about this issue, I think it's a reasonable maintainer's approach.
A startup founder often wants to work themselves out of a job to do more useful things, most will do all they can to insulate themselves into indispensability (a lot of technical people are super insecure, I'm proly one of them, and a lot of corporate gigs are cut-throat fiefdoms).
It's reasonable to make this mistake initially sure. But when it's pointed out to you that the documentation is flat out wrong for reasons outside your and others control and you are pointed at articles and research by the experts in this field and you elect to ignore them then the security flaws that will result are very much on you just as much as they are on the linux man page maintainer.
[1] http://sockpuppet.org/blog/2014/02/25/safely-generate-random...
[1] https://lwn.net/Articles/684568/
[2] https://news.ycombinator.com/item?id=11561340 (HN thread for [1])
Sadly they just had to include GRND_RANDOM then compound that with GRND_NONBLOCK.
If you think they should take the word of 'experts', how does one qualify as an expert? How do you verify that the expert is the one who wrote the blog post or mail etc etc.
The process is already there. Get the change for man page in kernel and user space will follow suit.
Yes. Anyone maintaining and distributing the security related code of a language runtime should understand as much as humanly possible about what is actually going on from the application level all the way down to the hardware.
Just because the implementation of /dev/urandom has good properties that make usable as a system-wide CSPRNG, doesn't mean that it's supposed to have those good properties. Only the docs can make that assertion. Without such an assertion, the kernel is free to make /dev/urandom have less-sound properties in a future release—because there's no spec saying they shouldn't. (They try not to break userland code, yes, but userland code only relies on things the docs say it can, so...)
Judging from the odd Torvalds-yelling-at-people mail getting linked on HN, it doesn't sound like "it's ok to break userland here because clearly they didn't read the docs" would fly.
In the mental model in which the /dev/random manpage lives, it makes (some) sense to read /dev/random until you get a short read, then make a userland pool from all the data you've read, "to get as much entropy as possible". That reasoning doesn't actually make sense, of course, but I would be surprised if nobody thought that such a construction would be a good idea.
Separately, there's quite a lot of code - and I've written some - that reads a byte from /dev/random at daemon start, specifically to block until the system has built up enough entropy. FreeBSD blocks reads, even from /dev/urandom, until the [EDIT: estimate of how much entropy has ever gone into the pool] gets high enough, which makes some sense; but there needs to be some way to block-until-random.
I'm fairly sure they would have to change the nature of /dev/urandom in such a way as would invalidate the other parts of the documentation in order to make the erroneous parts of the man page contain valid concerns.
require "securerandom"
module SecureRandom
F = File.open("/dev/urandom")
def self.random_bytes(n = nil)
F.read(n || 16)
end
end def f(n = 16)
n
end
p f # => 16
p f(nil) # => nilSo bug report understates the issue, nobody has gotten around to writing a good quality patch, everyone is just loudly complaining elsewhere.
But you may make a good point that perhaps nobody has submitted a good patch yet -- you're right that issue isn't a good patch, just an invitation to enter into a discussion toward one (an invitation that doesn't seem to have been taken up, at least on the issue tracker).
One would hope that the actual maintainers of the kernel subsystems would feel responsibility to improve a clearly insufficient man page. Which makes me suspect there are some underlying politics or personality conflicts going on. (Or just burn-out?)
But if all it needs is someone to write some good text -- is it really the case that none of the cryptographers who have been writing extensive blog posts about this for years care to submit a doc patch? If so, I wonder why?
Perhaps, just guessing, another potential issue is that nobody really wants to _take responsibility_ for such text, in case they make a mistake. So the existing clearly insufficient text remains. Tragedy of the open source commons?
The report suggests "clarifying" what the man page means by the (completely incorrect) statement that "Users should be very economical in the amount of seed material that they read from /dev/urandom"
Probably the reporter was just being polite, but in the absence of other comments in the bug or any kernel developers weighing in, it just sounds like an editorial suggestion coming from a single Linux user. Remember that the man-pages project is separate from kernel development.
They begin with the most indisputable ways the man page is insufficient, as a way of starting the conversation -- which was never taken up.
Perhaps an actual patch would have been just accepted? Maybe the problem is the issue-submitter assumed there was someone on the other end who understood the kernel features and was interested in discussing making the man pages better.
Come on Ruby, get your act together. You can do better, right?
This is a quote for the ages.
I end up installing haveged just because I don't want the system mysteriously locking up because some random daemon wants to create a 4096 bit key on first startup.
Also RDRAND can be executed in a VM, but I'm not sure if it is handled properly yet.
And like tptacek mentioned - why did you use blocking random in the app?
I don't even think the solution is to point everything to /dev/urandom. Why maintain two? Why constantly have to explain to people the difference? The BSD developers merged both devices into /dev/urandom and I think that's the right approach.
[0] http://www.redhat.com/archives/libvir-list/2016-March/msg010...
The way this currently looks to me, is that it all but ensures poor adoption rates.
bhyve on FreeBSD has virtio_random, and it simply hooks up to /dev/random (which on FreeBSD is nonblocking and can't starve).
https://github.com/technion/use_urandom
Note I said "start", so I'm aware it needs work. Feedback is appreciated however.
Then you don't have to do any work yourself.
def SecureRandom.gen_random(n)
ret = Random.raw_seed(n)
unless ret
raise NotImplementedError, "No random device"
end
unless ret.length == n
raise NotImplementedError, "Unexpected partial read from random device: only #{ret.length} for #{n} bytes"
end
ret
endEdit: or to put another way, first define what is it that linux's getrandom() and /dev/urandom doesn't provide you.
What it points out is that even a good random number generator can be used incorrectly, and without the right tools your efforts to produce truly random numbers are doomed from the start.
C++ has an embarrassing wealth of random number generators. The Ruby core has almost nothing that can measure up to that, yet it seems like a huge oversight.
I like that C++ has a generator for many different use cases, they all have their reason for being there, but Ruby has a singular one with unknown properties. Porting over what C++ has and making a proper Random library for Ruby would make a lot of sense here.
[0] http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt.html
If you're talking about an RNG library for cryptographic purposes, then it depends on your use case.
If you used a CSPRNG with a seed space smaller than the set of possible lottery outcomes, losers could argue (misleadingly, since we still couldn't feasibly bias the result) that not all outcomes were equally probable and try to get the results thrown out. That is, the fact that there are widespread misconceptions about /dev/random can very rarely be a reason to use it :P
However, I agree that the rule is that you should just use /dev/urandom.
But a CSPRNG which you need to explicitly seed with random bits as big as your output isn't providing much value (simply whitening) since generating the seed is the same problem you had before adding the CSPRNG. So you end up looking at a TRNG.
Combining two independent sources obtained by different people and using a cryptographic commitment scheme ensured that 1) no one person could fix the results or make it nonrandom (protection against Eddie Tipton-style attacks), 2) if at least one of the independent sources was random, the result would be.
[1]: http://sockpuppet.org/blog/2014/02/25/safely-generate-random...
(Also what if the linux RNG isn't seeded e.g. in a VM scenario? Or is the answer just "don't do that"?)
--
¹ http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/SFMT/index.h...
The short answer is to just use the OS provided one if available. Linux has /dev/urandom[1] and the getrandom[2] syscall and Windows has RtlGenRandom[3][4] (there's also CryptGenRandom on Windows which is the "official" version but is more awkward to use since it requires a CSP context to use). On Windows this uses AES-256 in CTR mode as specified in NIST 800-90.
Outside of those (if for whatever reason[6] you're not satisfied with just using what the OS provides you) you can also take a look at how BoringSSL does things[5] (and LibreSSL although I'm less familiar with it). It uses a ChaCha20 instance to filter rdrand output (if supported, otherwise it just uses the system CSPRNG directly). The system CSPRNG is used to key the ChaCha20 instance and then for every call to RAND_bytes it uses rdrand output equal to however many bytes you requested and that gets essentially filtered through the ChaCha20 instance. It's fast and pretty simple. I found the code to be pretty easy to understand[5]. I believe LibreSSL does something similar with ChaCha20 although I'm not sure it uses rdrand.
I think there should be a pretty good reason if you're choosing not to use what the OS provides for you.
tl;dr Just use /dev/urandom (or the getrandom syscall) on Linux and RtlGenRandom on Windows.
[1] http://sockpuppet.org/blog/2014/02/25/safely-generate-random...
[2] http://man7.org/linux/man-pages/man2/getrandom.2.html
[3] https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
[4] https://boringssl.googlesource.com/boringssl/+/master/crypto... (RtlGenRandom requires slightly special definitions to work).
[5] https://boringssl.googlesource.com/boringssl/+/master/crypto...
[6] There really isn't a "depending on your use case" decision to make. /dev/urandom (or RtlGenRandom) is the correct choice for all cases.
With that being said there may be two small caviats to that. Whether or not it has to be a CSPRNG and whether or not /dev/urandom (or RtlGenRandom) can provide the throughput which you require. In certain cases you may have to expand /dev/urandom with something like ChaCha20 if /dev/urandom is too slow (the BoringSSL devs mentioned that for AES-CBC IV generation on servers this can sometimes be the case, see [5] for the BoringSSL implementation).
You may be better off just using rdrand directly in this case depending on the throughput required. AFAIK you should be able to saturate all logical threads generating random numbers with rdrand and the DRNG (digital RNG) still won't run out of entropy.
Or as I also suggested look into expanding /dev/urandom output using a ChaCha20 instance like what BoringSSL does (which also combines it with rdrand since it's fast).
Weakness: https://eprint.iacr.org/2006/438 — "huge subsets of internal states which induce a strongly non-uniform distribution in the 8192 first bits produced"
Finally, why is deterministic PRNG suggested as a replacement for OpenSSL's random number generator? In general, the advice to write your own userspace PRNG replacement for OpenSSL is not a good advice, because many people are not competent enough to do it.
I certainly wouldn't go anywhere near another NIST DRBG..
> https://eprint.iacr.org/2006/438
From my brief understanding Aumasson's paper uses a different seeding routine from the example provided in the c implementation which allows the weaker states to be produced - indeed it's mentioned on the author's website.
> Finally, why is deterministic PRNG suggested as a replacement for OpenSSL's random number generator? In general, the advice to write your own userspace PRNG replacement for OpenSSL is not a good advice, because many people are not competent enough to do it.
If you read my above post I clearly do not suggest this.
OK, this is an instant red-flag to me to get out of the conversation.
Perhaps you are unfamiliar with? https://en.wikipedia.org/wiki/Dual_EC_DRBG
Simply replacing the OpenSSL RNG or /dev/urandom in userland code with the ISAAC routines is likely to blow your app up.
Rubinius copies MRI circa 2014 via rubysl - https://github.com/rubysl/rubysl-securerandom
> If you call: new SecureRandom() on Linux and the default values are used, it will read from /dev/urandom and not block. (By default on Solaris, the PKCS11 SecureRandom is used, and also calls into /dev/urandom.)
Of course, you can change a whole heck of a lot of stuff by setting properties on the command line or changing $JAVA_HOME/lib/security/java.security.
http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/s...
The core ruby team is mostly Japanese-speaking, and Japanese developers are less likely to speak English well even than people in a lot of other countries where English is not a common first language. I think that does serve as a barrier to communication and transparency (with/towards English-speakers!). I think that's all there is, nothing "about Asian culture", or "decisions are made as a group rather than on an individual basis," or cliquishness.
We English speakers have the luxury of seeing almost all open source development happen in our first language. Most of the world doesn't. Ruby is an exception, with a mostly Japanese core team. Just how it is.
Here's a useful (to those curious to learn more about ruby committers) interview from 2013 with 15 ruby committers, note the questions and answers have been translated between English and Japanese. http://www.sitepoint.com/meet-fifteen-ruby-core-committers/
My impression is that a lot of discussion on MRI happens in Japanese on Japanese listservs. Of course, it doesn't take a language barrier to make things seem not so transparent -- for comparison a few years ago, Rails core team started taking the majority of their discussion to private core-team-only listservs. (I gather because they were finding dealing with the peanut gallery made it too hard to get things done/decided). At least there's still lots of open (and English-language) communication on the github issues or open listserv, but there are often times when I'm not really sure who or how or on what basis architectural decisions are made for Rails too, a lot is done in private, is my impression.
But I suspect to curious Japanese-speakers, MRI development may even be _more_ transparent than Rails is to English-speakers! I think (but not entirely sure) all of the listservs MRI devs use to collaborate are actually open -- you just have to read/write Japanese to understand what's there or engage effectively. :)