“RNG broken for last 4 months”
lists.freebsd.org
lists.freebsd.org
They are heavily involved in FreeBSD development and I'm sure they're keeping on top of developments in -CURRENT, but I believe they are not using affected versions anywhere in production.
https://people.freebsd.org/~scottl/Netflix-BSDCan-20130515.p...
Unless Netflix enjoys filesystem wipes, file corruptions, and extremely reduced performance due to kernel level debugging flags that are enabled by default (and often can't be disabled due to ongoing development efforts), then they are not running -CURRENT. Remember, -CURRENT runs slowly because of debugging code!!
Honestly, this is probably some kind of confusion over a company or two that has -CURRENT as an upstream to their own fork of the OS because they are reliant on some kind of new changes in the system that aren't in -STABLE.
Even if that's the case, you'd hope they'd be taking extra special precautions considering what they are doing.
Because that's how you read in your comment.
The linux development kernel is called "mainline" Would you want them to change it to Nightly or Alpha?
A lot of computer people are indeed that way. But many others are not.
Invariably, when something goes wrong, the group that doesn't read the manual visits forums or chats to seek help from the group that does.
It is still the responsibility of the user to educate themselves on the things they use; but it's also the responsibility of the designer to make things straightforward and intuitive. I'm okay with blaming both of them, but don't pretend that the designer has no responsibilities here.
And FreeBSD should name things better. Those two things are not mutually exclusive.
There is RELEASE, CURRENT, STABLE. If you choose one without knowing the difference as to what those titles mean, you deserve whatever comes next.
I suspect much of this goes back to when development wasn't quite so public, so these are names that are meaningful to the developers, not the public.
Really, I don't get this attitude of 'if they can't be bothered to read the manual they get what they deserve'. It's everybody's loss to have more compromised machines on the net because compromised machines allow bad elements to gain a foothold and from there it can get much worse in a hurry.
So you treat this as a communications problem and reduce the potential of error by properly labelling your releases (this costs $0) instead of telling those that got bitten by it that they 'got what they deserve'.
Whether you have respect for them or not doesn't enter into it, it's a security issue, not a respect issue.
I don't know how suggestive it is to someone who doesn't use FreeBSD, but if you as much as download and install it, there's no way you don't stumble into at least one "-CURRENT is not what you want for production" fine print.
This isn't a communication, security or respect issue, it's a competence issue. You don't just install an OS on a server and "somehow" not know you're installing a development version!
One of the lines from those books that stands out very clearly 25 years after reading them last: "If a variable is tallying the number of horses name it 'nhorses' not 'qdogs'".
Names matter. So if 'CURRENT' means 'MAYCRASH' name it so.
There is literally (and I'm using the word in its proper sense, not as a hyperbola) no way you end up with FreeBSD on a server without knowing this. No one just installed it "by mistake" in a production environment or mistook it for a STABLE version.
I don't get this attitude of people who think projects should fit their model of how things should be named. Many of us have no problem understanding the labelling, its really fairly intuitive. We also have no problem understanding alternative schemes used in other projects. Just because its different to what you're used to doesn't make it wrong.
Whether or not I have respect for them matter quite a lot if they're trying to market security related products. Basing themselves on -CURRENT and not understanding the potential consequences speaks volumes about that organisation or teams competency in their chosen field.
Don't blame the user for upstream's mistake.
Do FreeBSD developers (I mean, the ones who had nothing to do with this but also use the dev branch) "deserve" it, too?
In general, do you always blame the victim for someone else's mistakes?
edit: I am not a BSD user. I am reading now that the dev branch is purposely crippled anyway and known to be terribly unstable. If that's the case, maybe it is idiotic for non-developers to use it. I'll leave this comment here instead of deleting in so nobody else says the same thing. In the meantime, I'm glad to be using a rolling release Linux distro that always has bleeding-edge software without ever having any problems.
Did you seriously just suggest that Arch never has any issues? How did you manage to achieve that, because I gave up on Arch due to it breaking at least once a fortnight. The "bleeding edge" is called that for a reason: you're likely to bleed when using it.
One big difference is I don't run a desktop environment (kde, gnome, or whatever crap is out there now). I suspect that makes things a lot less likely to break when I do updates.
The closest linux analogy is Linus's unstable git branch or maybe the redhat rawhide https://fedoraproject.org/wiki/Releases/Rawhide - even that has warnings:
from the Rawhide wiki: "Not recommended for production systems
We do not recommend that you run Rawhide as your primary production operating system. Instead, we suggest you could install and run Rawhide:
As a live environment only
In a virtual machine (VM) instance
On a secondary system
On a multiboot system, alongside a stable release of Fedora or another operating system
-CURRENT is the freebsd version of that: you know that it's going to be where things are being torn out and put back in on a hour to hour basis.
The group of people using -CURRENT is akin to the group of kernel maintainers in linux-land.
I do admit though - the rawhide warning is more verbose and clearer than the FREEBSD-CURRENT warnings. That's one part that I can see the community improving. I might even write some verbiage myself for contribution.
Yeah, and that's why you should use it, to be sure your app works ok with the upcoming changes to the core OS.
https://www.freebsd.org/releng/#freeze
and
https://www.freebsd.org/relnotes/CURRENT/relnotes/article.ht...
and
https://wiki.freebsd.org/WhatsNew/FreeBSD11
and be subscribed to the -CURRENT mailing list - which the development documentation indicate as mandatory when using a -CURRENT build.
Then you would know whether the compiler suite is being torn out (remember? FreeBSD switched from GCC to LLVM/Clang - yes - that was done in the last -CURRENT development cycle)
So your software might not even build because of certain base OS issues. BUT you would know all that already - and you would run -CURRENT in a VM that you blow away and rebuild as needed.
Or in continuous integration like jenkins. https://wiki.freebsd.org/201405DevSummit/Jenkins
BUT NO - you would use it in testing only. You wouldn't use it as a main platform.
Unless you don't really care - then it's on you.
Many devs run 10.1 and then run -CURRENT in VM's or on a spare laptop or machine for this purpose.
Because it's the dev branch and they could have any number of good reasons for disabling, removing, or temporarily breaking the RNG while they're rewriting or refactoring it.
It's very hard to have sympathy for people who may be using this in production. You have to go out of your way to get -CURRENT, and the download page clearly says it's aimed at "developers and bleeding-edge testers only." That's why the article is a message on the mailing list and not a security advisory somewhere.
By definition, the dev branch of any software project is unstable and not suitable for production.
> In the meantime, I'm glad to be using a rolling release Linux distro that always has bleeding-edge software without ever having any problems.
I think any long time user of Debian's unstable branch (almost equivalent to -CURRENT) would disagree with that statement.
Yeah, because it could totally not slip through to a release... Like, you know, Heartbleed et al...
Living with broken software isn't a compromise a lot of users make.
If they're confused on that, they probably have other isses to worry about.
I'm more inclined to believe that there major companies are running on -STABLE which is still somewhat risky, but at least typically everything that is there was somewhat tested.
Ouch.
http://fxr.watson.org/fxr/source/dev/random/randomdev.c?im=3...
In short, I think it's a bit worse if you're actually vulnerable, but vulnerable systems will be much more rare.
For some, it's a gleeful traipse into the 5m deep concrete pour, briefly regretted. I salute those brave souls! Why they do what they do may never be known, even to those who partake, but they do it with conviction! And, it's a damn fine thing, if you ask me!
Security requires a threat model. Something can only be "the worst possible vulnerability" if it causes a large amount of damage consistent via the threats which are in the model. The threats that you're thinking of are not usually contained within the models of cutting-edge-dev-systems, which will be protected by firewalls etc. on intranets.
There's really a need in both cases for "an active attacker to intersect with the vulnerable host while it's still there." A broken RNG in such circumstances is much, much weaker than kernel RCE.
This is indeed a serious issue, and I'm disappointed it was in the source tree for four months. But -CURRENT is the development branch, and is not supported by the FreeBSD security team, or supported for use in production.
From https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/c...:
FreeBSD-CURRENT is made available for three primary interest groups:
Members of the FreeBSD community who are actively working on some part of the source tree.
Members of the FreeBSD community who are active testers. They are willing to spend time solving problems, making topical suggestions on changes and the general direction of FreeBSD, and submitting patches.
Users who wish to keep an eye on things, use the current source for reference purposes, or make the occasional comment or code contribution.
FreeBSD-CURRENT should not be considered a fast-track to getting new features before the next release as pre-release features are not yet fully tested and most likely contain bugs. It is not a quick way of getting bug fixes as any given commit is just as likely to introduce new bugs as to fix existing ones. FreeBSD-CURRENT is not in any way “officially supported”.
I can understand the sentiment that you should be prepared for even horrible bugs like this if you're running -CURRENT, but "if you can deal with crashing, you can deal with this" is completely wrong. Instability is a completely different beast from potentially generating easily crackable crypto keys.
I'm not sure what you're trying to argue for/against unless you are looking at things from a pure number theory/computer science research point of view? What would people do then? Create perfect flawless code that never malfunctions even as they write it? Code that never has bugs does not exist.
Perfect algorithms don't exist either.
-CURRENT is exactly the place for this because RNG bugs exist in reality.
If you read emaste's post above, this bug "potentially generates easily crackable crypto keys" for the safe to a monopoly game that holds monopoly money.
It's a development kernel.
Normally you could use a system like this on a trial basis and keep the results. For example, you might run some image processing on it. You'd want to make sure the output was good, but having done so you could then use those images for real-world tasks without worry.
But replace "image processing" with "generate a private key" and now you're deeply screwed.
Well, now we know. Don't generate crypto keys on pre-release OSes that you're going to use for anything important. That's reasonable. But I don't think many people would have thought that way before.
And of course pre-release OSes aren't the only place the danger lies. Debian had a similar problem that made it to releases and stayed undetected for a couple of years. The point is just that "silently break all your crypto" is a much worse vulnerability than mere crashes, or even corruption.
My point is that -CURRENT may be rather unstable at any given time, and is explicitly not supported. One should not be using it a production deployment where sensitive key material may exist.
In that regard the 'crashability' of the branch is indeed more significant than a random-numbers exploit: the crashability of the branch is a mnemonic for its general permissiveness towards unaudited (or not-sufficiently-audited) code. When you remember that your adversaries can inject code into the OS, that changes your threat modeling.
Of course, as you say, not-crashing back-doors can make it into releases, too. Open source's generic promise that more eyes will make bugs shallow is not a guarantee.
What we know about every system that installed FreeBSD-CURRENT is that the systems administrators at the time fully accepted an operating system:
1. That is not in any way "officially supported". (FreeBSD's words, not mine.)
2. that may for short periods of time "not be buildable."
3. that "is not a quick way of getting bug fixes as any given commit is just as likely to introduce new bugs as to fix existing ones".
4. that is much weaker in guarantees than the FreeBSD-STABLE branch, which expressly disclaims, "one should not blindly track FreeBSD-STABLE. It is particularly important not to update any production servers to FreeBSD-STABLE without thoroughly testing the code in a development or testing environment."
If someone has signed off on these topics, then there is no such thing as "game over". The server isn't important enough for "game over". If it is, then the security vulnerability was not the broken RNG but tracking FreeBSD-CURRENT in the first place.
I say it's a game-ON plan because thank goodness it got caught in -CURRENT now - that's the way the development process is supposed to work.
I myself run -CURRENT 2 ways - one is a sandbox development physical box that has no access to any of my other servers (I use it to try ports to see if they will still work on -CURRENT), and the 2nd way is in a VM on my laptop to fire it up to see if it boots from time to time.
BTW, links to the referenced revisions:
How do you reach this conclusion? If nobody runs -current, then the damage associated with a -current RCE must also be scaled down by a similar factor, no?
On the other hand, a -current test system which later gets upgraded to the next release for production will still have busted keys on it, but would have had the RCE fixed.
I guess if my house is burglered because I left the door unlocked, you'd say I'm blameless as long as my "threat model" assumed that any burglers always come in through the window.
What you're trying to point out is that sometimes the real threat model (i.e. the security concerns list of the stakeholders) is not the same as the explicit modeling they do of threats. And that's actually very common: HTTPS, for example, is based on a model of eavesdropping threats which only applies to a small case of real-world security problems (public WiFi), models a lot of threats which haven't been so important (ISP malfeasance, DNS hijacking) and the resulting certificate system may be worse for some bigger matters of network security (protecting from government eavesdropping, for example) when compared to something like SSH.
But HTTPS is a good security standard for what it does, and we don't claim that HTTPS sucks just because a lot of people have malware on their PCs which can trivially transmit their traffic to untrusted third parties. Rather, it's your responsibility to revise your explicit threat models to accord with your own implicit expectations. Your clients' students are suddenly upset that your client was storing Social Security Numbers on your servers, which could be read by the public via your API? You have hereby discovered that your "explicit" threat model, which left the front door unlocked, was failing to live up to an "implicit" threat model which you had. You now need to either punt the problem off to the client ("we're not storing your sensitive info, our service is not appropriate for that!") or update your explicit model ("we're re-securing our systems and assuming that everything is potentially sensitive, so new permissions systems will guard access").
Running and continuing to run -CURRENT when you know that occasionally the front door's locks can fall off for any or no reason, means that you've accepted this reality.
This article mentions DSA private keys and poor RNGs: http://rdist.root.org/2010/11/19/dsa-requirements-for-random...
They found a bug, and it will be fixed. The fact that this bug was found in -CURRENT is a good a thing. This is a pre-alpha, not for production release. The development model is working as intended, no?
1. STABLE prod fleet
2. CURRENT dev workstation
3. TLS certificate private keys in prod generated on CURRENT workstation to be signed by authority
4. Potentially vulnerable TLS keys existing in STABLE fleet
You folks can twist this to try to deflect severity by pointing to CURRENT, but in the real world, it existing at all for four months is extremely serious and must be taken seriously. My production TLS keys were created on my OS X laptop, because I don't often have to think about a compromised random number generator and this is a tradeoff I make in my own life.
I guess if I truly cared about my TLS keys, I should have made them on a copy of airgapped Warty Warthog and run them through a shitload of random analysis tools before shipping them off to be signed. My bad.
Or a tested, non-alpha, security team verified edition of FreeBSD?
I mean come on, We are not saying that you should only generate keys from cosmic rays and personal messages from $DEITY. Only that you do it on a production ready OS, is that really too much to expect?
On a side-note. Unless your devs are involved in the development of FreeBSD, why are they on -CURRENT?
Unless they recompile world all the time (a nontrivial amount of work and a waste of time if you are not actively developing the OS itself) - CURRENT is actually much much slower because of debugging flags that are enabled by default. Some of these flags make the kernel panic upon certain types of errors and drop into kernel debugging mode allowing the examination of kernel dumps. I can't fathom why anyone would generate real security certificates in this kind of environment.
You can recompile with the WITNESS (kernel lock counting & validation - incurs performance loss but useful for counting kernel data structures) and INVARIANTS (run-time assertion checks and tests for kernel data structures) options off, but then you would have to spend time recompiling the kernel instead of working on your software.
https://www.freebsd.org/doc/en/books/developers-handbook/ker...
Note that CURRENT is packaged as snapshot binaries without binary upgrade capability - which means that once you install "one day's" -CURRENT the only 2 ways to upgrade are a) download the new iso and reinstall the entire OS or b) check out the freebsd source code via svn and recompile the system (often world changes as well so you need to do both kernel + userland).
If 2. is actually true, then the person you know who is doing that, unless they are being paid to work on freebsd features or perhaps validating and porting freebsd to a different hardware platform like ARM or PS4 or an embedded device like a medical imaging device or a router (ie Junos), they are wasting a lot of their employer's time and foot-gunning themselves massively in security.
If you show them this post, ask them what they are doing with -CURRENT?
I am open to learning new things so I'd really like to know!
Thinking about this again - yah - the poster might be referring to some kind of workflow like this:
production machines are platforms that run -STABLE
there is some kind of device, embedded or otherwise, that they keep locked up somewhere in a lab. It could be the new xxyz multicore switching fabric [imaginary name] that is running a new version of BSD-OS variant that's undergoing verification testing - they run hardware development and need -CURRENT's capabilities for debugging the system itself. It generates some keys that will be distributed into the pool of machines running -STABLE so that in the future, when this new variant comes into the market, there will be "pre-seeded" keys for compatability (ie. older versions of systems will be able to interact via signed certs with the new system)
Since FreeBSD is BSD licensed, there can be any number of things people are doing with it without anyone else knowing - so maybe to give the benefit of the doubt, I can envision a workflow that needs -CURRENT as a workstation / dev platform.
I think one weakness to my thinking is that VARIANTS/WITNESS/kdb/ddb can be enabled on -RELEASE and -STABLE distributions as well! Why not just do everything on -STABLE even if you need kernel debugging?
If they need -CURRENT for new hardware support, it shouldn't be too hard to figure out from the svn log and the rolling release notes. It's kinda fun trying to reverse engineer the job of the thread parent's acquaintance!
So it's not a trivial problem, because (among other reasons) nondeterministic, statistical testing is not well-understood in the testing culture.
It's also a good lead for a further search query on the topic. Almost all test sets include a birthday test ;) For example this page: https://sites.google.com/site/astudyofentropy/background-inf...
[1]: dieharder: http://www.phy.duke.edu/~rgb/General/dieharder.php
[2]: http://csrc.nist.gov/publications/nistpubs/800-22-rev1a/SP80...
It can't be beyond the wit of man to add such tests to automatic build and regression tests if the project has such a process in place, though it would potentially slow that process down depending on how you define the "acceptable degree" and therefore how aggressively you test.
With knowledge of the algorithm, you can do a lot more.
On the other hand, a cryptographic adversary actively seeks to break the RNG, which may include tricks that statistical tests simply do not account for. As an example, the Mersenne Twister passes many statistical tests, but after observing 630 or so outputs, an intelligent adversary (with some math) can predict all future outputs of the RNG. That is not something a simple test can uncover.
So, essentially, if your RNG fails statistical tests, it is totally unworthy of any consideration at all from a cryptographic standpoint. If it passes all the selected statistical tests, good for it---all that means is that it might not be totally broken.
That isn't to say that statistical tests are without value. If they were in the build checking process, they could spot when the RNG has failed catastrophically, sometimes. They could not spot when cryptographic problems arise, though.
But we can of course talk about a degree of randomness, like we can also try to approach KC. Good randomness is all about unpredictability (given the first half of a random string, can you use that to predict the second half?), but that does not mean that proper randomness should be void of identifiable patterns (such as "00000000111111111" in a random binary string). Such orderly-looking patterns do appear in proper randomness, because the absence of those patterns would make the randomness more predictable, not less.
You can measure level of randomness with statistical methods [1], compression [2], visual methods [3] and die-hard tests [4]
[1] A simple method is the chi-square test.
[2] Compression ratio tells us something about the randomness. Random data can not be compressed by everyday-use compressors. That no one claimed the money for the challenge to compress RAND's digits in a binary file tells us something about the rigor that team had in coming up with random numbers. The more you can compress a string, the more order it contains and the more predictable it is.
[3] You can plot random points inside a circle. After a lot of points are added, you should see no patterns and a properly, evenly spaced circle. Another method are Moiré patterns: Take a field of random noise, copy it, slightly rotate it, and overlay. Non-random patterns will become more visible. But these patterns are visible without Moiré rotation too when using very basic PRNG's like the standard Python random library.
[4] The programmer's way of brute-forcing a lot of simulation runs to see if the PRNG works as expected: http://en.wikipedia.org/wiki/Diehard_tests
Something like 'DEVELOPMENT' or 'NOTFORPROD' instead of 'CURRENT'?
Additionally, it's stamped everywhere [1], if one cares to look, that CURRENT is unsupported, bleeding edge, buggy, "will not build sometimes", etc. Do we really want to modify a development process that has been in use for over a decade because of a few clueless/extremely-gifted people?
Another possibility is that this person is a FreeBSD jedi, well aware of the risks and payoffs. In that case, this bug is no surprise and she/he is prepared to act on it and regenerate some keys, review commit logs (if a developer), and look for signs of intrusion etc.
I still think this is a "non-news" and poor attempt to use HN to spread fear and trigger useless discussions. The usual drama show.
Do you want linux to change to "DEVELOPMENT" or "NOT FOR PROD" instead of "mainline"?
It's a matter of staying literate and actually reading what the docs say.
"This does not effect programs that directly used /dev/random or /dev/urandom."
openssl should use /dev/random for key generation and keys generated by openssl is not affected?
Name?
At around 23:50 in the video, Rick Reed talks about slide #17, they say they are running 9.1 - 9.3, and looking at 10.1 (not in a hurry as 9 works for them).
[1] https://www.youtube.com/watch?v=TneLO5TdW_M
[2] http://www.slideshare.net/iXsystems/rick-reed-600-m-unsuspec...
All of those freebsd versions you refer to (9, 10) are part of the production release cycles and thus are not -CURRENT.
If whatsapp uses 10.1, then they are NOT using -CURRENT
Theoretically, WhatsApp could generate the keys and forward them to clients, and you could still plausibly call it "End to End"