Is OpenSMTPD worthy of OpenBSD inclusion?
thread.gmane.org
thread.gmane.org
(Edit: Corrected by vezzy-fnord, started in 2009.)
The only currently-extant memory-unsafe language.
I... just... argh... when we will finally be rid of this albatross? What hope is there for the world of computer security if we can't even greenfield a solution to what has already proved to be one of the most dangerous security domains there is (thanks, sendmail) in something other than C? Why are the words "buffer overflow" appearing in relation to such young code? What am I doing with my life? Why am I even trying to write secure code in a world where this is even remotely considered an option?
Some variants of Forth?
OpenSMTPD actually dates back earlier to around 2009, when it was introduced in 4.6.
Not that this is even notable, as virtually all major OpenBSD projects are C-based.
Concurrent Pascal was an interesting project for its time, heavily dated by now. The greatest victory for concurrency was having architectures with thread-level parallelism instead of the dominant instruction-level parallelism, and one specifically optimized for CSP in the occam language. This was, of course, the transputer. So much for that.
Erlang/OTP leads in the modern era.
But I don't see any relation to our current topic of discussion.
Think all the race conditions, deadlocks, etc that have hit server apps would've happened on Solo, Ada tasking, or SCOOP? Probably fewer to none. So, it's still relevant and Ada/Eiffel are actually building on it. Especially SCOOP R&D community.
Both OpenSMTPD and qmail seem to rely on concurrency through forkexec, which would entail deferring to the kernel's CPU scheduling and job control.
"Event loops seem to have been the norm for C projects for a while."
I'll take you word for it as I've seen that in the few, top-tier ones I scrutinized. There's a similar trail of research dealing with that alternate model. Have a feeling the meme might apply there, too, but won't accuse them until I see for sure. ;)
I want to respect OpenBSD. To a great extent, I do. And there are reasons to prefer tested C codebases to untested code bases in new languages, sure. But to the extent I can't give my full, unadulterated respect to OpenBSD, it is because I don't see that they consider everything being in C a problem to be solved.
I mean, I get engineering considerations and that we can't rewrite everything at the drop of a hat and performance matters, etc. But it is my opinion that any engineer that cares about security really ought to see "written in C" as starting out with a huge entry on the debit side, and take appropriate actions. And every year that goes by that's more true.
On the other hand, the issue is a tad more subtle. There seems to be an almost Pavlovian revilement and panic against C code, and this usually comes from people who haven't dealt with the language in a long time or at all in any depth.
For one thing, the blanket assertion that secure C code is a Herculean task, doesn't quite pan out. djb showed one way, based on a few general principles espoused in qmail, that has since successfully been reapplied by people like Wayne Marshall, Laurent Bercot, Bruce Guenter, Paul Jarc, JdeBP and so on. Much of it comes from abandoning the C standard library in favor of more robust alternatives like libdjb variants, clearly architecting process/module boundaries, using simple formats like netstrings and text0 that need not be "parsed" in the conventional sense, etc.
Further along the line, we have the SEI CERT C Coding Standard that comprehensively identifies many pitfalls. [1]
Deeper still, cryptography is one area where abstraction can be an enemy. You actually have to deeply worry about time complexity, branch prediction, precise timing, compiler optimizations and so on [2]. A tool like C becomes desirable here.
Lastly, but crucially - you're on Unix, for better or for worse. POSIX, SUS, all of those standards? They're deeply written with the conventions and API/ABI semantics of C in mind. Such is life. When you primarily write code at that level, the easiest way not to have any impedance mismatch is to just write it in C.
[1] https://www.securecoding.cert.org/confluence/display/c/SEI+C...
For crypto, at this point, C is now too abstracted from the hardware to be useful. You need assembler, and that's getting dangerously abstract too, since it's actually an abstract language over the microcode that actually execute, the entire point of which is generally to provide lovely, wonderful things... that trash timing guarantees that crypto would like.
"I had to write code that allows remote attackers to own the machine because it was the only way to avoid impedence mismatch."
Incidentally, since it hasn't come up, note that I consider "C + useful static analysis tools" to be a different language. "C + useful static analysis tools" can be a sensible choice. I still at least have questions about the cognitive load being loaded on to the developer vs. a language that removes the problems entirely, but it must be acknowledged as having radically different cost/benefit profiles. However, I observe that very few people seem to be using useful static analysis tools beyond -Wall.
Gave details here on C and UNIX history with counterexamples from other systems:
The OpenBSD folks have done great work through auditing and writing safer code in C, and I agree that using a language that ameliorated at least some of the common mistakes programmers make would be a net positive.
Related - this blog from @tedunangst is worth reading, regarding the limit to which even a language like Rust can protect against specific exploits (the OpenSSL "Heartbleed" vulnerability, in this case) http://www.tedunangst.com/flak/post/heartbleed-in-rust
1. Object X frequently removes limbs from the user.
2. Object Y does the same thing, but more safely.
3. Wait, Object Y causes hangnails if you misuse it hard enough.
4. It's OK to keep using Object X!
Here's an example from one of the leading groups in the field:
http://www.cs.cornell.edu/jif/
That was turned into SIF, Swift, Fabric, and I think Civitas voting system. Swift even auto-partitions the system between client and server side while maintaining confidentiality/integrity policies. Fabric kicks butt on all kinds of issues in the hardest space of them all. So, given the above, I'm sure a problem like Heartbleed in a mere SSL protocol would be trivial to prevent at language level if software and compiler developers merely chose to adopt info-flow security at least for important projects.
Note: One can use stuff like that even if not supported in main tool set by creating a mock-up of software in supporting toolset. The abstraction gap presents issues but a problem in one might mean problem in other. Especially for what amounts to covert channel analysis of software.
1) Rust's use of LLVM means it will not work on some hardware platforms that the OpenBSD folks are committed to run on.
2) The Rust project currently does not offer a long term support release.
These are important to the OpenBSD team. Additionally, I am sure that they value the memory safety that Rust offers.
I wonder if the OpenBSD team would ever design a systems language?
If it works out there, OpenBSD could audit and import things at their leisure, at least in theory.
To a first approximation, every single one of those wouldn't happen in a memory-safe language.
Of course, plenty of other things will. But it's not like choosing memory-unsafety makes those other things less like (memory unsafety does not, for instance, prevent incorrect ACL specification), so that's not very compelling.
Think of it this way. Suppose C didn't exist, and somebody proposed a new language today that was memory unsafe. Would it have any success, or be laughed out of existence?
In fact, go ahead and suppose C did exist, because it would still not be taken seriously.
If it's so wonderful that C is memory-unsafe, where's all the new languages that are also memory-unsafe?
(Note having specially-called out "unsafe" operators does not make a language "unsafe" the way C is.)
C being a plague still stands.
Compare new to new and tell me: how many web, smtp, imap, dns servers in Python, C# or Rust do we run today? There were some attempts but very few successful ones, despite of all the goodness that comes with better languages, libs and tools. I bet there's a reason behind it, other than just "people didn't try hard enough". There's old stuff out there being used today that we should definitely phase out. But we'll replace system software with new one written in system languages. Most likely C, perhaps Rust. But definitely now with ones written in "safe" languages that should deliver (if they are so awesome) but somehow didn't.
I've already detailed some of what was known about avoiding reliability and security problems before C and UNIX in the link below.
https://news.ycombinator.com/item?id=10273656
I know at least MULTIC's tricks were known at time of UNIX because it's name is a pun on MULTICS. Anyway, the better parts were largely eliminated to make C and UNIX efficient enough to run on a PDP-11. So, it's a fact that good stuff was known, they stripped it out due to constraints, and they never fixed those decisions as constraints lightened. That language, toolset, and coding style were adopted by many developers that continued to do things that way.
Meanwhile, in parallel, other trends began running. The evidence by the 1970's suggested that building secure systems require specs of behavior, a security policy, analysis one possesses the other, careful choice of language, pentesting, minimal TCB's, interface protection, and so on. Starting with MULTICS security evaluation, which talked about "insufficient argument validation" (ex: buffer overflows), that crowd demonstrated in one project and paper after another how to build secure systems. Plus, where most bang for buck was in terms of efforts.
In parallel, people like Margaret Hamilton, Dijkstra, Wirth, Hansen, etc determined how to make software that was correct by construction. Hamilton did that in 60's on Apollo. Dijkstra's method of layering of simple components with careful interfaces made nearly flawless software. Wirth & Jurg did a whole PC, safe (but efficient) language, compiler and OS in 2 years that performed fine. Kept building on that. Hansen and Dijkstra built things like Concurrent Pascal that eliminated concurrency errors on top of Wirth-style work that reduced sequential errors. One group after another built on such approaches with better results than UNIX and C crowds in terms of functionality vs labor available vs defects found.
So, this stuff was well-known at repeated points in the implementation of UNIX, server software, compilers, etc. Better approaches are finally mainstreaming with likes of Go (Wirth-inspired) and Rust (Ada redoux?). Other safe languages were plenty mature before they showed up. Yet, many projects focused on security still use C. So what I'm talking about not only happened then: it's still happening now.
"Compare new to new and tell me: how many web, smtp, imap, dns servers in Python, C# or Rust do we run today?"
Let me just apply your words to another problem. How much enterprise transaction processing uses C and UNIX today vs IBM and COBOL? There were some attempts to switch but very few successful ones despite all the goodness that comes with better languages, libs, and tools. I bet there's a reason behind it. Yep, but it's not that IBM and COBOL were best. I'd venture people not willing to put that work in, fear of breaking things, inertia, not knowing about better approaches, etc. The usual. Currently happening to C, C++, and Java stacks. :)
"But definitely now with ones written in "safe" languages that should deliver (if they are so awesome) but somehow didn't."
They did. Wirth made a whole series of them that delivered in many ways. Borland Delphi, Oberon, Component Pascal, and Modula-3 saw use in industry. GEMSOS used Pascal w/ coding guidelines to hit high security in the 1980's. Ada improved defect count in every empirical study I saw. A tasking profile for Ada and Eiffel both knocked out plenty concurrency errors. MULTICS used PL/0 to avoid issues that plagued C. Academics working on large projects like compilers and sometimes OS's had positive experiences with the LISP's, ML's, and Haskell. And so on.
They proved their worth every time they were used. People's gripes could've been fixed with less effort than what's been put into making C code maintainable, reliable, secure, etc. Instead, they kept trying to build modern stuff on a language & OS stripped of its best features to run on a PDP-11. And stayed on that bandwagon to the end.
So while I don't like C (or UNIX for that matter) I still do celebrate the fact that these people are actually doing something and even more importantly they are doing what they want despite the naysayers and doing it well.
You can't really divorce UNIX from C.
The only other language option for an smtpd server in OpenBSD base would have been Perl, and I bet people would complain about that just the same.
And perhaps we wouldn't be able to review it properly if it was written in Perl. Perhaps Marc Espie could. But OpenBSD has plenty of people on the team who are well trained at auditing C.
I'll be doing my part and read some smtpd code during an 8h train ride tomorrow. I already read one file and found one bug in it and told gilles about it. That's how this works. Changing the language won't magically fix anything.
Wonderful! This is exactly what I (and many others) were hoping for. Thanks so much for your willingness to jump in like that.
As much as he'd like to make it like we don't fix bugs, our public bug tracker is one click away and shows that out of the 25 open issues and the 493 closed issues, none mentions even remotely a pending crash that he is experiencing. He does have some feature requests and bugs though, with many handled already, so it seems that he does knows where to find it when he needs to.
As for his concerns, yes he did tell me, he sent me 16 mails yesterday, including the one with the trace. Unfortunately, he sent it past midnight, at about the same time he made it public because he just couldn't wait until morning to disclose.
Haven't you been paying attention? You're supposed to write your security-critical code in a language that was released 5 months ago and has no major deployments yet.
I think the project deserves more contributors and more eyes to look at the code. A fantastic foundation has been laid, but we can't expect Gilles to be able to do it all alone. This is a diamond in the making!
Regardless, the main worry areas are parsing, protocol handling, formats, and limiting effects of any given external action on system's state. Making sure those are correct should knock out most issues. Not that it will be simple or easy. ;)
"we can't expect Gilles to be able to do it all alone"
Just like with GPG. In FOSS circles, there's all kinds of people that want to take the code but few to create it. So, people gripe about the state it's in while putting no effort in to improve it. At least the author contributes significantly. However, gotta wonder how many users it has vs how many are contributing code/audit.
Bet the difference has an impact on the situation. It's why I spend time here and there investigating proprietary, open-source models that require money or contributions to use the software. Doesn't even have be a lot: at least enough for a significant product to support at least one developer. That way the users are always investing into it and esp bug fixes happen.
The real problem is that the OpenBSD community didn't realise the state of things internally, because gilles and eric were trusted to not fuck up. This happens, since not everybody has time to follow every corner of the tree all the time. But when focus on a particular area is needed we can do it.
It's good that Jason pointed out these problems, however the manner in which this was done creates huge public exposure which puts pressure on all of us (especially gilles and eric) to produce results very quickly while we prefer moving slowly and being careful because that produces better results in the long term.
Appreciate you adding detail on the project's situation. Also on how you all view the post and its effects. As HN gets exposure, do you a specific, alternative approach for anyone reading that might end up Jason's shoes to get the OpenBSD team focused on a particular project and digging into it? I'm guessing he didn't make an attempt outside the public disclosure or made an ineffective one.
If they don't reply after a few attempts, or give unsatisfactory answers, try mailing Theo. He acts as arbitrator in such situations.
If even that doesn't help, your last option is to write good fixes yourself, and keep submitting them as diffs to tech@ until you're either banned from posting to the list (at which point you should just give up) or your diffs end up being commited.
I agree it's the best option to rollup the sleeves and just fix the problem you're griping about. However, a write-up like Jason's is a valid option, too, if person doesn't have time/resources/skill for that option. Because, at that point, known issues would've been ignored by everyone up to Theo despite it being in default install and under OpenBSD's quality image. A write-up drawing attention to the issue would be justified to prevent users from placing unjustified trust in that component. Being informed lets them take action either on improving the component themselves, limiting damage it could do, or just avoiding it altogether.
Still agree that the gripe should go to the developers and Theo first.
Anyway, like it or not, in the OpenBSD community, working diffs, especially ones that fix a security issue, are the best way of winning an argument. You can try to blog about it but you probably won't be taken as seriously. So if you really care to convince the community from the outside and everything else has failed, and you can't code C youself, find a friend who can and ask for help.
On the other hand, as shipped, I have had no problems with it on non-mail server. I will probably try again next time I need to revise my mail server.
I'm casual OpenBSD user and may be I don't understand something.
for 5.7 refer to http://www.openbsd.org/errata57.html for what has been patched
If compiling it yourself is not an option, then refer to snksnk's comment and use https://stable.mtier.org
I've tried this in the past but some ports (typically big ones like web browsers) will run out of some resource or other during the build.
Also it seems that if you build from ports all the dependencies are built from ports too, even if they're already installed as packages? It ends up taking hours in some cases.
If library bumps happened, the packages might take a while to catch up, though.
What I do is I build a release with patches I'm interested in (my own or other people's). After upgrading my machines to that release I try `pkg_add -u`. If that complains about libraries, I just wait a day or two for a new set of packages to come out and try again. Sometimes I cheat and compile the missing library in a slightly backdated src checkout, or fetch it from a system that happens to have a copy of the right .so file.
If all that is too complicated, or if you want -stable instead of -current, try mtier.
Yikes.
"pkg_add -u" is just for ports/packages. Security updates to OpenBSD itself are distributed via patch and are listed here for your version here: http://www.openbsd.org/errata57.html. Explanation on how to apply them is found in the patch file itself.
"Could people have just improved the configuration or setup of already rock-solid qmail?"
improved the configuration or setup
IMPROVED the CONFIGURATION or SETUP
Do you have an answer to my question about refusal to improve an already secure and proven solution? Or do you have another strawman point ready?
Think of the configuration method as using duct tape to permanently set the switches/levers of an f1 race car instead of just getting an automatic Toyota car that is simpler for almost all people to drive.
A lot of the security and simplicity is supposed to come from not having so many options/knobs/levers of operation.
I assume 5 years later it's much improved but at the time forking the code of an SMTPd that had security vulnerabilities didn't seem like a worthwile investment and everything else has a OpenBSD prohibitive license.
Edit: It was all pretty thoroughly discussed on various mailing lists and whatnot if you're curious
http://cr.yp.to/qmail/qmailsec-20071101.pdf
Far as their decision, that makes sense. I still think they should've leveraged proven design principles or tactics in the other software. I've just seen enough of these situations where I believe what you say about how their evaluation and decision process went. I figured licensing would factor into it, too.
http://cr.yp.to/qmail/qmailsec-20071101.pdf
Do you think they could apply any of that to current project? Particularly in sections 3-5. I defer to you here as your experience is directly relevant. Would any of his approach, on top of looking for top 10 C issues, have improved the security of this app? Or was none of that applicable?
Most people run Postfix out of the box (apt-get install, answer which domains you want to route mail for, done) and it's completely solid. Wietse Venema had a name for himself in security circles long before he started writing it. DJB too. Neither is likely to be the vulnerable link in your system.
This misinformation is my only problem with the article. That claim has probably never been true. If it was, they wouldn't be regularly fixing shit that might be exploitable. Still, I came up with a way to test it years ago on Schneier's blog for anyone that wanted to try:
"Watch mailing list for bug announcements. Quickly determine if it's exploitable. If so, turn it into a payload. Hit target system. Profit." So, it's special in terms of quality focus but still can be hacked if anyone bothered to try. (Few bother.) Especially at layers below kernel that they don't care about, privilege escalation because they refuse M.A.C., in a VM because Theo doesn't worry about those issues, and covert channels because they probably never heard of them.
Note: It's reputation adds more risk as it's often used for "fire-and-forget" appliances. This can be reinterpreted as "they don't patch or they patch very rarely." The more they trust it, the more likely the bad guys get in. ;)"
Well, pretty easy to disprove. All you have to do is name a single remote hole in the default install in the last, oh, 8 or so years.
That would be a start. My guess was that OpenBSD team didn't track (and update web site claim for) the number of defects that might have been exploitable: just fixed all defects as they found them instead. If so, that makes for great QA practice but inaccurate claim on web site. Otherwise, there's a tracker somewhere listing all bugs found in privileged software, their exploitability, and time it took to fix them. That tracker will have a huge list of issues with only 2 marked as security issues.
Ok, so the second is, "Have there been defects found in any parts of the default install after those parts were distributed? And were they just fixed or analyzed for exploitability with an update on the website if necessary?"
1. There's never been a defect that could affect security in any OpenBSD software, from kernel to protocols, after it was released.
2. For meaningfulness, the default install supports real-world usage (and usefulness) of the system.
The good news for No 1 means that they never had to issue a patch in a protocol or software due to defects with potential to leak data, allow hijacking, or experience DOS. Knowing that, once released, every OpenBSD system was unhackable and needs no patches past functionality fixes would greatly reduce administrator overhead. Truly fire-and-forget system. If the website claim is true, that is. :)
Further, if No 2 is true, users won't have to use anything but the base install to handle day-to-day tasks on their network. Anything critical is included, usable, and secure-by-default. Users would only need to rely on 3rd party stuff for non-obvious use cases or stuff that truly doesn't belong in a baseline. So, the ports collection wouldn't even be necessary in most deployments (esp infrastructure, console desktop) as the functionality and what counts in security claims were batteries included like the competition.
I'll leave it to users and people following their bug trackers to determine if either is the case. Especially No 1 which is my main point. The mere existence of security patches would kill the web site claim. It would instead become, "Of all the defects found, only two were analyzed and shown to be exploitable over X years." Makes for a totally different impression, eh? ;)
I certainly could dig back into OpenBSD (and C) to fully understand its current situation, its code, its mitigations, every source change from vulnerability 2, and so on. Those details could benefit OpenBSD in form of contributions I attempt or the reader following along to get a better picture of insides. However, it's not necessary for evaluating my concern about the 2 vulnerability claim. We can cleverly cheat around all that knowledge by going straight to fundamentals and claim itself. Then, only acquiring (me) and/or applying (others) that knowledge if the specialist skill turns out to be required. Others and I did this with success when drowning in product and project security claims with finite time to review them. Let's apply that.
"instead of reasoning about all theoretical directions OpenBSD could have taken."
My main question is a simple, concrete one that I shouldn't have to investigate at all: what evidence is there that there have only been 2 exploitable flaws in all OpenBSD releases? To claim this, there would have to be a tracker of every defect fixed in OpenBSD since its inception, an analysis of each one's exploitability, and a result saying only two were exploitable. There's trackers like that for many projects. Where's that record for OpenBSD or, to save you time, do they even analyze every bug for exploitability?
If that doesn't exist or that analysis wasn't done, then the only 2 vulnerability claim can be rejected immediately because it wasn't proven. Instead, the claim becomes "Of all X defects found, people only showed two were exploitable remotely. (Optionally) That's a defect ratio of X per Kloc with vulnerability ratio of Y per Kloc. (Not optional) The rest either weren't assessed for severity or were shown harmless." Alternatively, the claim is totally false if even one security fix for a remote attack was done after the 2nd vulnerability was fixed. It would have to be changed to a maybe pending further, specialist analysis. This isn't theoretical: it's a direct implication of the claim they're making, an extraordinary implication, and one which deserves concrete evidence to back it.
So, I encourage people to reject it until they have that evidence and in a way they can share easily. If I have time or resources, I might attempt a thorough assessment of OpenBSD in terms of design or code. A past study I did comparing it to prior secure UNIX attempts showed some things it's missing where there's surely issues. It has some other attributes, esp code quality & protections, that help it more than others (including old ones). Nonetheless, the issue I have is an extraordinary claim that pops up repeatedly in discussions over OpenBSD vs whatever with no evidence to back it and high chance of failure. See the problem? Also, do you see how I don't need to be an OpenBSD developer to spot it?
Totally agree with that.
> do they even analyze every bug for exploitability?
I guess in a real world production used operating system there are practical limits. From http://www.openbsd.org/security.html
"Our security auditing team typically has between six and twelve members who continue to search for and fix new security holes. We have been auditing since the summer of 1996. The process we follow to increase security is simply a comprehensive file-by-file analysis of every critical software component. We are not so much looking for security holes, as we are looking for basic software bugs, and if years later someone discovers the problem used to be a security issue, and we fixed it because it was just a bug, well, all the better. Flaws have been found in just about every area of the system. Entire new classes of security problems have been found during our audit, and often source code which had been audited earlier needs re-auditing with these new flaws in mind. Code often gets audited multiple times, and by multiple people with different auditing skills."
and
"Another facet of our security auditing process is its proactiveness. In most cases we have found that the determination of exploitability is not an issue. During our ongoing auditing process we find many bugs, and endeavor to fix them even though exploitability is not proven. We fix the bug, and we move on to find other bugs to fix. We have fixed many simple and obvious careless programming errors in code and only months later discovered that the problems were in fact exploitable. (Or, more likely someone on BUGTRAQ would report that other operating systems were vulnerable to a `newly discovered problem', and then it would be discovered that OpenBSD had been fixed in a previous release). In other cases we have been saved from full exploitability of complex step-by-step attacks because we had fixed one of the intermediate steps."
> This isn't theoretical: it's a direct implication of the claim they're making, an extraordinary implication, and one which deserves concrete evidence to back it.
Fair enough. Maybe they should change the claim to something like "Only two remote holes (known at the time of the fix) in the default install, in a heck of a long time!" Then again, isn't this what most readers would expect?
" Maybe they should change the claim to something like "Only two remote holes (known at the time of the fix) in the default install, in a heck of a long time!" Then again, isn't this what most readers would expect?"
As your quote says, they actually discovered and fixed a ton of them. Users thinking that claim is true might believe their system doesn't need patching if no new features are added. So, I'd ditch the claim altogether if I were them in favor of something reflecting what you quoted. Those quotes show the real argument which is lots of prevention and constant improvement at a higher rate than competing OS's for code quality. That mindset dictates one continue to patch and upgrade: no "fire and forget."
By default OpenBSD has not opened up and ports, daemons, sockets, or other resources that would allow a remote attacker to exploit it.
If I wish to start making changes to the default install, that is, modifying /etc/rc.local to start enabling daemons/services, then I need to take responsibility for those changes, and ensure that I stay current with the security issues associated with those daemons/services.
Most making a claim like that... one that's usually wrong.... have data to back it up. Most people's position has been that I should accept it by default, accept reversed burden of proof, and disprove the claim by countless hours of inspection. Whereas, from high-security to mainstream stuff, the status quo has been something is insecure until proven otherwise. I see no reason to change it.
There have been no remote exploits in the last 8 years in the default install.
My theory is supported by another post:
https://news.ycombinator.com/item?id=10337807
So, I'm dismissing their claim entirely and will counter it with their claims linked above along with my argument. However, even without that claim, the OpenBSD team can still make a great argument based on their QA processes and track record in fixing things as you linked. It will certainly have fewer 0-days than anything else which is why I told another OpenBSD developer I recommended Xen projects use it for Dom0 wherever possible.
Btw, thanks for the links. It answers the question about their defect tracking in general. Lots of detail in there and evidence of their QA working. As I expected. I'm keeping it to share if the topic comes up in another discussion.