Native OpenBSD hypervisor hits -current
undeadly.org
undeadly.org
* If I was building a system where UI and broad hardware and software compatibility weren't an issue (for example, if I were building an appliance or IoT device), an exceptionally secure, stable, reliable, open, well-maintained and well-documented OS would be wonderful.
* If I was building a Unix server with those specs, why not?
* Why aren't Google's, Amazon's, Yahoo's, etc. server farms running on OpenBSD? How about iOS? (Android needs broad hardware compatibility.) Are there essential power or system features that Linux or FreeBSD has and OpenBSD lacks?
As it's not that widely used, I suspect there are considerations or problems I'm overlooking, or maybe the advantages aren't as large as I think. It's hard to believe it's all politics or personality conflicts.
While it's possible to use OpenBSD as a general purpose server or desktop, that's not where it's strength lies.
It tends to excel as a firewall or router for the under 1Gbit/s crowd that has Unix sysadmin experience - basically, appliance applications where it would function in the same role as a Cisco or Juniper router device.
OpenBSD's leadership has a goal of pushing the security envelope, and force positive change on the rest of the Unix ecosystem, which it's succeeded in doing in many ways.
None are as stable as their Linux or Windows counterparts.
That is, they do not only fix code, but try hard to ensure that code which doesn't match their quality goals can't do much harm. And this is security.
The topic at hand (vmm) is the best example for that. At a first glance, this is about creating a clean and correct hypervisor. But what is this good for? To be able to run less secure systems within OpenBSD without them doing too much damage.
Another example is pushing hard to enforce W^X on all kinds of applications, even browsers:
http://www.tedunangst.com/flak/post/now-or-never-exec
This doesn't fix any existing bugs, but it actively mitigates the damage that a bug may have (ideally, it prevents a buffer overflow bug from being actually exploitable).
From their website:
http://www.openbsd.org/security.html
| Our aspiration is to be NUMBER ONE in the industry for security
Sure, security by clean and correct code is their main measure to increase security. They call it "proactive security". But there's tons of stuff in OpenBSD which is "merely" mitigation.
I get the impression that the dominant attitude among OpenBSD developers is that if you have to choose between simplicity and security, your basic approach is flawed and should be reconsider. This line of thinking - that you don't make things more secure by making them more complicated - also results in a simple yet powerful system.
So unless I am mistaken, "clean and correct code" and "security" are not two different goals, in a way they are two faces of the same coin.
This should get better soon as there's a lot of work being done in OpenBSD to make at least some parts of the network stack SMP capable.
With regard to business's using openbsd, I am currently using openbsd for my private server on Digital Ocean, and have thought it would be a good platform for a small business to use.
People with similar command of less popular OSes are even harder to come by.
[Edit] add a link: https://marc.info/?l=openbsd-tech&m=144561370912073&w=2
We've even got Minecraft.
My OpenBSD laptop writes 1GB of data in a few seconds to disk (RAID 0 with 2 SSDs).
And the reason Linux is faster with memory allocation is because malloc does not really allocate memory... It does a lot of weird stuff. Just look up overcommiting for starters.
There are technical limitations as others have noted. The attitude and focus, though, I think is what kept OpenBSD in their own niche.
The only technical limitation I've run into as far as using it as a daily driver OS, is that it can be a bit slow even on very fast hardware. It's simply not optimized for desktop use, though it is certainly usable with a few tweaks. It's great for older hardware too, where the speed issue is relative. In fact, I've found it to be a perfect fit for a couple of old laptops in my collection.
Harder to say whether it's easier to bring OpenBSD to where competition is on these or to bring competition's code to where OpenBSD is, esp as some safety/security tech makes that less relevant. Hard to say...
Regardless, it's a high-quality UNIX that's worth having around. I'm glad to see them improving in areas where it's weaker while not throwing away strengths like some projects would.
I recently inherited an old-ish laptop (I don't know the exact year it was built, but it has a Windows Vista sticker on it, so 2007-2009, I guess) and decided to install OpenBSD on it, mainly because I have always wanted to give it a try on real hardware as opposed to a VM.
I was pleasantly surprised to find that all the pieces I care about (sound, video, wifi, ethernet, ...) Just Worked(tm). That was nice. More heavyweight applications like Firefox or Clementine slow the system down substantially, but otherwise it runs nicely.
Systems in 2007 had no problem running Firefox. I'm surprised to hear it has a significant impact.
Maybe Firefox has become much more resource intensive? That would surprise me because processors haven't gotten much faster, Mozilla has focused on reducing memory consumption and improving responsiveness, and because of the rise of mobile, a resource-constrained environment.
Matter of fact, I can prove it with an example [1] that predates OpenBSD's approach while succeeding in market at least while under good management (sighs). Used a combo of good design, OS features, and code quality to achieve reliability most still haven't beaten. Coding approach, which I'm focusing on here, was to alternate between developing features and fixing bugs with tests (esp regression) run on weekends while developers were off. One week build stuff, tests on weekend, one week fixing by priority, rinse, repeat. That simple. OpenBSD took it further with security focus and systematic audits but this basic method produced robust and marketable code. So, what's other commercial players' excuse? ;)
[1] http://www.itec.suny.edu/scsys/vms/ovmsdoc073/ovms_archived/...
I've heard this, but what exactly is missing? Do you mean features such as a hypervisor?
* proprietary software. Yes, sometimes you need to run it. It's built for Linux but not for OpenBSD.
* hardware support. OpenBSD is better than it used to be, but it's still not as good as Linux.
* apt/dpkg. There's just so, so much more prepackaged software available on Linux. (I miss Debian/kFreeBSD...)
But primarily, I think, it's solely due to Linux network effects --- people learn Linux because Linux is everywhere so people learn Linux. There's just more people and more knowledge around. Which means that if you're porting an OS to a new platform, you go with Linux because you know that the skills are available, and probably a lot of the functionality is already there, and there's a bunch of different userland stacks that will already just work on it, etc, etc.
For many of the scenarios you listed, it really boils down to peer pressure.
People who make decisions know that Linux is Good (TM). Convincing them that there is a better alternative for <this particular project> requires them to grasp technical concepts that they simply don't grasp.
And then you have a lot of programmers who know that <this operating they used> is Good (TM). Convincing them that there is a better alternative for <this particular project> requires them to have some broader, often theoretical understanding understanding of things, so that they can compare approaches and judge which one is better for a particular task. A lot of them don't have it; it goes against their belief that there is an operating system that is Good (TM). In fact, they'll go to incredible lengths to ensure that any operating system is like their Good (TM) operating system.
As far as IoT goes, though, hardware and software compatibility is always a problem. Most of them are built on tightly-closed SoCs. You're stuck with whatever the manufacturer wants to support, and that's usually Linux. I guess that partly boils down to the fact that, back when manufacturers started looking into this free operating systems niche, fitting Linux in 4 MB was a lot easier than fitting any BSD. But at the moment, why they'd jump (often imperfectly) through GPL's hooks is beyond me.
> It's hard to believe it's all politics or personality conflicts.
You poor soul :-).
Basically, I think OpenBSD is a good operating system. It has a clean architecture, great code quality, and I am fond of its engineering principles.
I don't think it's Good (TM) -- there are things that other operating systems do better, and I don't think its engineering principles apply everywhere.
I admire its design, but I don't try to emulate it in any system I build.
Aren't most IoT products custom built and therefore the designer chooses the specs? I know that choice is limited to realistic availability, but some hardware that runs Linux also can run OpenBSD - I would guess that applies to IoT too, but maybe not ...
Hardware that can reliably run OpenBSD tends to be in the more expensive space -- x86 or PowerPC based. There are exceptions, and OpenBSD supports them (see e.g. http://www.openbsd.org/armish.html , http://www.openbsd.org/octeon.html), but these are exceptional cases where you have some documentation of reasonable quality released by the manufacturer (or outright support for porting from the manufacturer), open-ish specifications and so on.
In the IoT (read: severely price-constrained) space, you don't get to work with this kind of hardware.
To give you just one example, most cheap router SoCs are based on a MIPS core with a bunch of peripherals around (basically they're a very glorified microcontroller). Porting OpenBSD on one of these requires:
* Supporting the MIPS core. This is ok at the moment, MIPS CPUs are supported.
* Drivers for the peripherals (which include e.g. the wifi radio). This is very hard at the moment. Most manufacturers don't open source their drivers. You get a binary blob and a wrapper module, a la what nVidia does, and the specs are closed, supposedly in order to protect "intellectual property".
* And a bunch of boilerplate information, like the device tree, because these systems don't have a BIOS. There's no standard regarding this aspect, which is why even compiling Linux for these platforms is pretty painful. You generally need the manufacturer's SDK, which varies in quality, albeit after a while that gets adapted into a set of patches that get integrated in a distribution like OpenWRT or Yocto. If you're really lucky, the manufacturer might offer a Yocto or Buildroot-based distribution. If not, you get a bunch of build scripts developed in-house, with hardcoded paths and comments written in Chinese.
You do get to choose the specs, but you're still constrained to what manufacturers offer you. The cheapest available things run only Linux and maybe, if they're based on a legacy, but important product, they'll run VxWorks. But you're typically confined to what the manufacturer gives you, and the manufacturer usually gives you Linux. On 90% of the hardware popular in the IoT community, even that Linux barely works. The amount of hackery (in the pejorative sense), duct tape and outright crap code I've seen in manufacturer-provided kernelspace code is horrific. E.g. there's at least one fairly popular SoC, used in a lot (tens of thousands, if not hundreds of thousands or even million) consumer-grade routers in whose drivers I've seen synchronization based on sleep()-ing, writing to hardcoded memory addresses and so on. Yes, in kernelspace code. You don't want to see userspace code.
Thing is, most IoT companies don't want to bother with this anyway. They want something that runs an OS their development team in SE Asia or Eastern Europe (full disclosure: I'm in Eastern Europe) already knows, and they don't really care about kernelspace or userspace code because, preferably, their development team will program it in PHP and JavaScript, since three of them cost less than one C programmer who will stare at the code and quizzically mumble "how the fuck does this even boot?"
(This also accounts for the can of worms that IoT security is. A security-minded programmer would need less than a week to find several disastrous security-related bugs in the SDK of basically every popular-ish platform.)
tl;dr IoT is built on cheap, low-quality hardware where you get whatever the manufacturer gives you, and the companies are OK with this because really, they just want a box that runs a web server.
There is of course the advantage of being more obscure and thus less likely to be exploited, but most people probably consider desktop Linux obscure enough and companies usually don't care about such security mitigations.
For example they clearly state that one should use fgetln() instead of fgets().
http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man3/...
Of course, that won't fix the issues with using C, but the mentality at least is more security oriented.
What we really need is something like Midori reaching production. It would surely cost less money than all the Windows Phone restarts.
For example, if you want to deploy a Rails application, you can find working ‘init’ files for Debian & Co. easily. If you use FreeBSD, you have to write your own init files. It’s not hard but might be one or two hours of work for someone familiar with *BSD before you get them to work correctly.
Then you have docker on linux. FreeBSD added docker support in June 2015 and according to their website is not production ready. OpenBSD doesn’t support docker. For small dev teams supporting their products or for bigger teams which work with micro-services docker is heaven-sent. It allows you to run the exact same configuration on the server, out of the box by just mapping a few ports.
Likewise, there are many tutorials, howto’s, etc. which make it trivial to setup a HA-Proxy under Linux. But if you need to setup CARP+HAST under FreeBSD you have to go through the official documentation - which is top quality material - and find your way from there. It’s not a 10-minute copy/paste thing.
Add to that a rather elitist culture e.g. BSD is designed, Linux is grown - roughly translated as ‘we are engineers, you are hackers’ which in many cases is not far from the truth and you have an explosive mixture which renders BSDs obsolete.
That said, I find strange the fact that so many corporations choose to support GNU/Linux over BSD in terms of driver development or funding, given that the BSD license is way more permissive. This was the reason Linux picked-up early on and became way more popular than BSD among hacker, but when corporations jumped-in, they didn’t choose BSD, they went with linux. I am not sure if this was a data-driven decision or it was just a matter of technical people in decision-making positions have a lot more sympathy for Linux than BSD.
At this point in time, GNU/Linux has clearly won the race. However, if someone would ask me to setup a ‘secure’ mail-server, file-server, whatever without specific requirements (e.g. docker, LVM, etc.) I’d probably go with BSD.
Linux was already showing a bit better momentum by the time corporations starting contributing.
Also I suspect some of it came down to legacy Unix vendors (IBM, HP, SGI, Sun etc) deciding that if they were going to contribute to open source projects, GPL was actually preferable to BSD. IBM didn't want their contributions ending up in Solaris and Sun didn't theirs ending up in AIX etc. Maybe the same with competing hardware vendors and drivers too.
* There is no supported upgrade path from stable release to stable release.
* If you want regular updates to to packages, you need to be running current.
* It has much better driver and power management support than FreeBSD, but not as good as Linux, and is painfully slow in some contexts due to the various areas where technology is lagging: - SMP - Filesystems
* They've done the work to make the latest version of Gnome Shell run, but it doesn't perform well, and is not close to the experience on Linux with identical hardware.
https://stable.mtier.org/ offers package and base binary updates for the present release.
This is really good news because now I can move my VM host to OpenBSD.
OpenBSD is a clean simple user friendly OS.
OpenBSD has happily run as a guest for a long time now, with various virtio drivers being added some time ago. Solutions like virtualbox and xen reach far into the system and are still a no-go on OpenBSD.
vmm on the other hand is a very literal OpenBSD implementation of a hypervisor. Minimal cruft means very little device emulation, virtio support only etc... The code is simple and readable[1], as opposed to other monstrosities.
I would take a punt and guess that the reason OpenBSD hasn't had a hypervisor is that a) all of the current solutions were inappropriate, and b) nobody had the time/effort to implement an appropriate one.
[1] http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/arch/amd64/...
"x86 virtualization is about basically placing another nearly full kernel, full of new bugs"
"You are absolutely deluded, if not stupid, if you think that a worldwide collection of software engineers who can't write operating systems or applications without security holes, can then turn around and suddenly write virtualization layers without security holes."
He clearly hates x86 more than most but his point applies in general. He's not the only one whose made it and there's truth in it. It's why high assurance virtualization... well.. applied high assurance to those parts haha. Also, why attempts at robust, resource virtualization typically pushed as much complexity out of the hypervisor as possible upward to the OS's that were on less privileged rings or capability-isolated depending on which system.
However, I said then he was wrong because the hypervisor and even VMM functions are less complex than a whole OS. The past examples showed they can be implemented very simply. We got further confirmation with the NOVA microhypervisor, OKL4 platform, separation VMM's (eg LynxSecure), and so on. People are still finding kernel flaws in the UNIX-like OS's due to architecture, language used, and intrinsic complexity. Many less problems in aforementioned software.
http://marc.info/?l=openbsd-misc&m=119324926326885&w=2
> If people were saying: "Yes, it increased hardware utilization, and the nasty security impact might be low" it would be fine.
"But instead we have many uneducated people saying: 'Yes, it increased hardware utilization, and it improved security too'. And that's complete and utter bullshit."
Whereas, as I referenced, many VMM systems did increase security via isolation with something simpler than the arbitrary OS and monolithic software contained. Lowest TCB I saw with minimum necessary features was in 50-100KB range. What's OpenBSD's + VMM's TCB size, again? :P
Taking 2nd link into account, it still has that thing about it claiming virtualization can't improve security posture, prevention or recovery. That was repeatedly proven false in academic and production systems with some surviving pentests by pro's that regularly tore through UNIX OS's and commercial fodder. So, his statement against security potential of virtualization is "complete and utter bullshit."
Note: As with other link, it becomes true if one is talking about common offerings, esp on x86.
A premise which he rightly shat directly on, and is his statement is completely congruent with the presence of a VM hypervisor in OpenBSD.
A specific answer that only took a day to form.
"If you actually read the thread"
Still this again, though. That's three people whose position is that anyone wanting to know OpenBSD's position on virtualization should spend 20m-1hr digging through threads like that and many dozens of hours watching video presentations/interviews. Just in case the answer's there. That's quite unreasonable given one link to a definitive answer on a mailing list, site, etc is all it would take. It's not a one-off thing as this topic comes up endlessly with them having the same response minus some exceptions.
So, given them acting that way, it's a reasonable default for outsiders to assume they didn't give a crap, got way behind on virtualization, cite comments like that just to save time, and finish by adding they're finally doing something. It's actually more effort than OpenBSD supporters put in those discussions. At least one had the wisdom to send me a video with Theo straight up saying it wasn't a priority. QED on whole topic. See how easy that was?
Not so easy in certain circles it seems...
Okay, okay. Personally, I think the fact that OpenBSD did not support any of the current virtualisation solutions, and now have an appropriate one in the works says a lot about their position.
And, frankly, what use is an organisations "position" on VM hosting to a user? It either supports it or it doesn't, and if you don't plan on developing it the reasons don't really matter.
EDIT: I'm also going to point out that mailing list posts in general rarely stand on their own, and exist within a context.
It's ambiguous far as past is concerned. Two possibilities are (a) they didn't care for longest but caved on the issue or (b) they wanted it, waited for a solid codebase that never showed, and finally did one of their own. All I can tell is that it matters to them now.
"And, frankly, what use is an organisations "position" on VM hosting to a user? It either supports it or it doesn't, and if you don't plan on developing it the reasons don't really matter."
Not true. Very few develop for Linux or FreeBSD vs number of stakeholders. Nonetheless, many features non-developers would want came about because people needed it or were talking about it. I agree with you for OpenBSD specifically, though, as they've been clear about "code it if you want it so bad."
"EDIT: I'm also going to point out that mailing list posts in general rarely stand on their own, and exist within a context."
True, too. Probably best way to apply that is to just not quote mailing lists if it's a huge conversation. I only quote or back those references because later interviews corroborated them a bit for virtualization in general. We certainly shouldn't just grab things off mailing lists without a context, though.
So, given prior conversation, I thought of their recent efforts to do what they were against or not interested in to be a compromise between how they wanted it and what OpenBSD needed for more use in market. The first of hopefully many which might bring more coders and capabilities to the project.
Whenever Theo speaks about OpenBSD he uses the words "research OS". For every developer this might be different, but it's still worked on a From-Developers-For-Developers basis. If anybody else can use it then that's also good. If you want to change it, send patches. Don't demand features, send patches. Don't know how to setup things, read manpages and the faq, if it's not in there find out for yourself and then send patches for the documentation.
Most things are based on a need a developer once had and couldn't live without. That changed a bit since the OpenBSDFoundation got some money and is now paying for development time for features nobody dared to touch so far... like VMM (at least that's what I heard)
For me OpenBSD is not about market share but about best practices.
One problem that would be good to address (and maybe it is) is making it really hard to dump the memory of a VM. We don't have homomorphic encryption yet, so right now if you dump a VM you own any secrets it holds. So at the very least this should be made as hard as possible on production VM hosts.
Making it hard buys you some resistance to less skilled attackers. Since skill tends to obey a power law distribution, this is most attackers. Make it as hard as possible and you get more and more protection from more and more attackers, and so on.
Your average attacker doesn't know enough to bust out x86 assembly language to inject root code into the kernel and dump a VM's RAM. So make it so you have to do this, and you've ruled out your average attacker. The average attacker will only 'dd' the VMs' disks, which if they've encrypted data at rest will not get them the crown jewels. But make it easy to dump VM RAM and you've exponentially increased the damage from the average attacker (due to power law distribution of skill).
For comparison -- does your average Windows attacker know how to cryptanalyze the LM hash function? No, but they can run the tool just the same.
There are absolutely places where your the general argument holds water: "no security by obscurity" is a simplistic framework, and ignores a much more reasonable and also simpler framework of costs and incentives. In several cases, the costs of figuring out something obscure outweigh the potential incentives of breaking it.
But in this particular case, there are no marginal costs to the attacker. Once someone somewhere invests the time to figure it out, for any motivation including curiosity, the defense is immediately worthless for everyone.
You're trying to protect yourself from an imaginary class of attacker that just doesn't exist.
Unrelated to OpenBSD but related to VM security from administrators - https://4sysops.com/archives/shielded-vms-in-windows-server-...
If you can take a memory dump of the host sytem, you can from that devise the encryption key for the disk encryption with the M$ solution to this issue.
Maybe uclinux...? Or qemu - more nesting!
I remember reading about it when it was started, but haven't heard much about it lately.