The End of the General Purpose Operating System
morethanseven.net
morethanseven.net
We could have a layer that did all of this!
The layer is introduced, applications are rewritten to target the layer, and people slowly lose touch with what the world looks like beneath The Interface.
And some things that should not have been forgotten were lost. History became legend. Legend became myth.
As people target The Interface, it grows into a more and more general purpose machine. Layers of indirection build up. Once simple tasks must propagate up and down a big stack.
Until one day, someone stumbles across the layer beneath. Gosh, the underlying system does 99% of what we need, out of the box. Why do we need this layer at all? We can do most of what we need with a couple single purpose tools. And the rest of the complexity can be taken up by the application layer. I don't mind doing a little more configuration there if I can get a huge performance and complexity win.
And the developers, frustrated with how big and bloated their layers have been feeling, flock to this new simple tool, and they port their applications, with a little extra boilerplate and big complexity wins. And then they port another, and another. Until someone realizes
We could have a layer that did all of this!
And it might seem pointless, when I write it up in this snarky way, but it absolutely is not. This is the process by which we discover the fundamental building blocks of software. Each time we add a layer, and each time we take one away, we learn something new about what information is. I love it.
That does not result in people writing software that is very useful for a large part of a particular field, but takes 2-3 years to write.
But the fault is in the app store and the lemon market it creates, not in the operating system.
Microkernels
ChromeOS
MVC frameworks begat Vue.js and similar
Virtual machines to containers
But isn't part of the cycle, that we constantly forget what we just did?
The real "operating system" is the container orchestration system.
I jest but as an industry we do seem have an almost metronome-like tendency to lurch from one thing and back.
For example, all JEE containers allow for sharing libraries among applications, and they can be running bare metal for what I care.
I can even control in which order they get loaded in each application.
technically doesn't come by default with Linux (the kernel) anyways
> no display support
can be configured out. I'm pretty sure you can even configure out the whole TTY subsystem if you want, so you can't even use a serial port to debug your madness.
> no printer support
who still uses parallel ports? for anything else (except usblp which basically never works) you need CUPS anyways
> no hot plugging support
last I checked, hotplug is mandatory on x86 (but only for certain components, so you could configure out say USB support if you wanted)
> few drivers
sure, you can compile your own kernel if you want
> no battery management
I don't really know what "battery management" means.
> The OS underneath ought to be simple enough to be installed for the life of the hardware.
you can already try this with Linux. then you run in to problems like "what happens when there's a vulnerability in Xen", because https://marc.info/?l=openbsd-misc&m=119318909016582.
I assume he means "power management" (i.e. ACPI on x86). You can run the system without taking control of the ACPI hardware, but this is a terrible idea. It'll affect your ability to run the CPU properly, even assuming you don't care about power usage (and large server deployments sure do!). Not to mention all the nasty bugs and problems you'll run into because the hardware wasn't designed to run for long periods of time without ACPI active.
I know it's almost mandatory for the base machine running on actual hardware, but given that most of the servers we use now are just virtualized, how bad would removing ACPI support be? Also, would it be useful to remove that?
There are benefits to using the same components both inside and outside of the VM, just as there are benefits to using the same codebase bother on the server side and on the client side. There's less code to maintain, and you don't have to spend time worrying about compatibility.
I wouldn't mind the small inefficiency of carrying extra drivers and subsystems inside of my VM if it helps make my stack as a whole more uniform and predictable. We really don't need yet another Linux distribution.
Also I remember reading something about some hypervisors using the guest's decisions with regards to CPU power states to inform the hypervisor's setting of the same on the real hardware. I'm not sure if this is implemented in mainstream hypervisors, or if it's really useful, but that's another thing you'd lose.
True for both server-containers and mobile OSs.
Where before access was done via a limited terminal protocol, these days it is done via a much richer protocol (or stack of protocols, what with tcp carrying http carrying json or some such).
I don't think it was ever that local software was presumed secure. After all you wanted as little as possible to be suid, and if it was it should be as small a piece of code as possible to limit the chance of exploitable bugs.
Every piece of software should run within a sandbox, and the human user should have complete control over which resources are or are not exposed to each sandbox; that's the future operating system I want to see. I did some exploration around the idea of doing this with hypervisors and unikernels (http://www.github.com/marssaxman/fleet) but it got to look too much like rewriting all the software in the world. Containers are less elegant, but seem to be a more practical way of moving in the right direction.
I sort of agree with this but I think more in terms of projects/activities (e.g. casual media consumption, banking, working on different projects) rather than applications. To do different activities I need different combinations of tools but often the tools overlap. For example I want to sandbox the browser I use for watching cat videos on youtube from the browser I use for banking from the various browsers I use for work. So that some drive by on a casual browsing site can't attack my banking. But I don't want two browsers installed I want the same program in the same version with some common and some different config - but I don't want to manage the common config in 5 different places.
The multi-user model allows for some attempt at this if one sets up multiple user accounts for the different activities. It certainly isn't sufficient but I'm not sure that sandboxing each application will actually get us there either.
Your example of the browser for example, how do you know what is common config? What if changing a common config in cat-mode exposes a security vulnerability in bank-mode? And are you really going to log out of cat mode into common mode just to change a small setting?
Sharing files is another one of my favorites, say for example that me and my brother like the same music and share one computer with different log ins. We want to save space on the hard drive so we share all our mp3 with each other. But I might have some music or recordings I dont want to share, how should I handle that.
Windows XP tried very hard to get this working, for example your desktop and start menu was a blend of what was installed as a shared application (remember "install for all users/only for me"?) and what was installed for one user. Later if someone renamed things on what they thought was their desktop it would be renamed on everybody else's desktop also, it was very confusing even for me as a developer who knows how it works. Windows also has the concept of a dedicated shared folder and possibility to share additional folders, sounds so simple but I've never seen anyone set it up smoothly. Everything looks so good and simple on paper but requires just a tiny bit of user education and that's where the whole concept falls.
What about shared libraries? How do you define the access rights or set of processes that glibc is allowed to run with? (you might have to break up glibc into dozens of smaller libraries for that - and then manage their dependencies, the resulting web will not be too different from monolithic glibc)
Also it does not solve the problem completely (not possible to do that with security anyway) - see rowhammer exploits.
And this is already somewhat true anyway: binary package distributions are a little like this. Someone else handles the compiling.
So take it further: distribute binary diffs for packages to control file size, kill shared libraries dead. Have the compiler write dedupe-friendly executables so your filesystem removes duplicate blocks in binaries efficiently and let the kernel page sharing system sort out runtime.
$ go build -buildmode=c-shared
If you mean Go plugins, there is a proposal, anyone is free to implement it and do a PR.Or then just wait until it gets eventually implemented by the core team.
Either behavior conforms to POSIX, but only the musl behavior can satisfy the robustness conditions musl aims to provide. In particular:
Under glibc's approach, libraries not designed with dlclose in mind (which may not be the libraries directly loaded with dlopen, but rather their dependencies) may leave around references to themselves in such a way that removing them from the address space results in a crash (or worse) later on. This cannot happen under musl. Managing storage for thread-local objects is much more difficult if dlclose unloads libraries. If space is to be reserved in advance (needed to guarantee no late/unrecoverable failures), supporting unloading in dlclose seems to necessitate leaking TLS memory, which largely defeats the purpose of doing the unloading in the first place.
More people complain that statically linked binaries in Musl cannot call dlopen, which some things rely on.
If Windows can do leak-free TLS with FreeLibrary, Musl can too. I'm sick of hearing "X is impossible" when some other system has been doing X for decades.
However, said model still has connotations of a specific actual user, which is no longer entirely accurate in such applications of that model. It'd be nice to have a sort of "subuser" system where - within, say, the user for my own desktop account - I could further divide the software I run into "users" for things like Firefox or Spotify or what have me.
Basically, I'm in agreement that everything should be sandboxed. We have the technology to do it, and in fact have had the technology to do it for decades (maybe not quite as well as we can do now, but confining daemons to their own users has been possible for a long time).
My own dream system would be one where every "package" for my operating system is a filesystem image with `/bin`, `/lib`, `/etc`, and possibly `/var`. One of these packages would provide the root filesystem with a microkernel and the minimum supporting libraries and executables required to get a container-oriented `init` equivalent running; then, `init` would spin up each service in complete isolation by spinning up `chroot`s or something with various packages union-mounted on top of one another. One of these services could be for a graphical login, in which case said service would spin up another isolated container of sorts for my login session, and inside that I could run applications built up from the same union-mount approach with the same sort of isolation.
Plan 9 From Bell Labs is probably the closest thing to that ideal world.
However, said model still has connotations of a specific actual user, which is
no longer entirely accurate in such applications of that model. It'd be nice to
have a sort of "subuser" system where - within, say, the user for my own desktop
account - I could further divide the software I run into "users" for things like
Firefox or Spotify or what have me.
I think you might be able to use groups for this. sudo -u ff_user firefox
This way Firefox is run as a separate user within the current user's session, with all that entails. Of course, that would be a serious pita for your average user to setup.Reminds me of genode's application-specific trusted computing bases.
Take a look at NixOS and the Nix package manager. It works along these lines [0].
> then, `init` would spin up each service in complete isolation by spinning up `chroot`s
Systemd employs container features, some automatically like cgroups and some via configuration. There was an article on HN just the other day about some of its new sandbox features [1]. Parameters such as PrivateNetwork=, ProtectSystem=strict, PrivateTmp=yes, SystemCallFilter= provide chroot-like capabilities in a number of different namespace dimensions. It has support for chroot as well through the RootDirectory= parameter. See [2] for a good overview of these featuers and [3] for the reference.
Hopefully as these features mature and gain adoption, system software will operate more like what you're describing over time.
[1] https://news.ycombinator.com/item?id=12874731
[2] http://0pointer.de/blog/projects/security.html
[3] https://www.freedesktop.org/software/systemd/man/systemd.exe...
* Variable-length security IDs, with a much larger namespace.
* The ability of a process to employ multiple tokens during the course of its existence, including kernel-enforced restrictions on how it can employ tokens borrowed from service clients.
* Nonce security IDs, employed (for example) to partition a single user from xyrself when it comes to granting access to window stations and desktops.
* Universal security IDs for truly universal things, such as "Everyone" and "Creator Group", that are the same everywhere. "nobody" is one UID on FreeBSD and Debian, and a different UID on OpenBSD.
* Local security IDs for local things, such as all security IDs for domain user accounts incorporating the unique ID of the domain (controllers) making the domain user account SIDs globally unique. Two BSD/Linux systems can have two different users with the same UID 1001.
* Universal RIDs for things that every domain has, like "this domain's printer operator", "this domain's policy administrator", and "this domain's backup operator". FreeBSD's/OpenBSD's "operator" has the UID of Debian's "bin"; FreeBSD's/OpenBSD's "bin" has the UID of Debian's "sys"; FreeBSD's/OpenBSD's "operator" group has the GID of Debian's "tty" group; FreeBSD's/OpenBSD's "tty" group has the GID of Debian's "adm" group.
* Standard security IDs for granting permissions to log on locally, via a network, in batch, as a service, or via dialup.
* A Hurd-like mechanism for obtaining process tokens: an RPC transaction to a distinguished server process inside the TCB -- /hurd/passwd meet Local Security Authority Subsystem Service
Windows is not alone in this, SELinux suffers from exactly the same problem.
I just want to run my own containers or those written by others on commodity services from Amazon, Google or Microsoft. Or even spin up my own environment if it suits me.
I want to be able to install and remove software quickly, completely and whenever I want. I especially want to be able to install software I don't fully trust with no fear of consequences. I want to be able to install multiple versions and even multiple instances.
I want my PC (Mac/Linux/Windows) to be a thin hypervisor with everything in a container, including IO and UI.
I still want open source projects on GitHub that I can fork and contribute changes to at will.
I want containers to be small and ultra simple.
All of this means that my "OS" only needs to manage containers and resources. All traditional elements sould be in their own container such as logging, authentication, window managers.
Truth is, it is relatively easy to keep a machine secure. Just not if you want it to be a convenient machine.
You think you want containers all isolated from each other. Except for the uncountable times and ways you want to exchange data between them. This, at the least, includes the data of who you are, as the user.
So the people who build high assurance systems (which currently require a ton of work) are all too focused on convenience?
Specifically, remote access is a huge convenience that by itself moves the problem out of the easy territory.
It tries to move everything into containers, including hardware drivers, and if you read their architecture paper you will see their design ends up being nontrivial.
sounds like nixos might have some of the properties you want, without containerizing everything
Not to mention that your actual containers will still contain general purpose OS images.
Also, as an aside, cri-o would be a very bad choice for PID 1. It's specifically designed so that you don't need it to be a long-running daemon (which is something that Docker cannot do and is actually a very useful feature that we hope to keep). And you can't have your PID 1 just exit whenever it wants to.
My point was that even with everything on a system running in a container, you still need a general-purpose OS inside the container to do management. Not that in particular that a single-binary container-manager-as-init was the only case where that's true.
Very unlikely none share a certain idea, particularly that one.
A while ago I discovered that I can build my app (Haskell) as a Docker image, using the "stack" build tool. As far as I can see, Docker adds value by providing a layer of abstraction that makes building deployment images as easy as building the executable.
"... fight between Docker and systemd is inevitable" -- this fight has been going for a while (at least a year): https://lwn.net/Articles/676831/
"... The reason why you'll do this, rather than compose everything yourself, is compatability. Whether it's kernel versions, file system drivers, operating system variants or a hundred variations that make your OS build different from mine. Building and testing software that runs everywhere is a sisyphean task. " -- kernel versions, file system drivers, operating system variants (in form of docker daemon version) are still going to be around and would still affect containers. So you traded "test on Fedora, RHEL, Ubuntu" with "test on AWS docker, Tectonic, OpenShift". Yes, you no longer have to worry about .so versions, but there are still plenty of reasons to test in multiple environment.
"... the operating system is an implementation detail of the higher level software. It's not intended to be directly managed ..." -- yeah, so I have this proprietary SAN.. or a dual 10G cards which need to be tuned and bonded... or a mix of fast and slow disks... or even any non-trivial RAID setup... Those things are the most annoying parts of managing your own servers and unfortunately they are not going anywhere.
The author seems to feel that everything will concentrate at the hypervisor level - kernel and user land just a "detail" as they are single-purpose. However, you should spot that really the kernel and userland can be compressed into one layer. So why are there 3 layers at all? Then we have kernel and user land again and hey we just came full circle.
Again, it's just a reflection of how the organizations are arranged. Given a single organization with authority over the entire stack, it would be a terrible waste to have that 3 layer stack.
With IncludeOS, you don't even have a full kernel. You just have the bare necessities, and run your code in real mode in ring 0.
I reckon the point of IncludeOS is to run an application directly on a hypervisor like Xen (instead of needing a separate guest OS).
Ask anyone that deploys fat jars to run on the JVM. They've been leveraging containers for ages now. Ask the golang folks how they like deploying single binaries. The OS is not going away. Better process isolation is the future so I'm gonna say the author has a slightly too futuristic view of things. The current container ecosystem and the churn around it is a half-measure for better process management.
The mainframe industry already went through this years ago. They invented virtualization and leveraged it to transition through major new CPU architectures, complete re-designs of IO stacks, and switching out entire operating systems. None of these are new arguments. Virtualization won then and it will win again.
You could say the entire history of the industry is increasing levels of abstraction; containers and virtualization are just another step along that path.
A JEE container, is already a full OS adding the remaining services that a plain JSE installation might still lack, it doesn't need yet another layer to waste resources and add more administration work in maintenance and security.
For code that cares about word size or something similarly pervasive but platform dependent there are templates and constepxr to have the compiler evaluate things. I won't be putting constants like 32 or 64 in my I will do things like "align_to(system_details::cache_line_size)" and let my compiler handle the details.
Back in those days, we were deploying code across heterogeneous OSes (not all POSIX), using the OS vendor provided compilers, which were still catching up with C++98, let alone C++03.
So this really restricts how much you can make use of the standard library and which third party libraries to use in a portable way.
Then is time for #ifdef party.
This isn't a new technique, but for some reason people adopting C++11 and C++14 seem more willing to do it than people who wish C++ was really just C with classes.
Also, write unit tests to test the class. Run the Tests in CI on every platform and a variety of configs. It is not hard and there great free or cheap tools Like Jenkins, TravisCI and Appveyor.
This was exactly part of the problem.
Most of the enterprise code I used to deal with was not even that, rather compiling C with C++ compiler.
However my point with rich runtimes was that you shouldn't bother to even do that, the runtime is the OS, kind of.
Nothing forces you to start using non-portable code to reach more devices. It's strictly an option, and you can emulate a rich runtime by always answering "no".
So where can I get PS4 BSD or those Google layers not available in the AOSP repository?