DIOS: Data Intensive Operating System

Cautionary note: the APSYS paper was written in April 2013 -- at the time, there was little more than the idea of "maybe someone should look into distributed OSes for data centres". Much time has passed since, and there is now a DIOS prototype as well as far more design work. Things have concretized significantly since.

The target environment for DIOS is a shared, multi-user, multi-job cluster, such as Google's warehouse-scale data centres: an environment where many different tasks run, which selectively share data but which must also be isolated from each other. (Consider, for example, a front-end web server that obtains records from a back-end key-value store, and which produces logs that are later processed by batch jobs.)

"Data intensitivity" comes into this by virtue of the scale of the overall system (thousands of tasks/machines), but also via the applicable optimizations. For example, one key optimization is caching of data in memory (cf. Spark, Tachyon). True, this can be done in middleware (as these examples show), but arguably the OS already does it (viz. the buffer cache). Unifying the OS notion of caching (usually at inode/block level) and the distributed system's notion (at object level) seems like a reasonable proposition. Likewise, the OS kernel scheduler's long-term load-balancer and the cluster scheduler likely have information that they would benefit from sharing (e.g. about the interaction of different tasks sharing a machine).

DIOS is a research project that aims to find out if there is gain to be had from changing the OS -- we set out to answer the question, rather than knowing what we wanted to find! (The APSYS paper was testing the waters in terms of what other people think, and we got some useful feedback from it.)

So far (bearing in mind that the work is ongoing), it looks like the compelling advantages from changing the kernel abstractions are:

1. Complete control over data-flow: being able to actually enforce policies like "the task may only use its inputs to deterministically generate its outputs" (as commonly assumed -- but not enforced -- in systems like MapReduce, Dryad, CIEL etc.). When running an application with DIOS system calls only, we know for sure that there is no way I/O can have happened other than via these system calls and on the objects exposed to the task. This is something only the kernel can ensure.

2. Opportunities for end-to-end performance optimization: by having clear semantics of abstractions that are valid across machines, we can move away from the conventional wisdom that the OS should treat all communication equally. Concretely, the buffer copy implied in using BSD sockets for network communication and the generality of a kernel network stack are two examples of one-size-fits-all OS design that we can side-step: in one use-case, we fast-track UDP packets destined for a coordination service in user-space through a network stack bypass that delivers them at far lower latency. DIOS can make this optimization because it has semantic information about the low-latency needs of the application available. (For sure, there are other ways of implementing this, but they either involve kernel changes or give up the ability to track network data-flow [e.g. kernel bypass solutions]).

3. Convenient removal of scalability-inhibiting abstractions: it's well known that some POSIX APIs induce poor scalability (see e.g. the Commuter work from MIT in SOSP 2013). One key example is the notion that FD numbers are allocated in a monotonically increasing sequence (thus requiring synchronization between threads). By redesigning the abstractions for data-intensive applications, we can fix these sorts of problems in passing.

Hope that helps!

Agreed -- the reliance on Linux is definitely something that we've found to be a mixed blessing.

We actually looked fairly seriously into other candidate OSes as starting points for building DIOS. We considered L4::Fiasco, xv6 and Barrelfish, and investigated Barrelfish in depth. (L4 and xv6, at least at the time, were available for 32-bit only, which clearly wasn't going to cut it for a warehouse-scale OS.) Ultimately, we went with Linux over Barrelfish due to the more comprehensive driver support, better documentation and the fact that it works on our test machines.

Barrelfish had great trouble booting on one of our machines with a lot of physical memory; after patching it, we managed to boot it, but did not have PCIe devices available (required for networking). To their credit, the Barrelfish team were very supportive, but progress was slow and working with Linux allowed us to get started right away.

However, using Linux has also bought us into a bunch of annoying non-scalable implementation choices. For example, DIOS relies on fast memory mapping, but Linux serializes all access to the mm_struct for a process using a single semaphore.

That said, DIOS is (deliberately) written in such a way that it should be possible to port it to other host kernels. Specifically, the kernel code consists of three parts: 1. a small patch to route the new system calls to handlers; 2. a BSD-licensed core module that contains the DIOS logic, but which is implemented in standard ANSI C and does not rely on any Linux-specific kernel features; 3. a GPL-licensed "DIOS abstraction layer" (DAL), which offers access to the Linux kernel facilities for process and memory management, VFS calls, etc.

While our current prototype is for Linux only, we intend to revisit the Barrelfish port and will also look into porting DIOS to a BSD OS in the future. Barrelfish especially should be interesting -- it's a very good fit for DIOS's abstractions.