Systems software research is still irrelevant | Hacker News Reader

Systems software research is still irrelevant

trmm.net

61 pointsmbrubeck43 comments

Ugh, this old troll. Something that took me a while to figure out: when someone says that "nothing interesting is happening any longer in [discipline]", it is virtually always code for "I feel old." Yes, things change and different periods of innovation are (of course) different, but it is wrong to cast value judgement on one era over another. Speaking as an aging technologist, I can say that it can be hard to resist this -- and in particular, it's tempting to view one's late twenties and early thirties as a Golden Age, and everything later as derivative and uninteresting. If it needs to be said: for those who are making this claim, they are often projecting their own lives onto their discipline, and may be confusing their own personal Golden Age with a broader one that may or may not have existed.

My domain -- operating systems kernel development -- has been particularly vulnerable to this (in part because it has become increasingly specialized) and this is something that I have heard for my entire career. Of course, there has been tons of innovation in the kernel over those two decades (and I've been lucky enough to be close to a bunch of it) -- and no one now would seriously go back to the systems from two decades ago. (If anyone disagrees, they should kindly put their money where their mouth is and run only systems software from 1994.)

So yes, systems software is still relevant -- and so are a lot of things that might feel "done." And if you are an aging technologist like me (protected class, baby!) and you feel tempted to tell these youngsters that there is nothing new under the sun, please check yourself: yes, you should educate the rising generation about the problems that have been solved -- but never go so far as to say that there is no room left for new ideas. Aside from the fact that it's a demoralizing thing to say to a younger technologist, history will likely prove you wrong!

The argument is as silly as "we're still using Von Neumann architecture, clearly nothing groundbreaking has happened since the 40s".

I'd say it's as good as it ever was right now.

Nice selection of very different approaches: http://operatingsystems.io/

Love the work you've done Bryan, but your response above does nothing to engage the salient points made in the link && elsewhere. Yes, we have better ways of getting performance metrics than truss, circa Solaris 2.5…and thank you!

But that isn't what this is about.

Sorry, what salient points am I not engaging? The author of the linked piece is decrying what they perceive as a lack of "OS research and innovation" -- and I'm saying that OS innovation remains alive and well, if perhaps more technical and less accessible than it once was. I can of course be more concrete and rattle these things off -- but I feel safe in leaving software systems advances since 2000 as an exercise to the reader...

In the link, you can see a chart, showing the hardware capabilities of a high-end workstations, sampled at roughly ten-year intervals. In twenty-five years, hardware capability has changed dramatically, while the kind of software used to get work done remains static (mostly).

Why is that? Would you make the chart differently?

The utah2K paper referred to in the link is admittedly (by the author) a polemic, so readers must understand that the writer may be shading things in a way more subjectively negative. Of course, that doesn't make it a trolling attempt, unless you can think of a particular chip the author wanted to knock off of a particular shoulder...

You acknowledge that OS innovation "may be more technical and less accessible than it once was". Why is that? In what manner? I don't think that you and Pike are using "OS research & innovation" to refer to all of the same things at the same time.

Here (from utah2K):

"New operating systems today tend to be just ways of reimplementing Unix. If they have a novel architecture -- and some do -- the first thing to build is the Unix emulation layer.

How can operating systems research be relevant when the resulting operating systems are all indistinguishable?

There was a claim in the late 1970s and early 1980s that Unix had killed operating systems research because no one would try anything else. At the time, I didn't believe it. Today, I grudgingly accept that the claim may be true (Microsoft notwithstanding)."

...I can't see why that isn't as true today as when it was written fourteen years ago. You can say that our Unix systems have improved, but that's a matter of degree, and not of type. We've also made great progress in breeding lean pigs that grow to marketable size in record time...but no matter how fine they are, they are still pigs, and aren't lobsters.

This is non-sequitur, mainstream hardware has only evolved as well so why would there need to be a "forget everything you know" revolution? Conversely, is hardware research dead?

The biggest change from 2000 is prevalence of NUMA, and this was even available back then in SGI and Sequent/IBM systems. I don't understand why systems software would have to substantially change until something like memristors drastically change systems architecture.

For a while there has been less emphasis on individual systems and more effort put into distributed systems.. first with Hadoop style clusters and we're now breaking into Mesos style clusters that provide OS-like semantics and services over a network of systems.

But there's a lot of good systems research going on today. Some of it is additive, some subtractive, but it's exists and is important.

A lot depends on how "systems software" is defined. Very few people have much use for a whole new kernel. Even little pieces of new kernel code are less useful than before, as various functions (e.g. storage) can be handled just as well in user space. So if "systems" == "kernel" then relevance really has been declining. On the other hand, if one considers "system" to include user-space storage and networking, distributed consensus and service discovery, package and system management, JIT and GC in language runtimes, etc. etc. etc. then we're actually in a bit of a renaissance.

The fact is that kernel/user hasn't been a very interesting boundary for a long time. There are hundreds of systems-programming problems that can be solved on either side of that boundary, except that if you try to solve them in the kernel debugging will be harder and you'll get embroiled in the constant turf wars that are the hallmark of a declining specialty. The idea that only kernel hackers do systems programming needs to die.

Does this mean we probably need a new, more minimalistic concept for a modern kernel?

We've been moving that way for quite some time, despite Linus's historic dismissal of microkernel ideas. Granular capabilities, FUSE, namespaces, cgroups, containers etc. all take us in that direction. Over the next several years, I think even the kernel grognards will start to see that the "center of gravity" for systems software has shifted to better languages and tools than are available in their sandbox.

There's been plenty of designs which are minimalistic, but haven't caught on for a variety of other reasons. See exokernels, for example. They're basically the minimum possible kernel.

This is exactly right. The OS isn't a very interesting system any more. Unfortunately the author still identifies it as the system of interest. For example:

> A quote that still holds true, "There has been much talk about component architectures but only one true success: Unix pipes. It should be possible to build interactive and distributed applications from piece parts."

This is completely false. REST/JSON has been fantastically successful.

There is a huge amount of relevant systems work going on, and distributed systems research in particular has never been more relevant. Major systems today (Cassandra, HDFS, Riak, etc.) are directly informed by research. It's just that no-one is really interested in ttys and pipes anymore, and that's what the author identifies as systems.

>This is completely false. REST/JSON has been fantastically successful.

I am a little unclear as to how REST/JSON competes with Unix pipes. Can you clarify your thinking here?

It's not competition. It's just that the components people wire together aren't command line processes but web services, and JSON/REST is the de facto standard. The author is still concerned with OS processes when the field of interest has moved on to distributed systems.

Plan9 is more into distributed systems than everything written with json/rest.

The distributed system of the fairly near future is json/yaml over websockets/http2/spdy. All accessible from the command line or other.

9fans will bemoan the bloat but it will work (mostly :-) and will be popular.

Not sure to agree; there are quite a few interesting projects in both OS and system programming language fields. To name a few:

Mirage OS[1] and unikernels in general (ErlangOnXen, HalVM)

OSv [2]

Chromium OS [3]

Programming languages

Julia [4]

Rust [5]

D-Lang [6]

Nimrod [7]

[1] http://www.openmirage.org/

[2] http://osv.io/

[3] http://www.chromium.org/chromium-os

[4] http://julialang.org/

[5] http://www.rust-lang.org/

[6] http://dlang.org/

[7] http://nimrod-lang.org/

"There has been much talk about component architectures but only one true success: Unix pipes. It should be possible to build interactive and distributed applications from piece parts."

Java has many success stories in this area, starting with the amazing reusability of POJOs (plain old Java objects). I can take Java code from 1996 and almost always compile it and use it now without major refactoring and without having to really look at the internals. I can do it with the API docs only. Beyond POJOs there are many abstraction techniques like beans that have been used to build huge software systems that have stood the test of time, albeit with a bit of clunkiness. But Unix pipes are a bit clunky too.

I coded a lot with Java years ago. It's been out of vogue for quite some time -- though it's still very popular just not trendy. Lately I've been reflecting on some of the pain points of other languages and thinking "wow, Java really did have a lot of things right."

I've also been thinking that the JVM is almost an OS, and that it sort of wants to be. There were some efforts in that direction, but none really went anywhere. It's not quite perfect, but I feel like it's the only thing we've had in the past 30 years that looks like a step toward a post-C/Unix operating system that could really work. I also feel like in the age of many-cores we could do away with problems like GC pauses completely. I could forsee a future many-core box where 2-4 (of 32-64) cores are just almost flat out dedicated to GC, JITing, tracing, optimization, etc. When you've got that many cores you could dedicate a few to the OS and not worry much about it. The benefits would be things like forgetting about manual memory management, etc.

Also note that the JVM can run more than Java. Java would just be the "core language" of this hypothetical OS, taking the place of C on Unix.

> I also feel like in the age of many-cores we could do away with problems like GC pauses completely.

Unfortunately its not a resources problem, or else there wouldn't be an issue with GC pauses on fast hardware.

There is a solution for that though, and Cliff Click of Azul Systems is willing to sell you the only pause-less JVM GC there is for the low low price of 7500$ per server (see http://www.azulsystems.com/products/zing/whatisit).

My guess is he's using lock-free data structures + a lot of cleverness.

Somewhat more than that: http://www.azulsystems.com/sites/default/files/images/c4_pap...

Including a read barrier variation and batch VM remapping operations (change a bunch of large pages, only invalidating the TLB once at the end).

In fact, there's a unikernel specifically optimized to run the JVM (as its only process): http://osv.io/

Isn't this exactly what Pike and other did not want? POJOs are not really components, you can't send them over the wire to something else, or split a system into subsystems with good security between. And CORBA did not work out. You can probably not get something like Rust to run really well on the JVM (the latter has GC and Rust tries to avoid that) etc. You can still build great system with Java but you can already do that with many other things like C++, Go, C# etc.

Xen largely came from the Systems group at Cambridge I think. Seems fairly relevant to the current state of computing to me. Not sure what is being worked on now that might be relevant in a decade and a half from now [edit to add: but there is probably something].

> Not sure what is being worked on now that might be relevant in a decade and a half from now.

Mirage OS is one of those things - http://openmirage.org - though I expect (hope) it'll take much less than decade to be relevant. Some of the same team that worked on early Xen are working on Mirage.

And there's OSv, http://osv.io/, from the people behind KVM.

I've just realised that my last sentence which you quoted could be read as saying that what is currently being worked on won't be as relevant when it was really intended to mean that there is probably some things that are being done now that will be relevant in the future I just don't know what they are.

Xen is an example, but, more generally, virtualization has been the big story in systems software for the period mentioned in this article (post-2000). VMware came out of systems research at Stanford and Xen out of Cambridge. The cloud world of AWS/Azure/GCE with tens of millions of VMs running for pennies per hour is a radical shift in how we think about systems. The new wave of Docker/containers promises to extend this idea even further up the stack. How can we call that "stagnation in software" since 2000?

Virtualization as an operating system design feature predates that entire chart. A lot of the hard problems in the Xen/VMWare research related to implementing virtualization on a processor architecture (x86) that was extremely popular, but didn't actually support virtualization. Which is just the kind of stagnation he's talking about, I think.

Hrm. I think there are some decent xounter arguments. I will agree that pure academic systems research is being superceded by industry development or academic commercialization. As a trend I think commoditization, parallelism, and management are where effort (and sucess) has gone, as opposed to novel research. Make it cheaper, wider, and possible to manage.

Influential counterpoints: distributed systems management/configuration/orchestration, distributed datastores (CAP, CALVIN, Dynamo, etc), system languages (Go, Rust), distributed processing (tilera, niagra, larrabe, gpus), network fabrics instead of trees, &stream processing jnstead of batch (kinesis, millwheel).

> , system languages (Go, Rust),

How is Go not another example of how "programming languages have stagnated"[0] ? You could argue that it is really practical and productive, but that is a separate point to whether it or not it is "more of the same".

Or maybe it is more of the same, just in the vein of languages that most people have forgotten about by now, so it is for all intents and purposes new. Certainly native compilation and some control over memory layout (I guess, with structs and such) and indirection have not been common features in recent memory (that is; those two points and automatic memory management, and overall simplicity compared to C/++).

[0] From the article:

> It included a chart lamenting the stagnation in [...] programming languages

Meanwhile, on the HN front page, we have:

https://news.ycombinator.com/item?id=8651499: Kill init by touching a bunch of files

https://news.ycombinator.com/item?id=8650952: on Linux, 'less' can probably get you owned

One can argue that the research needed to prevent those was done decades ago, but this article doesn't make that argument.

We may have hit upon a 'good enough' solution that is so hard to displace that it may never happen, but I think systems software research is very much relevant. Someone will have to polish a more robust/secure OS (may already exist in unpolished form, but possibly even must still be written) until it is shiny enough to replace the C-based (or equivalent) mess we have now.

I don't think I entirely agree - as notacoward says I think we're approaching a renaissance in certain areas.

- Rust and Go are becoming extremely popular extremely quickly.

- I think/hope we may soon see some sort of battle between OS container tech (Docker/LXC/cgroups, FreeBSD Jails) vs unikernels (Mirage, OSv, Linux unikernels) on paravirtualised platforms (Xen) as the full-weight-VM-running-Debian model ages.

edit: Oooh look, an exciting relevant systems software paper! http://anil.recoil.org/papers/2013-asplos-mirage.pdf

> Rust and Go are becoming extremely popular extremely quickly.

Unfortunately, that's yet another SV bias. Rust has hardly any adoption, and Go has moderate adoption and is growing nicely, but nothing close to justifying "extremely".

To be fair, we in the rust community are telling people to wait for 1.0 unless they are okay with constantly updating their code as we clean up the language for our first release. It's fair though to speak about momentum. Comparing Go's contributor list (https://golang.org/CONTRIBUTORS) with ours (https://github.com/rust-lang/rust/blob/master/AUTHORS.txt) has us with about 100 more than Go. I'm not saying our community is better than Go's or anything, but I'm certainly impressed a new language has been able to gain so much interest so quickly. When I first started contributing a couple years ago, we had like 15 people in our irc channel. Now we got over 500.

That makes a lot of sense to me; Rust is a much more ambitious and interesting - from a programming language point of view - project than Go. I think that it as a language would appeal much more to the prototypical "language geek" (or at least the impression that I have of such people, including myself).

That's not to slam Rust for being "over-complex"/"ivory tower", or slam Go for being "simplistic". But Rust tries to solve such a hard problem - be expressive without any unnecessary overhead, retain memory safety - that it seems to naturally lead to a design and language that appeals to people that are interested in programming languages "for their own sake". Meanwhile Go tries to be sufficiently expressive and then get out of your way to solve whatever problem in whatever domain that you're trying to solve.

Oh, I didn't mean to criticize either of those languages, just to point out that SV developers are a tiny, tiny minority among the software community, and their early adopter style is not shared by the industry at large. Switching programming languages carries immense costs (probably in the billions of dollars across the industry), and companies make it only when the benefits are dramatic and incontrovertible. Most companies wouldn't even consider switching a programming language more than once in a decade, and some would only consider such a switch every 15 years or so.

It took python about 15 years before it took off in popularity.

He seems to think progress would look like "a new wave of built-from-the-ground-up operating systems sweeping away all the old ones every decade or so". Say, if Plan 9 had become dominant in the 90s, would we by now be replacing it with something else? What would everyone moving to new operating systems every 10 years gain us?

There has been mind boggling progress since the 90s, just not in the direction he expected.

I think there is a revival of interest, as mass scale automation is changing the way we use servers, and security issues are finally changing the desktop/mobile. I found a lot of interest when I organized a conference [1] which is tomorrow (videos will be available), which si encouraging.

[1] https://operatingsystems.io/

The interesting thing about the table shown in the article is that even while the author laments it, it actually illustrates a good development: the table has an "hourglass" structure.

There is a stable part (fundamental technology such as the kernel, TCP/IP etc.) at the center which acts as an interface between the top part (hardware) and the bottom part (languages).

All innovation has to be anchored to some core set of stable interfaces into which the new innovations "plug in". The nature of OSs is such that they are naturally part of these core sets. That may be frustrating for OS researchers, but they'll have to learn to live with it.

Overall, I agree with Pike in 2000 and this article today. Systems software research has become stagnant and significantly derivative. The point isn't that a new OS would come along and sweep everything away, the point is that nobody seems to be developing new OSs at all. Sort-of.

In 2000 or so, systems research had turned almost entirely into server performance measurement and tweaks. That trend seems to have changed, but things like Xen and VMware are not really new. IBM invented virtualization back in the middle ages; most of the more interesting things going on recently have involved using the idea. Likewise, moving stuff from Plan 9, etc., into Linux is cool (very, very cool), but it's not new, groundbreaking research.

That being said, there are a lot of nitpicks one can make with this article and with Pike's original talk.

The processor speed comparison in the table is not terribly valid, in my opinion. The more important number is front-side bus speed, which has not changed anywhere near as much. (Front-side bus speed dominates most of computing loads I see, and a great whopping chunk of the work in those areas where it doesn't dominate, video games and data-heavy computing, is in preventing the front-side bus from being the bottleneck.)

Pike was wrong about the language environment in 2000 and the table is wrong today. Heck, he might have been wrong about 1990. Programming language research in general and its specific applicability to systems (Hi, ATS!) has hotted up a lot since the '80s and '90s.

Pike's comment, "There has been much talk about component architectures but only one true success: Unix pipes. It should be possible to build interactive and distributed applications from piece parts," bothers me a lot. A great deal of component research has been a dead-end in general (See that pile of crap with CORBA and SOAP laying on top?) Further, I believe Pike never really understood distributed systems. I have seen Pike complain that Nagel's algorithm has no place in TCP/IP (although I haven't been able to find the reference in years), but if you want a more recent example, see Go---everybody likes CSP, right? Everyone except someone who has tried to implement a synchronous rendezvous on a network.

Finally, Plan 9 didn't fail. But it also didn't have any chance to succeed, if by success you mean taking over the world. It was a great research project, but I'd believe making someone who didn't grow up at Bell Labs should be against the Geneva Convention. (Anyone know when Plan 9 started supporting color?) Plan 9 is my prima facie evidence that Not Invented Here syndrome was invented at Bell Labs.

There's a lot that's wrong in here, but one thing that absolutely needs to be called out:

...things like Xen and VMware are not really new. IBM invented virtualization back in the middle ages

Actually and in fact, what VMware did was new -- to the point that prior to this work, many reasonable people believed that x86 couldn't be virtualized without unacceptable performance penalty. Please read Keith and Ole's excellent ASPLOS paper[1] for an in-depth discussion of the novelty -- and the citation for this technology when it won the ACM Software Systems Award[2]. If you insist that VMware's binary translation work wasn't novel, please be specific about how these peer-reviewed bodies got it wrong.

[1] https://www.vmware.com/pdf/asplos235_adams.pdf

[2] http://awards.acm.org/award_winners/bugnion_3886645.cfm

The ACM citation indicates that virtualization was first explored in the 1960s; Adams and Agesen's paper cites a 1974 paper:

"Popek and Goldberg's 1974 paper [19] establishes three essential characteristics for system software to be considered a VMM:[...] In 1974 a particular VMM implementation style, trap-and-emulate, was so prevalent as to be considered the only practical method for virtualization.... We review basic obstacles to [trap-and-emulate] of the x86 architecture and explain how binary translation (BT) overcomes the obstacles..."

In other words, the contribution of that work was an implementation technique for virtualization. In fact, that particular paper exists to compare binary translation to trap-and-emulate hardware support that had recently been added to the x86 architecture. Their conclusion is that sometimes one is better, sometimes the other.

Look, I'm not saying binary translation wasn't new, or that it didn't change the face of computing as she is known---which is what the award was given for:

"VMware Workstation 1.0 spurred a shift to virtual-machine-based architectures, allowing users to efficiently run multiple operating systems on their desktops [, and allow] large-scale data-center operators [to] increase the efficient and safe use of shared computational resources."

I'm not even arguing that it didn't open up cool new research areas; I'm going to go play with unikernals as soon as I get a chance. (Even though that is a return to pre-timesharing systems.)

But we're still talking about using virtualization to run legacy operating systems on modern hardware. Just like VM/370.

    init cms

Reply on news.ycombinator.com