CoreOS and the the Future of the Internet | Hacker News Reader

CoreOS and the the Future of the Internet

packet.net

116 pointsstresler45 comments

"Consensus is a problem that has plagued computer science phd’s for literally decades"

No, not really. The Paxos paper was in 1989. One might reasonably argue that Raft is an improvement, but if so it's only an incremental one - not a cure for a "plague" as the author suggests.

The same obliviousness to actual history and reality plagues (heh) the rest of the article as well. Saying that package managers were invented because "compiling code from source became cumbersome" is way off base. Tell it to Gentoo, which has long had a package manager to manage dependencies but still compiles locally. Also, the "ubiquitous overlay network that comes attached to all virtualized platforms" seems conspicuously non-ubiquitous except in the VMware universe.

I don't want to seem gratuitously negative, but the author's argument is incoherent. The conclusion that we need something like CoreOS might be correct, but counterfactual axioms do not support that (or any) conclusion. Containerization is not the solution to configuration-management problems. In fact, technologies such as Docker rely heavily on tools and lessons from the configuration-management space. Where they have abandoned that heritage, they have often left huge gaps (e.g. security). The evolution we're seeing has less to do with previous approaches not working than with combining and applying those approaches in new ways precisely because they have become robust/mature enough to do so.

Part time parliament was written in 1989, but not published till 1998. Probably not fair to claim to much recognition of its value till the mid (late?) 90's.

http://research.microsoft.com/en-us/um/people/lamport/pubs/p...

It was published well before 1998, as SRC research report #49.

http://apotheca.hpl.hp.com/ftp/gatekeeper/pub/dec/SRC/resear...

There were also other papers based on that before 1998, including the Petal/Frangipani stuff. That something already so well known wasn't "properly" published for nearly a decade was a bit strange, as Lamport himself explains in the page you cited.

Even if that weren't the case, something that was published in 1998 still refutes the "plagued for decades" claim.

Interesting. I'd thought that "The Paxos algorithm for consensus in a message-passing system was first described by Lamport in 1990 in a tech report that was widely considered to be a joke [1]" was pretty much the standard interpretation.

Perhaps i'd read the Turing award writeup to literally. "Unfortunately the setting as a Greek parable made the paper difficult for most readers to comprehend, and it took nine years from submission to publication in 1998. But the 1989 DEC technical report did get noticed. Lamport's colleague Butler Lampson evangelized the idea to the distributed computing community [19]. Shortly after the publication of Paxos, Google's Chubby system and Apache's open-source ZooKeeper offered State Machine Replication as an external, widely-deployed service."

It's a little odd, nobody seems to have actually used it before oh, say 2000 or so, but maybe it was in secret proprietary kinds of stuff.

[1] http://www.cs.yale.edu/homes/aspnes/pinewiki/Paxos.html

[2] http://amturing.acm.org/award_winners/lamport_1205376.cfm

PhDs may be only ones were both aware of consensus as a problem and not plagued by it.

As a Zookeeper user however, I can promise you that anyone trying to actually implement or use consensus was very plagued.

> "Consensus is a problem that has plagued computer science phd’s for literally decades"

No, not really. The Paxos paper was in 1989. One might reasonably argue that Raft is an improvement, but if so it's only an incremental one - not a cure for a "plague" as the author suggests.

The problem with consensus for PhDs is not that Paxos doesn't solve it. The problem is that it is SO hard to understand.

Notably, soon after Lamport's 1998 Paxos paper he published Paxos Made Simple[1] which noted The Paxos algorithm for implementing a fault-tolerant distributed system has been regarded as difficult to understand

This paper was supposed to make it simpler, but it wasn't enough. When the Raft paper was release, the title was In Search of an Understandable Consensus Algorithm[2] which noted the following:

most implementations of consensus are based on Paxos or influenced by it, and Paxos has become the primary vehicle used to teach students about consensus. Unfortunately, Paxos is quite difficult to understand,in spite of numerous attempts to make it more approachable. Furthermore, its architecture requires complex changes to support practical systems. As a result, both system builders and students struggle with Paxos

I'd say that is reasonable evidence to suggest that, yes, consensus really has plagued computer science phd’s for decades.

Additionally, I'd note that the famous maxim There are only two hard problems in computer science: Naming things and Cache Invalidation (and off-by-one errors) basically refers to the difficulties in establishing distributed consensus in both the "naming things" and "cache invalidation" cases.

[1] http://research.microsoft.com/en-us/um/people/lamport/pubs/p...

[2] http://web.stanford.edu/~ouster/cgi-bin/papers/raft-atc14

We don’t need more options. We need fewer.

We’re moving past the days where we take an operating system off the shelf and plug things into it until it can serve our application and into the days of purpose built OSes that push software complexity into the realm of software engineers.

These two statements contradict each other.

On top of that, it isn't even anything exciting, per se. Special-purpose and tailored OS have been the norm for plenty of fields, though I suppose not web application deployment.

And of course, the obligatory false dichotomy about init systems:

Just because something like systemV has worked very well for 32 years doesn’t mean it should not be torn apart and re-invented.

"Full reliance on the init suite" should rise some alarms about fault tolerance, honestly. But I don't know.

I don't really think so.

Pushing complexity into the software layer means you need fewer options at the operating system layer. I'm specifically addressing the OS.

As far as taking things off the shelf, I'm referring to a 'server edition' vs a 'desktop edition' just being a difference of installed packages. There isn't a desktop version of Core for a good reason. These are two different workloads.

And totally agreed about systemd being a bit bleeding edge. It's a new thing, and there are definitely speedbumps to be had.

Thanks for commenting. I'm loving the dialogue that is happening here.

These two statements contradict each other.

Well, if you read carefully the "we" is operations, and they're just jiggering things around enough to throw the complexities over the wall to development.

I'm with you that this (at least for the Core OS case) isn't terribly exciting conceptually. The exciting part is someone is doing the heavy lifting to introduce this level of flexibility into Linux. It brings the highly tailored OS approach to users that haven't had a business case to do all that work in house, and builds the ecosystem around the idea in a way that doesn't happen with one-off implementations.

Don't have to read too carefully. It's stated explicitly in his closing remark.

"Just because something like systemV has worked very well for 32 years doesn’t mean it should not be torn apart and re-invented."

Actually, that's exactly what it suggests, unless there's a strong reason to do otherwise. Note that terribly written desktop linux wifi init scripts do not qualify for the latter.

We're still a ways away from having to make our choice of container orchestration framework(s), but it seems like it's getting to be an increasingly difficult decision to make. There are a lot of great pieces that can be used independently, alongside one another, or on top of another.

For example, Kubernetes + Mesos, Kubernetes + CoreOS, Mesosphere + CoreOS, Swarm + whatever else, etc.

I guess it's a good problem to have, but I'm looking at this as a question of what the jigsaw puzzle is going to look like for us in six months to a year. I'm not sure I buy the notion that all of this reduces complexity, it just shuffles it around.

Yep, I'm interested in this model, but finding it very hard to effectively evaluate all the permutations. This is hugely exacerbated by the fact that nothing really seems complete, so pros and cons could either be due to different design decisions, different priorities, or different levels of maturity, and it's very difficult to tell which.

You don't need to evaluate all permutations, at least not in detail as long as you know the main selling point of each. Go for the setup that has the smallest number of components yet is complete enough for your specific case. For me - it's a single CoreOS VM, no etcd or fleet, no higher level abstractions like Kubernetes or Mesosphere. Most people who need to get a bigger system running will need to take advantage of etcd and fleet + multiple VMs but can still decide that Kubernetes/Mesosphere is too much. And then there are the people who need the full package.

I think most people looking into deploying a production system using containers will find themselves interested in evaluating the "full package", and also find themselves interested in more than just the main selling points. That's where my comment is coming from. YMMV I suppose!

For me, it was three instances with dokku-alt, and deploying exactly the same to all three... fronted by two (load-balanced) caching/ssl-terminating nginx instances. Using my cloud provider's table storage for configuration (caching in an app module, that clears it's cached results every 5 minutes)... realistically my configuration doesn't change that much.

> You don't need to evaluate all permutations, at least not in detail as long as you know the main selling point of each.

I need to know how the software works in order to evaluate it effectively. Selling points don't tell me that; they are conjured up by marketers to influence managers, not engineers.

I'm talking my own book here, but: Cloud Foundry.

It already works. You throw code at it, the code is staged, deployed, logged and monitored.

To support containers, is Linux necessary? Would it be sufficient to run Xen under a container loading system? That would simplify things enormously. Server farms need very few drivers. They don't need a command line environment, daemons, development tools, etc. Those are all elsewhere.

Administration tools don't run on the target platform any more. They're either devoted to creating containers or managing container instances on the farm of target machines. The server side could get much simpler.

I imagine that if OSv becomes popular it will look like that.

It is the revenge of the mainframes and micro-kernels.

Also a return back to the days where the language runtimes were the OS.

I love how layer shape-shifts up and down like this.

Maybe I'm off but it reminds me of 'the unix programming environment', it was a programming system, binary processes as functional primitives, PATH and environment for late binding, files as memory and communication mean.

"The language of the system" to quote clojure author.

I feel more that way in regard towards the Xerox PARC systems, but I get your point.

I know the Container world is full of idealistic developers who are busily waxing poetic about the benefits of portability and interoperability. But a Containerized app must still look to the OS for its DLLs and other underlying code.

To which end it looks like a Linux OS is required to run Apache containers, and a Windows OS would be required to run a .NET web app. Or we can run cross-OS binaries, aka WINE & Cygwin.

These files have to be somewhere. Unless we're baking them into the Container, at which point it's not much different than a VM.

Administration tools haven't been tied to the target platform for decades. A great many green administrators still connect directly to the platform to administer, but the tools to do the work from your own system have existed on almost every platform for a very long time.

Server farms need about 1 driver per microprocessor in each server's chassis. But this isn't about drivers, it's about not having to think about the part of the system you don't want to care about, to reduce your cognitive load.

"Unless we're baking them into the Container, at which point it's not much different than a VM."

A little more than a VM, but not much more. A file system (which might be on another machine) and a connection to a network interface are all that are necessary. .dll and .so files are part of the application, not part of the OS. Besides, static linking is making a comeback; Go and Rust are usually statically linked.

A docker container is basically statically linking your binary one level higher. Rather than just including the libraries, you include all the support files and other processes. A VM is then like the next level above that.

To support containers, is Linux necessary?

Nope, it's just software. With enough money, you can do whatever you want. Linux isn't necessary, nothing really is, but it's a very practical choice. The shorter path is to strip down linux as much as possible and then diverge from traditional use, rather than build something entirely new from scratch. Plus you have a rich environment from the get go, and can share platform advances back and forth with ease.

"share platform advances back and forth with ease."

Which means "do system administration". The advantage of not having much machinery underneath is that it reduces the need to change it. It also reduces the attack surface for security holes.

Which means "do system administration".

Not really, to me I mean code reuse. The article is CoreOS, so lets look at that context. Right now if a performance bottleneck is fixed in the network subsystem of the linux kernel, CoreOS feels that benefit. But micro-kernel-deisnged-for-running-containers-on-top-of-Xen-OS won't share that benefit. Of course one could argue that MK-X-OS doesn't have the bottleneck in the first place, and maybe it doesn't, that's not my point. I'm saying it's duplicated effort.

I don't know what you're on about with too much machinery underneath, I'm not saying to keep libfoo and interactive logins around if you don't need them. Just that nobody has to reinvent the libbar wheel.

The industry swings back and forth between coarse-grained and fine-grained solutions, centralization and decentralization, development and operations, depending on the currently available technologies.

Build a coarse-grained system, and you find excessive coupling. Build a fine-grained system, and you find excessive moving part interactions. Which kneecap gets the bullet?

Linux is increasingly become more 'appliance-like' and I think the steps taken here are actually positive. Too many shops flat-out ignore the OS compoment of the stack. This will at least force integrators to think about the OS more.

For what it's worth, this is just SmartOS re-implemented in Linux (though lacking certain features of SmartOS), and funded by Google with apparently more flexibility on where you can run this.

I personally cannot wait for the day when I am able to treat Linux like a dumb appliance everywhere, running atop more purpose-built systems.

It gives you the bare minimum you need to schedule and start containerized workloads and leaves the rest to however you want to build and deploy your software

Isn't this a bit of pushing the problem somewhere else? It seems that complexity will be "load balanced" to all the containers owners, which is great for infrastructure owners but does it "solve" the underlying problem (as it's stated in the article)?

Yes, exactly. That is what the final sentence was meant to indicate,

"We’re moving past the days where we take an operating system off the shelf and plug things into it until it can serve our application and into the days of purpose built OSes that push software complexity into the realm of software engineer."

I wasn't trying to indicate that we 'solved' this, but rather that the approach of managing software in the OS layer was putting it in the wrong place.

Configuration management is definitely still a problem. But I think handling it in a compartmentalized area separate from the base OS makes sense.

Thanks for the feedback.

Container-friendly OSes are definitely the future. While CoreOS isn't the only one (Atomic, Photon, Snappy), there's very little need for most OS features that we traditionally expect in servers when lightweight containers wrap all our applications, and cluster schedulers manage failover.

As for load-balancing the problem, there actually is added complexity in new architectures because we add a layer of orchestration on top. However, that layer on top is becoming more and more necessary, so it makes sense to move a lot of functionality out of the OS layer.

If you want an example, think about how modern disk systems expect JBOD instead of RAID. When there is innovation on top, the underlying system can become simpler.

Seems like a trend these days. Tired of dealing with a hairball of code? Put it up a layer and let them handle it instead...

I'm seeing it with TTYs, i'm seeing it with mice, it seems to be going on all over the place...

i've never once had a "page"-type problem (i.e., getting woken up or interrupted) that could have been prevented or solved by containers. usually it's the result of poorly written/tested code, malicious traffic, or a personal fuckup by someone like, turning off the wrong thing by accident.

i really do wonder what this guy is talking about. containers have a set of use cases but eliminating complexity or downtime sure as hell aren't among them. i don't even know how you can say that with a straight face.

I really wish this didn't give the Linux community a bad case of "i have a shiny new hammer, so all my problems are ipso facto nails!".

What is CoreOS' relationship with Google? Wikipedia says CoreOS is a fork of Google's Chrome OS, which is itself a fork of Gentoo. Chrome OS seems like an odd choice for a server product. Why not use something simpler?

CoreOS received funding from Google Ventures and uses Google's Omaha updates distribution system and Kubernetes cluster management utility.

One of the major ideas behind CoreOS is that system updates should be frequent, automatic and reliable. As such the self-updating Chrome OS was chosen as the starting point.

https://coreos.com/using-coreos/updates/

co-founder of CoreOS here.

We chose to fork from Google's ChromeOS early on because they had a nice SDK setup for taking a set of Gentoo ebuilds and taking them through a pipeline of cross-build, image build and then OEM build steps. Because CoreOS is more like an embedded Linux than a normal general purpose one I was evaluating different options like Tizen, OpenEmbedded, etc but I arrived at forking the ChromeOS SDK because ebuilds were familiar and the build pipeline was close enough to what we needed.

Since forking we have essentially diverged in every possible way a Linux can diverge: different Kernel, different init system, different bootloader, etc. So, even though technically it is a fork, and arguably the Wikipedia should be updated, it was a hard fork to get a set of build tools.

If you are interested you can see our SDK guide here: https://coreos.com/docs/sdk-distributors/sdk/modifying-coreo...

Thanks. I didn't realize it was a hard fork. I guess you've got to start somewhere. :) And as an OS company, you would want full control of your OS distribution and not depend on upstream's decisions.

What about ARM? Seems Ubuntu Core (Snappy) is way ahead of CoreOS.

So CoreOS is cool because it's minimal? Serious question, why is CoreOS a better container host than say, boot2docker?

(CoreOS employee, would love to hear other opinions)

Three main things:

1. everything that systemd provides (dependency chain, common logging, cgroup management, socket activation, timers)

2. designed for sane management from 1 to 1000s of machines. Easy PXE booting, etc.

3. self-updating (you can turn this off!). State of the art for servers is "never touch it again" or a multi-day fire drill to patch hundreds of systems for the next heartbleed.

Basically, CoreOS should remove a lot of the tedious upkeep tasks from your ops team so they can focus on moving your infrastructure forward.

Anyone else read this title and think "that seems awfully self-important"? And walk away from the article wondering how the future of the whole entire INTERNET is at stake?

Reply on news.ycombinator.com