Hacker News Reader

My team ran about 50% of a Fortune 500 software company on CoreOS up until a few months ago when we migrated to Flatcar. We even paid a lucrative support contract while we ran it. I can count the number of major OS issues we had with it over several years on one hand- and we've had even better success with Flatcar so far.

Calling CoreOS unsuccessful is a massive misunderstanding of the market.

Chris from Kinvolk here. Happy to see you're having success with Flatcar. We, of course, agree that CoreOS Container Linux was a huge success. The uptake that we've seen in Flatcar usage, especially since the CoreOS EOL date on May 26th, has been extraordinary. So from what we a can see, the market is there for a minimal Linux for containers and and we're happy to continue filling that need with Flatcar.

Do you have any resources for understanding OSS/FOSS around minimal Linux distributions for containers?

I'm not exactly sure what you're asking. But Flatcar Container Linux is completely open source. In fact, everything we do at Kinvolk is. We do not build open-core products.

For example, we've gone a step further than CoreOS did and have a fully open-sourced update server, Nebraska (https://github.com/kinvolk/nebraska). We also generate a list of contents and licenses for each build. Here is an example from the most recent stable: https://stable.release.flatcar-linux.net/amd64-usr/2512.2.1/...

That sounds really interesting! I currently work on a side-project which is a distributed application and I plan on incorporating kubernetes/cloud-native into some of the core concepts, so I will check it out once I'm at this part of the implementation!

Maybe you're right - it's not unsuccessful after all, it really is a bullshit project... Now _every_ part of the OS is in a container! Wheee!

https://en.wikipedia.org/wiki/Container_Linux

(for those who don't know, it was renamed from CoreOS to Container Linux)

Would you please stop posting flamebait and unsubstantive comments to HN? You've been doing it a lot, and we ban such accounts.

https://news.ycombinator.com/newsguidelines.html

1. I repeated an argument made in the article itself. That cannot be considered flamebait.

2. I've always expressed strong opinions in my posts. Sometimes this results in multiple upvotes, sometimes in multiple downvotes.

3. About being "unsubstantive" - I underscored the main concept/rationale for Core Linux, and linked to its Wikipedia page. You could argue that it's not substantive _enough_, but when you phrase it this way it's as though you merely dislike the fact that I repeated the inflammatory term from the article.

Your company running on it (and not flatcar, heh) is what shows a massive misunderstanding of the market.

Just because people buy snake oil, it doesn't make it a good product.

We've been running CoreOS since before Flatcar _existed_. And we seem to be doing just fine on the OS.

I want to make constructive comments about this blog post, but the way it conducts itself makes doing so extremely difficult because nothing about it is constructive. This is troublesome for HN - I almost wish I could flag it for being impossible to discuss without turning into a flame war. It feels like several disjointed complaints based on a desire to jam any and all inconveniences into a unified narrative about Modern Software and Google. Dismissing CoreOS is the opener, but there’s almost nothing on the page that doesn’t wave red flags in some way, and I’m trying desperately to be open and objective.

I have worked with (and under) big names in system software and so many of them think saying "terrible", "garbage", and "trash" will motivate people to do productive corrections or improvements. Nope.

Times have changed and the very people he criticizes are infinitely better at branding and inspiration. There's a reason they're the ones gaining traction.

Like you, I want to be constructive, and I do agree with the author's core complaints. He just doesn't realize he's actively self-sabotaging and harming his own reputation.

It is essentially a rant with not much substance. If this was a comment on HN it would be flagged probably. Like to hit at so many things in 3-4 paragraphs you aren't making an argument for or against something, you are just making a scattershot blast on like most of systems stuff today.

The author is talking about keeping things simple which is not a bad idea. I am genuinely interested in why etcd switched to gRPC?

First of all etcd API is still available on a JSON interface. https://github.com/etcd-io/etcd/blob/master/Documentation/de... and some historical discussion here: https://github.com/etcd-io/etcd/issues/1980

Many reasons to switch from JSON to gRPC: * gRPC uses HTTP/2 which means you can concurrently make multiple requests on a single TCP connection, while on a typical JSON API which probably uses http/1.1, you can't. * Type safety of protobuf types. * Bi-directional streaming e.g. Kubernetes controllers use the Watch API which notifies the object changes/add/deletes in etcd to the control plane. * Client libraries are automatically generated, and not error prone. * RPCs are already optimized for bytes on the wire efficiency, whereas JSON is not. This matters a great deal as Kubernetes objects get large in size/quantity over time but controllers still work effectively by not spending too much CPU on encoding/decoding like they do on JSON. * Similar to the previous point, most json decoders don't reuse objects, so every decoded object is a new alloc, whereas gRPC can Reset() and reuse the same object while decoding/encoding. * Builtin authentication primitives (such as JWT/tokens or even adding TLS to client and/or server). * gRPC has support for interceptors which are like middleware functions you can inject to requests/responses on both client and server-side for logging, authorizing etc.

The list goes on, but something to note is that etcd was not developed by Google (and as far as I know, not by ex-Googlers). Both etcd and gRPC are owned by the same open source foundation, so it's natural that they make use of an existing technology.

> gRPC uses HTTP/2 ... while on a typical JSON API which probably uses http/1.1, you can't.

What are you actually talking about? An HTTP API by itself doesn't actually care about which http version is used, that's something only the HTTP server that serves the API should care about (nowadays, pretty much any http server in any language supports HTTP/2 and 1.1).

> Bi-directional streaming e.g. Kubernetes controllers use the Watch API which notifies the object changes/add/deletes in etcd to the control plane.

Even HTTP/1.1 has mechanisms for that, like long polling and chunk encoding (which allows infinite streams to keep a dual channel of communication open where each chunk can be treated as a message - with "headers" and all).

> Client libraries are automatically generated, and not error prone.

This is true, but we've had similar technology since the SOAP days... you could easily do the same with a XML-based HTTP API.

By the way, most of your points are against JSON, not a HTTP API per se, which usually would allow a number of formats , including XML and JSON at least.

> Builtin authentication primitives...

HTTP also has that when you include cookies and something like OAuth/OpenID.

> gRPC has support for interceptors which are like middleware functions

This kind of thing is better done by using the specific platform you're running on (depends on the language) so I don't see something like this as something desirable on the specific RPC framework you're using. With HTTP, caching, logging etc are trivial to do and one of the strongest advantages of using HTTP in the first place, so I think you're confusingly making a point for HTTP here, unless your focus is on the RPC side of things? In which case, you would have to make a case on why RPC is a better fit than HTTP for etcd, which you haven't.

So all in all, I found your points utterly unconvincing, but presented with so much conviction that you're actually right that I could not resist to respond (even if I don't want at all to get into a pointless discussion on the merits of HTTP VS RPC or JSON VS Protobuffers)!

By using gRPC, we get much more without thinking too much. Also, bi-directional streaming is instantaneous and it is not like afterthought long polling.

For example, whole etcd API is located at https://github.com/etcd-io/etcd/blob/4c6881ffe4b3bae257c0720...

It is pretty straightforward and not too hard to understand the API. Designing APIs with protobuf makes things convenient and it brings lots of already written tools with it.

Since gRPC built on top of HTTP/2, we thought gRPC as easiest and performant way of writing a HTTP API with good defaults.

"we get much more without thinking too much" is precisely one of the reasons for the original push to REST (and against RPC at that time SOAP and CORBA and RMI etc.) was made in our industry 10-15 years ago. By falsely representing a _remote_ resource as if it was a _local_ one we open a whole can of worms; not just performance, but resilience, infrastructure issues, etc. Transferring documents over HTTP with a common language of HTTP's verbs was supposed to get programmers to model their applications as the sets of resources that they are, discourage them from making excessive round trips, make debugging easier, and make use of standard HTTP load balancing hardware.

Yes it's easy to make that RPC call. But should you?

I've been off doing other things in the intervening period, but while I had my back turned the industry seems to have turned its back on REST and gone whole hog on RPC. Again.

At first I was thinking this was just internally here at Google, where protobufs and gRPC reign supreme. But it seems to have taken hold everywhere.

What did I miss? Why have we swung this way. Again. Is the pendulum going to go back?

> By falsely representing a _remote_ resource as if it was a _local_ one we open a whole can of worms

I've heard this argument before, but how does gRPC itself cause this issue to manifest? I'm curious to hear what your opinions are on a better alternative, and how not to represent a remote resource as a local one.

You could use gRPC responsibly, yes.

But the fact that it presents the remote resource in an API which resembles a local object means that programmers often get lazy in the manner of which they think about these things. The REST semantic is supposed to make this more explicit.

A remote object is not an object in your program or even your computer. It's something you're taking from something that is computationally miles and miles away. Compared to the microseconds it takes to dispatch a local call, it's an eternity away, even on a local network of the highest speed.

Accumulate those latencies over thousands of dispatches, and trouble can ensue.

I am reminded of an observation from when I first joined Google, coming out of their acquisition of the scrappy awesome ads company I worked at (Admeld).

We had a little service that kept track of ad impression caps / budgeting. We didn't want to serve an ad a single time more than the customer wanted us to, etc. Serving many thousands of ads per second, a process distributed across multiple machines in multiple data centres, this is a bit of a tricky shared state problem. The people who came before me had designed a rather clever solution which used a form of backoff to trickle down the number of ads served as they got closer and closer to budget cap, and to synchronize this state across clusters (this was before there were Rafty services to make this kind of thing easier, BTW).

We had a bit of a show and tell with Google when we first joined. I wanted to know how they were handling this problem, since their scale was many times ours, so I asked the question and got a puzzled/annoyed look:

"Oh we just make an RPC call to our budgeting server."

Summary: if you're at Google you don't have to worry as much about these problems. You still do, but there's an insane amount of infrastructure and horsepower and an army of SREs to help make it happen.

So, yeah, my point is -- just because Google does something or has invented something doesn't mean it's the best way to do it, especially in a smaller more cost conscious organization.

Thank you for taking the time to write this up, and the interesting anecdote.

What I don't see though, is how is making a (g)RPC call any different from making a REST call? Like you said, the REST call is supposedly more explicit, but at the end of the day, it seems more of a convention and some hard underlying difference. What's the difference between `httpClient.get("...")` and `grpcClient.foo(...)`?

The latter encourages you to model things as remote procedure calls, the former, well, in my experience it's open to incompetence and abuse, too, so... ehh... but done properly... well, go read the Roy Fielding paper :-)

I mean, internally at Google we have load balancing for grpc (I'm sure the outside world does now, too, but it was new to me when I joined) -- but load balancing HTTP requests containing readable JSON or XML documents, that's far more sysadmin friendly, wouldn't you say? Off the shelf infrastructure, nothing exotic, easier to monitor. Same goes for caching, for proxies, etc.

Being able to just stick a URL for a given resource in your browser, or hit it with wget/curl to read it, that's a serious bonus.

In general URLs follow conventions similar to those down by our Unix forefathers, when they designed the filesystem API. We are all familiar with this model. And in some ways REST done right is very Unix philosophy -- provide a consistent model upon which a bunch of little tools can interoperate.

I could go on... have to go put my daughter to bed tho

Thanks again for the detailed response.

> But it seems to have taken hold everywhere.

I don't think that's accurate... as the article claims (and it seems believable) it has only appeared in etcd because of Xooglers interfering with the project in the name of Kubernetes (also from Google). Or do you actually have examples of gRPC being used in many other non-Google-related projects?

"The gateway accepts a JSON mapping for etcd's protocol buffer message definitions."

Was a "JSON mapping for etcd's protocol buffer message definitions" the original "HTTP API"?

"A Critique of the Remote Procedure Call Paradigm" (https://pdfs.semanticscholar.org/e125/57a7582881a62040eee68b...)

"A Note on Distributed Computing" (https://github.com/papers-we-love/papers-we-love/blob/master...)

Speed, efficiency, and type checking are the main reasons.

It must speak gRPC before it can be subsumed by systemd. (joking)

the need for speed, or something.

Could you share some info about the switch to gRPC, not for the authors benefit, but for the rest of HN? As someone not familiar with this story, I’d be very curious for your insights!

https://news.ycombinator.com/item?id=23835651

Entirely agreed, CoreOS Container Linux was a great project that served its purpose well. Yes it went EOL last month and we just moved to one its successors, Fedora CoreOS, also works well. The authors tone is incredibly off putting.

I didn't even know CoreOS was going away, looks like there is a Fedora CoreOS which seems essentially the same.

https://coreos.com/os/eol/

Fedora CoreOS is slightly different and not 1:1 compatible. https://www.flatcar-linux.org is a fork of the original which is actively developed.

Yeh I don't even know what I'm doing and kept a 3 node cluster running with 100% uptime of small/jokey internal apps for 2 years and everything up-to-date. Replaced one node due to failure and often pull the power from one at random as a party trick.

I think he added a sidebar to his web page.