Comment by melted | Hacker News Reader

Comment by melted | Hacker News Reader

Parent Full thread

To understand scenarios under which Docker/Kubernetes/LXC are useful, you need to understand the environment in which Linux containers originated. They were born in Google data centers, where engineers may routinely need to reliably deploy tens of thousands of preemptible nodes and tie them together into a service by providing health checks, monitoring, endpoint enumeration, resource limits, isolation, and so on. Docker/Kubernetes let you do exactly that. At Google, though, Borg is run by SREs, and engineers don't have to worry about managing it, so it is quite economical to spin up even single task jobs there. Google also makes deployment much easier by using static linking, and structuring their build system outputs in such a way that they can be either easily deployed by copying over or packaging into an official, versioned deployment package (+providing command line flags through Borg config files). When tasks/jobs go away (get preempted or killed -- servers in this environment usually do not support orderly shutdown), whatever they wrote to local disk gets cleaned out, including binaries and data files. Persistent data is written to persistent, distributed storage backends, where it belongs.

As you can see, most parts of this picture map nearly exactly to how Kubernetes/Docker are supposed to be used. Used in this way to manage large deployments, containers provide an unbeatable value proposition.

I have zero interest in understanding how Google or Facebook or Microsoft or Apple or IBM or even Twitter does anything right now. They are big. I am not.

These companies have deep-pockets of capital to spend on such luxuries as individual contributors who can make it their jobs to deep-dive into single, esoteric, poorly specified tools. I do not.

Late September last year, Twitter laid off 336 people. In 15 years, I've never worked in a company that had more than even 150. A company the size of Google could probably hire and fire more people in a single year than I've ever been in professional contact. I cannot.

So how they operate, from soup to nuts, probably has zero bearing on how I should operate.

Put another way: Google has built several generations of internal PaaSes.

This unblocks continuous deployment at the final step, so latency from idea to production falls from years/months/weeks to hours. Or minutes.

Docker's had an interesting life: they built a PaaS, discarded the PaaS, now they're building a PaaS. Because that's what most developers actually need for their daily lives.

It turns out that tinkering with V8s is a lot of fun for a lot of people, but most drivers just want to know how to turn on the car and have the same basic interface work for any workload: wheel, accelerator, brake.

Disclaimer: I work for Pivotal, which is the majority donor of engineering effort to Cloud Foundry, a PaaS inspired in part by Google's experiences.

I wouldn't quite go as far as to call Borg a PaaS. It's at a somewhat lower level, closer to IaaS. You get a known quantity from it in the form of a stable, tuned, stripped down underlying Linux image, plus a relatively small set of services, but you can deploy pretty much whatever the hell you want without a lot of constraints a "true" PaaS would force you to accept.

My understanding is that BOSH is the closest to this from the Cloud Foundry ecosystem.

This mirrors my understanding: that Docker is awesome if you want to build your own PAAS. Otherwise use CF, Heroku or whatever.

"Google makes deployment much easier by using static linking..."

Unless you only are referring to only the Go compiler, can you provide a cite for this statement?

I also use static linking as much as I can. Sometimes it requires quite a bit of work due to how open source developers structure their build systems. I wonder if some folks at Google have also spent significant amounts of time unravelling idiosyncratic build systems found in open source projects to make static, portable binaries.

I often see commenters on HN and elsewhere making commments against static linking. It would be useful to be able to cite to Google's internal practices; these commenters also seem to attribute superior knowhow to Google staff.

They actually don't completely statically link everything - there is a small set of base libraries (glibc and I forget what else), and generally all compiled dependencies other than those from the base runtime are linked into a binary.

Google released a bunch of their main build system recently, so you can see various hints about their internal practices there, starting with the documentation: http://bazel.io/docs/be/c-cpp.html#cc_binary.linkstatic The OSS bazel defaults to linkstatic=0, but I think internally they use linkstatic=1 there. "mostly static" mode.

That's just C/C++ of course, but a similar approach applies to Python, where there are zipped single-file executables that contain all of a python app's dependencies except for python itself and things like libc6. There's a primitive form of that in bazel at https://github.com/bazelbuild/bazel/blob/master/tools/build_.... Twitter's "Pants" build system is ideologically descended from Bazel, and along with it came the Pex python executable format: https://github.com/pantsbuild/pex

Just the fact that Go does mostly-static linking by default is a decent hint in this direction, considering where Go originated.

In the end, neither approach is ideal for all scenarios. General purpose linux distributions have various very good reasons to prefer dynamic linking wherever possible. Cluster operators and anybody distributing binaries for multiple environments have lots of reasons to prefer static linking. It's not a decision to be made in a vacuum.

There's a simpler reason to link nearly everything statically. Besides libc, almost everything is built from source, and not just from source, but from source as it exists at the current revision. So compatible re-use of prebuilt *.so's would be a nightmare and you have two realistic choices:

1. Build them all at current revision and package them as a subtree. This is utterly pointless since due to containerization there will be no library sharing anyway. The only real reason you might want to do this is to save time on static linking.

2. Link statically and ship a single (often enormous) binary with just the stuff you need, saving a bit of disk space, and a bit of time at runtime. Fewer moving parts, what's not to like?

I agree. Thank you.

http://static.googleusercontent.com/media/research.google.co...

"Borg programs are statically linked to reduce dependencies on their runtime environment, and structured as packages of binaries and data files"

I really can't shake the feeling that too much of the Linux community, that has final say about the user space, is off doing some kind of architecture astronaut impression in an attempt at bootstrapping the next Google.

well, except that openvz is an example of linux containers for which Google can't take credit.

Reply on news.ycombinator.com