kelseyhightower

We all want a strong foundation and will continue to work to create one. In parallel, many of us like building and sharing things, and given that most things in Kubernetes are just abstractions, the things we are building today will continue to work tomorrow, even if the "secrets" backend is reimplemented on top of something like Vault.

I'm hopeful people find things like konfd useful, even if you only use configmaps and avoid secrets entirely -- a use case supported by konfd.

First, thanks for taking the time to offer feedback regarding what we call "secrets" in Kubernetes. As you've rightfully pointed out the current implementation of "secrets" are stored unencrypted in etcd; something like Hashicorp's Vault [1] would be a much better choice for the given use-case.

I've already began exploring how an integration between Vault and Kubernetes would work [2]. There are some things to work out, but those discussions are under way [3]. The current prototype/example demonstrates one way of leveraging Vault from applications deployed on Kubernetes, without changes to the Kubernetes core.

While the vault-controller [2] works, we can do better. One idea is to consider what a deeper Vault integration looks like. Ideally we can modify parts of Kubernetes, mainly the kubelet agent that runs on every node, to handled the secure generation and renewal of unique Vault tokens for each Pod during the Pod creation phase.

Another idea would be to rethink "secrets" altogether and rebuild the entire feature on top of the Vault API. We can "hide" Vault under the current "secrets" API, which would let us remain backwards compatible. That would be phase one. Phase two could introduce new Secret extensions, which would enable users to declaratively manage Vault tokens, backends, and secret renewals through the Kubernetes API.

[1] https://www.vaultproject.io

[2] https://github.com/kelseyhightower/vault-controller

[3] https://github.com/kubernetes/kubernetes/issues/10439#issuec...

This is great feedback. I'm still on the fence about a watch mode for the following reasons:

1. The template configmap will not change as often as the secrets and configmap key/value pairs they reference. This means you'll always need a sync loop to regenerate the template and compare it with the current generated version. This is how most configuration management systems work. Maybe we can watch the data and uses changes as triggers?

2. Watch is a bit racy when a template references multiple secrets and configmaps. It's really hard to prevent a partial config. One idea is to scale the konfd replicaset to 0 before making changing a bunch of secrets and configmaps, then scaling konfd back to 1 to trigger the processing of the templates to pick up the new values. The other option is to wait the "sync-interval", but you'll run the risk of partial configs or needing multiple runs to generate the complete config.

One possible solution to prevent 1 and 2 is to only reference one external configmap or secret for each template resource. I'm still testing these patterns in real life and will update the docs on which patterns work best for each situation. With that said, I think I'll still add a watch mode so others can test and provide feedback.

I think sidecars accomplish similar goals by moving the LB logic closer to the application. So now the question is: Why would you want to move this kind of thing closer to the application?

Control.

Client-side load balancing enables you to adopt application specific LB logic. For example, when using a RPC framework like gRPC your application MAY prefer reconnecting to the same backend when a temporary failure occurs while streaming between two endpoints.

Client-side load balancing MAY give you more flexibility in terms of adopting a different LB strategy per application. I know some centralized LBs support this, but that often requires configuration changes and MAY impact other applications sharing the centralized LB. A new LB configuration may require more memory or trigger some obscure bug. This all or nothing approach usually means most applications standardize on a one-size fits all solution.

For some people client-side load balancing is much easier to understand. While many platforms hide internal proxies, DNS lookups, etc, those things MUST be understood to fully support any application that pushes data through them. Everything is fine until something goes wrong or when performance does not met your business requirements.

Client-side load balancing does not solve every problem, but it's a tradeoff that MAY yield improved performance and reduce total complexity.

Short answer is no. I'm a Google employee, so I'm biased.

The Container-VM is just a Linux distro optimized for the Google Cloud Platform (GCP). As a former CoreOS employee, I can tell you first hand that a lot of effort goes into shipping and maintaining CoreOS Linux so it works well across every cloud provider and bare-metal target. This is not a goal of the Container-VM.

The Container-VM also reduces the bootstrapping overhead required to run containers on GCP by including Docker, Kubernetes, and any patches required to run container workloads well, and securely, on GCP.

Why not use distro X?

While the majority of Linux distros do a great job of reviewing and accepting patches, there is a ton of value in producing a custom Linux distro and controlling the release cycle. A custom Linux distro gives the GCP team the ability to:

* Enable kernel features, by default, that support Docker, Kubernetes and other GCP offerings; even if those kernel features offer little value on other platforms. This ability also extends to other tools and utilities commonly found on a Linux distribution.

* Provide a better security and support story. When critical patches are required, Google has full control over when, and how, we roll them out. This is not to say other Linux distros are doing a poor job, it's to say Google should be in position to support GCP customers at this level.

The Container-VM simply provides GCP customers another option with deeper support from Google.

So how does this not result in lock-in?

Here are a few reasons:

* The Container-VM is built using common open source technologies such as the Linux kernel, Docker, and Kubernetes, with minor patches for stability and performance. These patches are usually cherrypicked from upstream or being proposed for acceptance.

* Just about every Linux distro must be customized for each cloud provider. For example, on AWS you'll want to optimize for Xen and ship the AWS SDK. On GCP your target will be KVM and the Google Cloud SDK.

Virtual machine (VM) images have never been portable, which is one reason so many people are pushing for containers to deliver workload portability across providers. The Container-VM is all about running containers, the same containers you'd run on any platform, we're just aiming to be the best platform to do that on.

I was able to contact GitHub support and reported the issue. Looks like relative links from the main page are working again. Thanks for the heads up.

Thanks for the feedback. My goal is to augment those tools with step-by-step documentation so people get an opportunity to learn how things work. Yes, I will attempt to "support" people as they attempt to learn things the hard way.

I would also like to note that I'm not in competition with those automation tools. I'm just really focused on helping people learn. Some parts of the Kubernetes community really want to learn this stuff without an automation tool so they can skill up to build their own, custom, tools that match their preferences and tradeoffs.

Happy to improve the tutorial to incorporate the production approach that should be taken for each task. The main goal is allow people to run through a cluster bootstrap step by step so they learn how things work.

If you don't mind filing a few issues on the project I'll be happy to rework each lab to start addressing these concerns. Until then I've added a note to the README regarding production readiness, but that should not prevent people from learning.

I've updated the tutorial to include AWS a few days. Based on your feedback I've added a note regarding AWS support toward the top of the project README and in the project description field.

In some cases you'll find side-by-side labs that focus solely on AWS like this one for bootstrapping the underlying compute: https://github.com/kelseyhightower/kubernetes-the-hard-way/b...

In other parts of the tutorial you'll find sections for both GCE and AWS that highlight the different commands required for each platform - only a small fraction of the tutorial requires something different between GCE and AWS. Both cloud providers are now treated equally in the updated tutorial.

Thanks for mentioning this. It looks like there is something wrong on GitHub's end. The current, temporary, workaround is to visit the README directly and the relative links should work.

I've updated the Kubernetes Certificate Manager based on the dns01 exec plugins proposal. I've also pulled out all native support for DNS providers and now the Google DNS provider requires the googledns exec plugin. For ease of use the kube-cert-manager container image includes the googledns plugin, but it's easy to now build and ship your own:

https://github.com/kelseyhightower/kube-cert-manager/blob/ma...

I'm proposing an exec based plugin system for the Kubernetes Certificate Manager. Ideally this will make it easy to support any DNS provider written in any language.

The goal is to push all DNS providers, including the Google DNS provider, to exec plugins that live out of tree. To ease deployments the "official" Kubernetes Certificate Manager container will include a few DNS providers via exec plugins out of the box. There will be instructions on how to build new containers with additional providers for custom deployments. More details soon.

See the proposed exec based plugin model: https://github.com/kelseyhightower/dns01-exec-plugins

Based on user feedback, this is going to happen. I'll need a few days to clean up and refactor things, but it should be pretty easy to add support for any DNS provider with an API.

User feedback and clear demand for other DNS providers. Once I have that I'll consider refactoring the DNS backend to support anything with an API or some sort of plugin system.

Current goals are to vet the idea of using DNS-01 challenges and get the UX right.

I've been a big fan of ipvlan since it was added to the kernel[1] mainly for the reasons you've outlined and the overall reduction in complexity compared to bridges and overlay networks. It should also be noted that ipvlan offers a bit of an improvement over macvlan because ipvlan also works in places where L2 is not an option (i.e cloud providers).

In the post you mentioned one of the drawbacks to adopting ipvlan was the lack of tooling to manage it. To address this issue, and a few others[2], Kubernetes has recently adopted the Container Network Interface(CNI)[3] standard as the solution for managing network plugins in Kubernetes. CNI ships with a few plugins including ipvlan and, maybe more importantly, the ability to allocate and manage IP addresses[4]. Over time CNI should give Kubernetes the flexibility to work well with multiple container runtimes including Docker and rkt, while allowing third party solutions to fill in the gaps around networking.

[1] https://www.kernel.org/doc/Documentation/networking/ipvlan.t...

[2] http://blog.kubernetes.io/2016/01/why-Kubernetes-doesnt-use-...

[3] https://github.com/appc/cni/blob/master/SPEC.md

[4] https://github.com/appc/cni/blob/master/SPEC.md#ip-allocatio...

Go's TLS support should be greatly improved in Go 1.6 thanks to the performance work you referenced by Cloudflare. The licensing issue was resolved and the work was merged 11/10/2016[1]

While GC will always carry a cost, great improvements have been made over the last year and continues to be an area of focus for each Go release. Rick Hudson gave a nice update on Go's GC improves at GopherCon 2015 [2][3]

[1] https://go-review.googlesource.com/#/c/8968

[2] https://talks.golang.org/2015/go-gc.pdf.

[3] https://www.youtube.com/watch?v=aiv1JOfMjm0

I don't think looking for better is the right way to go about this. There are many things that are different in Go which you might find a nice fit for a specific use case. Also, nothing wrong with learning another language to increase the number of tools in your toolbox.

I'll list a few use-cases where I found Go to be a great fit for me.

Building command line tools:

Both Go and Python are great for building command line tools. One place where Go is different is that it's super easy to produce statically linked, self-contained, binaries. The result is less dependancies to manage at install time. Many users find tools distributed in a self-contained binary easier to get started with -- no need to have the right runtime (python 2.x) installed.

Example: https://github.com/kelseyhightower/confd

Building web services:

I'm always building small services to solve a problem and this is one area where Go really shines for me. Out of the box Go ships with a "high performance" web server. Just import the "net/http" package and off you go. While the built-in web server is not a full replacement for something like Nginx (rate limiting, rewrite rules, etc), it does allow you to go really far without the need of an external web server for performance.

Example: https://github.com/kelseyhightower/coreos-ipxe-server

There are other benefits to Go like performance gains if you are willing to trade in the flexibility you get from a dynamic language like Python and migrate to a strongly typed language that must be compiled before running your code. While Go is strongly typed it does not get in the way of getting things done. The Go compiler is fast and stays out of the way.

I'm not sure what your background is, but I did a talk a few years ago at GopherCon 2014, "Go for Sysadmins" (https://www.youtube.com/watch?v=wyRbHhHFZh8), that provides a nice introduction into why I like Go and some live demos to drive key points home. Hope this helps.

You're welcome.

You have managed to capture the spirit of the post. The goal was to highlight areas where developers can take action, and how improving even the little things can go a long way to improving the entire system -- even the one you can't see.

In the early parts of my career I would often take pride in building complex systems to accommodate for misbehaving applications. Throw in some fancy Nagios alerts and a sleep depriving on-call rotation; I looked like a hero.

Then I learned how to write code.

This was the turning point in my career (2006). I was now brave enough to modify "legacy" production applications to take advantage of "new" infrastructure features like service discovery (use DNS records instead of IPs), and logging directly to syslog (asynchronously with proper ring buffers).

I was willing to learn any language too: Python, PHP, or Java, it did not matter because it allowed me to take action and contribute at the heart of the application.

I'm not saying platforms that also handle "misbehaving" applications or complex failure scenarios are unnecessary. I just consider those platforms as extra layers of protection, not a free pass to ignore building applications that take responsibility for reliability from startup to shutdown.

CoreOS was CoreOS before I got there.

I came to CoreOS because of the open source nature of the company and the awesome collection of projects such as etcd, CoreOS Linux, and their work around containers. The release of Clair proves this will continue without me.

While much of my work at CoreOS was seen through the eyes of people attending conferences and readers of my blog entires, my true contribution was helping grow a community around containers, and more importantly exposing the benefits of distributed computing to a much larger audience.

Based on feedback, I consider my community efforts largely successful for many reasons including the following:

* I've been willing to help anyone learn this stuff by doing 1:1 hangouts, engaging in social media, or writing a book on Kubernetes. Whatever it takes.

* Building and sharing open source projects like confd[1], and my collection of prototypes such as motorboat[2], or exploring new integrations between tools like Docker Compose and Kubernetes[3].

I also spent a fair amount of time hacking on open source projects at CoreOS including rkt, etcd, and working with the community to ensure they could adopt our technologies, then incorporating feedback from users when we (CoreOS) made it hard too. As a result I gained Product Management responsibilities to go along with my advocacy work.

During my time at CoreOS Kubernetes came out, which represented a turning point for what I consider the majority of the industry (well the part that cares about application containers). Kubernetes represented many of the ideas we had been working towards at CoreOS, and in many ways was the perfect addition to the CoreOS stack.

I was an early code contributor of Kubernetes, which resulted in gaining commit access. But once the number of outside contributors grew, I felt I would have a larger impact building tools to fill in the gaps between Kubernetes and people adopting the platform. One of those tools being kube-register[4], which made it easy for people to automate the scaling out of a Kubernetes cluster using fleet.

Overtime I started to shift focus towards education, because what use is a platform for next generation infrastructure if no one knew how to use it? So came the workshops, book, and more conference talks.

In many ways I'm doing the exact same things -- working to improve the future of computing based on containers and core concepts of distributed computing, but at a different company, Google, which has many of the same core values regarding open source, community, and vision.

What happens to CoreOS?

I expect CoreOS to keep shipping stuff based on the same core values that attracted me there in the first place, and regardless of who provides my paycheck I'll always be a member of the CoreOS family. This is the power of community. I never had to leave.

[1] https://github.com/kelseyhightower/confd

[2] https://github.com/kelseyhightower/motorboat

[3] https://github.com/kelseyhightower/compose2kube

[4] https://github.com/kelseyhightower/kube-register

I gotta say this is such a well written post and especially timely for me. I'm currently employed at a startup in a highly competitive market and I constantly think about "time to market", and "the competition". The amount of stress caused by those thoughts can become a distraction at times.

But the main thing that keeps me sane and motivated is the fact that we have users that love our projects. I'm in a fortunate position where I get to meet many of the people in our community, and get a constant reminder why we do what we do.

The other hidden gem in this post is the fact that each new "competitor" who enters your space can have a positive effect on everyone and actually grow the market. While this means you won't obtain a monopoly, you'll have a bigger arena to compete in.

Maybe more conversations like this one can help foster more healthy competition between startups competing in the same space.

I hope not!

I can tell you first hand the majority of our team members work on our opensource projects. It's one of the main reasons I came to work for CoreOS over a year ago. But the concern is still valid -- will CoreOS really keep funding our opensource projects or will CoreOS pull a fast one and go open core?

So far our actions prove we are committed to opensource and that continues with todays announcement. We wanted it to be really clear that CoreOS is all about opensource projects and collaboration, even if that means our competitors can compete with us, that we took major effort in keeping the two brands separate. There is coreos.com for "Open Source Projects for Linux Containers", and tectonic.com that combines those projects in a commercial offering. There are some non-open bits in the commercial offering, but they don't conflict with the opensource projects.

You are right that we will allocate more resources towards making money, which is required if we want to continue the good work we are doing on the opensource front. You are also right in that the community will pick up the slack, but please keep in mind that CoreOS is part of that community and we plan to continue leading the way.

It's true, the CoreOS team has been moving really fast. It's what startups have to do in the beginning -- before the money runs out :). While our announcement today introduces Tectonic and some funding, it also communicates that we have found our market fit. This means we have the right set of features to go to market with, and add value by making people successful with containers.

Now it's all about execution.

CoreOS is now focused on balancing new features with maturity and stability. For many of our projects this focus started almost a year ago. Take etcd for example. CoreOS spent the last 9 months effectively rewriting etcd and it's raft implementation to meet the stability requirements we need going forward. Other projects have demonstrated great signs of stability such as CoreOS Linux, our Linux OS focused on containers. It's not perfect, but there are many people that run CoreOS Linux in production today. We expect this trend to continue.

I should also note that we have many more team members employed by CoreOS than we did a year ago. It takes time to build the right team, but so far I think we have done a fantastic job. I'm still amazed at the number of projects we actively maintain and ship with a team our size. Now we have the people power to make our projects/products solid.

Thanks for raising the question and hopefully we can continue to answer it with actions and shipping products that you will fall in love with -- or at least get the job done without the pager going off.

This is something that all companies that set out to build great products have to figure out. Do you take the Apple path and attempt to build the best product possible, even if that means limiting where and what it runs on?

While we try to answer that question, I'm pleased that CoreOS has taken great effort to keep our opensource projects[0] separate from our commercial offerings[1]. This allows the community, and even our competitors, to support the "chose your own fundamentals" use case. Tectonic simply represents the CoreOS fundamentals, which we believe encapsulates the right patterns that will lead a large group of people to success.

[0]https://coreos.com

[1]https://tectonic.com

This is a great question.

CoreOS is currently working on adding support for Rocket[0] to Kubernetes. There is still a lot of work to do here, but things are progressing nicely. Kubernetes is a true opensource project and accepts contributions that makes sense. Supporting another container runtime engine makes sense in terms of choice for the community and forces clean separation between Kubernetes, the cluster manager, and the underlying container runtime engine.

When it comes to Rocket vs Docker I think the competition will be good for both projects and the community as a whole. At the end of the day, both projects and the communities behind them want people to be successful with containers.

[0] https://github.com/GoogleCloudPlatform/kubernetes/pull/5518

It's better to compare RedHat's Atomic Host with CoreOS Linux. Both projects are focused on providing a slimmed down Linux OS focused on running containers in a server environment.

Tectonic would be closer to RedHat's Openshift project/product, but that's not quite right either since Tectonic is not a full blown PaaS solution.

Kubernetes is the perfect complement to CoreOS, and it's the type of cluster manager we would have built over time -- thanks Google!

We started building fleet from the ground up with plans to build a higher level tool on top, which would provide the application centric workflows you see in Kubernetes. But like all decisions at CoreOS, we only build stuff we have to build. When Kubernetes came out we quickly started evaluating it for our needs. It turns out it's a great fit, which is why we built Tectonic around it.

Kubernetes adds a lot of value to the container ecosystem and is grounded in the same roots as other CoreOS projects: opensource and highly collaborative. Both CoreOS and Kubernetes want to see our components leveraged outside of our commercial products, but we also feel that we can offer the best experience by combining them into focused offerings for people looking for a complete package.

You bring up a really good point.

Many of the docs on etcd promote the use of the discovery service[1] as a convenient way of bootstrapping an etcd cluster -- really useful when you don't know the IP addresses of each cluster member up front. The discovery bootstrap method is also great for demos and testing environments, but as you correctly highlight this is not the ideal way to run a production setup.

With the release of etcd 2.0, I strongly recommend using the static bootstrap[2] method for provisioning an etcd cluster. The static bootstrap method provides the key documentation clues that help you reason about an etcd installation.

Finally, etcd 2.0 introduces support for bootstrapping using DNS[3], which provides the convenience of the discovery bootstrap, and the explicitness of the static bootstrap.

[1] https://coreos.com/docs/cluster-management/setup/cluster-dis...

[2] https://github.com/coreos/etcd/blob/master/Documentation/clu...

[3] https://github.com/coreos/etcd/blob/master/Documentation/clu...

For those looking for a little more context or who want to see the App Container Spec in action we recorded videos following the initial announce of Rocket -- the initial implementation of the spec.

Rocket & App Container Spec Overview by Brandon Philips - (https://www.youtube.com/watch?v=GYgpaAZMe-E)

Rocket Tutorial and Demo by Kelsey Hightower - (https://www.youtube.com/watch?v=E9WVjxaRkKU)

Google Hangout with CoreOS: Rocket and the App Container Spec - (https://www.youtube.com/watch?v=U3UmFQbUsN8)

Disclosure: CoreOS employee.

The application container specification (ACI) and the initial implementation, Rocket, are here to provide balance to the container ecosystem. If you review the draft spec (http://bit.ly/1s4Om8p) you can see it's clear that the proposed specification is designed to remove the need for a "vendor" when it comes to building, distributing, and running containers. As an example everything in the spec can be accomplished using a handful of Linux command line utilities such as gpg, tar, and systemd-nspawn -- you could use lxc or libcontainer as well.

When it comes to distributing container images the ACI spec only requires hosting ACI images (tarballs) behind a HTTP webserver and leverages DNS instead of custom registry APIs. The ACI spec skips the app store model in favor of simplicity -- think RPM and YUM.

docker vs rocket?

Don't really care about that right now. Nothing wrong with Docker providing value, and yes, docker provides tons of value. Rocket is just our (CoreOS) attempt at providing value in a slightly different way. IMO the only people that should "own" the container spec are the people using it, and right now I believe that is the main driver behind our (CoreOS) intentions for proposing the ACI specification.

I know I'm biased by association, but trust me, I want using containers to be a simple thing. In reality using containers should be about as exciting as:

$ yum install app; service start app

That was my first question as well, but there must be a reason why Kubernetes was brought in.