Kubernetes 1.10 released

> Is there an easy way to get a single-node production-ready Kubernetes instance?

Not really. There are plenty of ways of getting a single node instance. None of them will give you a "production-ready" one, because they don't define it that way (and I happen to agree). You can of course do whatever you want.

Since you are using VMWare anyway, why can't you spin up more VMs (maybe smaller)? You can vmotion them away to different nodes when you are ready to actually make the cluster HA. It is a really really good idea to keep master and workers separate, even if you run a single node for each.

Failure of the worker will of course bring down your applications. When you recover it or spin up another one, K8s will recover your apps for you. Failure of the master will not adversely affect your systems only the cluster's ability to manage itself and its self-healing capabilities (which will affect uptime at some point).

Failure of the combined master/worker/etcd node should be recoverable, but frankly, at this point, should you care? I would just shoot it in the head and add some automation to provision a brand new cluster and deploy those applications again. Since you are not worried about HA and just want a place to deploy the containers, just make the k8s single-node-cluster cattle.

Sure. Install kubeadm on the node, "kubeadm init", install a pod network, then remove the master taint

I am actually in a very similar situation. I want to run a microservices system on a single k8s node for testing purposes, put nginx with SSL in front of it and add Jenkins for automation.

Took the day to setup minikube on a CentOS server and play around, however, I wasn't able to expose anything to the outside world. Looking into Ingress at the moment, however, documentation is a bit loose there, I think.

Another comment suggests to setup a single-node cluster and remove the taint on the master, maybe I will try that instead.

edit: any advise much appreciated!

You can run Google Kubernetes Engine with a single node, just set the size to 1. It will complain about the high availability, etc. but it works just fine if you don't need that.

To the best of my knowledge, there is no such thing as single node production ready k8s cluster.

> I don't have anything that needs to autoscale or fail over or do high availability

You should not be using Kubernetes.

You could try "The canonical distribution of Kubernetes" https://www.ubuntu.com/kubernetes . That has a single node deployment using lxd, IIRC.

I think kubespray[1] does what you want, but I'm not 100% sure.

[1]: https://github.com/kubernetes-incubator/kubespray

I've built a bunch of simple scripts for this that you can find here [0]. It's not polished for public consumption or updated to k8s 1.10 yet, but it's what I use for production clusters, some of them single-node. Run the preparation script, then the master script, setup an .env file beforehand with the few required variables, and you're good. Feel free to ask questions here or in the repo issues section.

EDIT to add: It's assuming Ubuntu LTS as the node's OS, not sure if that fits your use case. Should be possible to adapt this to ContainerLinux or anything else without much trouble.

I haven't worked with GL's Auto DevOps yet, but I think the cluster should have everything necessary to get going with that.

[0] https://github.com/seeekr/kubeops

If you want a middle ground between hand-written shell scripts and full-blown Kubernetes, we use Hashicorp's Nomad[0] on top of CoreOS at $dayjob and are quite happy with it.

Similar use case - self-hosted VMs, for low-traffic, internal tools, and no need for autoscaling.

I can't speak to how well it integrates with Gitlab's Auto DevOps, but Nomad integrates very well with Terraform[1] and I'd be surprised if there wasn't a way to plug Terraform into Gitlab's process.

0: https://www.nomadproject.io/

1: https://www.terraform.io/

i would be very cautious about calling anything single-node production-ready

You can set the node count to 1 when you create the cluster in GitLab

Use GKE, it's super easy to set up and the master is free.if it's for internal tools you could even run it on a group of preemptive instances, which will auto restart and self heal when one gets terminated.

At GitLab we're making the application ready for Kubernetes, we have not switched yet. It required us to untangle many parts. For example we used to store uploads on disk before moving them to object storage, now it goes directly to object storage. There where many interactions between our applications https://docs.gitlab.com/ee/development/architecture.html#com... over disk or sockets that we need to clean up.

When its done we expect to be able to scale the different parts of our application independently. This also makes it easier to detect problems (why did Gitaly suddenly autoscale to twice the number of containers).

We have switched last year and achieved 45% cost savings. This is mostly because we can now easily run all CPU heavy activity to preemptible nodes if they are available. It is also much simpler to gracefully scale down the number of nodes/containers outside peak hours.

We also no longer have to manage servers.

All of this was possible without Kubernetes, but it is so much easier with it. Although admittedly much of the ease of use is due to not having to manage Kubernetes itself – we use Google Kubernetes Engine. I would not want to install or manage a Kuberenetes cluster/master myself.

I can recommend managed Kubernetes to anyone that runs many different apps/services.

We run all of our production backend on it at Monzo (we're a bank.) [1]

We first deployed v1.2 nearly 2 years ago, and I can say Kubernetes has made some amazing improvements in that time – in terms of functionality, usability, scalability, and stability. This release continues that trend.

We've invested a lot in it too with things like making sure our etcd clusters backing Kubernetes are really solid [2], and we've even added some of our own features to Kubernetes like allowing CPU throttling to be more configurable to get more predictable >p99 latency from our applications.

We've been through our share of production issues with it (some of which we've posted publicly about in the hope that others can learn more about operating it too [3]), but I don't think there's any way in which we could run an infrastructure as large and complex as ours with so few people without Kubernetes. It's amazing.

[1] https://monzo.com/blog/2016/09/19/building-a-modern-bank-bac...

[2] https://monzo.com/blog/2017/11/29/very-robust-etcd/

[3] https://community.monzo.com/t/resolved-current-account-payme...

One data point: I've wanted to but so far have not made much progress. I'd say my biggest impediment has been documentation: I can get it installed, but making it work seems to be beyond the scope of the documentation. I got closest once I found out about "kubespray" to install the cluster rather than using the official Kubernetes installation docs process.

I spent a couple weeks not quite full time going through tutorials, reading the documentation, reading blog posts and searching for solutions to the problems I was having. My biggest problem was with exposing the services "to the outside world". I got a cluster up quickly and could deploy example services to it, but unless I SSH port forwarded to the cluster members I couldn't access the services. I spent a lot of time trying to get various ingress configurations working but really couldn't find anything beyond an introductory level documentation to the various options.

Kubespray and one blog post I stumbled across got me most of the way there, but at that point I had well run out of time for the proof of concept and had to get back to other work.

My impression was that Kubernetes is targeted to the large enterprise where you're going to go all in with containers and can dedicate a month or two to coming up to speed. Many of the discussions I saw talked about or gave the impression of dozens of nodes and months of setup.

Other options I'll probably look at when I have time to look at it again: Deis https://deis.com/ , Dokku http://dokku.viewdocs.io/dokku/ , Flynn https://flynn.io/ , LXC https://linuxcontainers.org/lxc/introduction/ and (though I'd been trying to avoid it) Docker Swarm https://docs.docker.com/engine/swarm/

We're running Kubernetes for minor amounts of traffic in production, but we're still not in a great place due to a few limitations in running Kubernetes on your own gear.

I know that people like to talk about cost savings and stuff like that, but I'd like to see if it actually lowered your app latency and increased sales/conversions/whatnot. Things that matter a lot to a growth business.

I ask because the various overlay/iptables/NAT/complicated networking setups in Kubernetes lend themselves to adding more overhead and being much slower than running on "bare metal" and talking directly to a "native" IP. I really, really wish that Kubernetes had full, built-in IPv6 support. It would remove a lot of this crud.

Our solution works around this by assigning IP's with Romana and advertising those into a full layer 3 network with bird. The pods register services into CoreDNS, and an "old fashioned" load balancer resolves service names into A records. Requests are then round robin'd across the various IP's directly. There's no overlay network. There's no central ingress controllers to deal with. There's no NAT. It's direct from LB to pod.

The nginx ingress controller is not a long term solution. It's a stop-gap measure. Someone really needs to build a proper, lightweight, programmable, and cheap, software-based load balancer that I can anycast to across several servers. That or Facebook just needs to open-source theirs.

We use Kubernetes in a very unusual way. We built a tool that allows you to take a snapshot of a Kubernetes cluster (along with all apps deployed inside) and save it as a single .tar.gz tarball.

This tarball can be used to re-create the exact replicas of the original cluster, even on server farms not connected to the Internet, i.e. it includes container images, binaries for everything, etc.

But if the replica clusters do have internet access, they can dial back home and fetch updates from the "master". People use this to run complex SaaS applications inside their customers' data centers, on private clouds. These clusters run for months without any supervision, until the next update comes out.

Thanks to Kubernetes, you have a complete, 100% introspection into a complex cloud software, which allows for a use case like this. Basically if you're on Kubernetes you can no longer be tied to your one or two AWS regions and start selling your complex SaaS as downloadable run-it-yourself software. [1]

[1] https://gravitational.com/telekube/

My company is running it in production. We started with the monitoring stack and then our application, which is a SaaS product. The product itself is a stack deployed on a per customer basis. With rough configuration management using chef and terraform it took an ops person hours to complete a deployment a year ago. Today, it takes a minute and it's self-service for engineers.

Efficiency is increased a great deal. Previously, I'd have idle instances costing money when they were unused. Now my resource consumption is amortized across my entire infrastructure.

It also drives an architecture that lends itself to better availability. A pod in k8s may be rescheduled at any moment. If it's a stateless service, one may just increase the number of replicas to prevent a service interruption. If it's a stateful service, one is forced to think about how to persist data and gracefully resume operations.

It took a while for everyone on my team to get familiar with it, but once we started to really grok it, the sense of safety went way up. I feel pretty good about the fact that if someone accidentally blew away a cluster with thousands of pods in it, that I could easily replicate the entire thing and get it back up and running in tens of minutes without a lot of hassle.

Company was moving microservices and containers anyway, so Kubernetes is one of the few sane options to run them.

I can't say that anything got worse. Once you (collectively) get past the initial learning curve and add automation in place, it's way better than most other deployment scenarios. Kubernetes worker nodes fail from time to time, AWS kills them and we barely notice. POds get rescheduled automatically and mostly work.

Frankly, the remaining headache-inducing things are mostly related to the software which is not running on Kubernetes (mostly stateful infrastructure, and mostly for non-technical reasons). Managing VMs is a pain.

There is one thing that one needs to be careful. If you are moving from a VM for each service, to Kubernetes, now suddenly your service is sharing a machine with other services, which wasn't the case before. So I'd suggest that proper resource limits be set so that the scheduler can work properly.

One thing that can get worse is network debugging. There are way more moving parts, so it is not as easy to just fire up wireshark.

We use it in production. It's declarative nature is excellent. We tell it how many of which application we want to run, where our nodes are, and it does the legwork in deciding where to run the applications.

So far zero problems.

We have been migrating to Kubernetes for the past year. Overall it has been great and the extensibility of the platform is amazing once you combine Custom Resource Definitions (CRDs) into your cluster.

Moving a lot of our build infrastructure to it. The effort has essentially been moving alpha to beta grade.

The knowledge/documentation base just flat out sucks. It feels like documentation expects you to just know what you are doing and are really just reading it for the smaller feature switches to change how things work. That or expect you to be running on GKE.

That said, it's the best at doing what we need which is a scheduler and manager for running multi-container workloads.

I expect it to be more fulfilling/difficult when we move to more long-lasting pods but ultimately still use kubernetes.

At Reddit, we’re beginning to move our stateless services to it. We’re still in early stages and want to have a story that can at least be better than our current infrastructure. A lot of that means heavily utilizing Kubernetes abstractions, sticking close to the community and it’s tooling so we can provide more functionality than we could before with just a small team of infra folks. What I mean by this specifically is benefits of things like having an API for deployments, being able to provide different rollout strategies, offering devs access to more infrastructure safely etc. Another thing we’re hoping Kubernetes helps us solve is being able to become multi-cloud tenants. The benefits of this to us would be cost savings, and hopefully more reliability.

P.S: if you’re interested in working on Kubernetes at reddit, send me a message at saurabh.sharma [at] reddit.com

At my last meetup. (Detroit Kuberentes) We heard the story of a company that was able to rebuild their entire production environment from scratch in 1 hr. They run on kuberentes, of course it's not the only story, I'm sure using something like Terraform/Cloudformation is also part of the story.

Our app was originally a single Node/Meteor app running on an Nginx server in AWS. We used shell scripts to deploy. "meteor build" was ran on each devs machine when we wanted to deploy. We experimented with a small microservice running on a docker instance on EC2, then eventually moved to a Kops-provisioned EC2 cluster for everything.

We don't really have tremendous auto-scaling needs. But there are a few reasons why we did it.

We've since scaled to a small handful of services (frontend web app, backend node api, legacy stuff, custom deployments for high paying customers). We knew that standardizing on Docker was a good idea just to simplify the build process. What were previously service-specific, hard to maintain, hard to read, just weird shell scripts became a simple Dockerfile for each service, often only 5 or 6 lines long. Once you have that, setting up CI is ridiculously easy; AWS CodeBuild into ECS ECR took all of a day to implement. If we were on GCP it would have been even easier.

Comparatively, we'd spent longer than that actually maintaining the old scripts. The new ones require zero maintenance; they "just work", 100% of the time.

So we knew we wanted Docker. And that was a very good choice. No regrets. We start there. But once we had docker, our eyes turned to orchestration.

Its worth saying that there are a lot of nonobvious answers to questions in the "low/medium scale up devops" world. One big one we ran into early is "where do we store configuration?" Etcd or Consul are fine, but we're a pretty small shop; we didn't want to have to manage it ourselves. We could go with Compose or something. But, how do we get that configuration to the apps? We were in the process of writing new services and we wanted to follow 12-Factor and all that, so envvars make sense. But to get configuration from a remote source would break that, so we'd need some sort of hypervisor for the app to fetch that...

Additionally, how do you deploy? Let's say we go with a basic EC2 instance running Docker. We'd have to patch it together with shell scripts, ssh in, pull new images, GC old images (this was a huge problem with our "interim" step in the first paragraph, our EBS volumes filled up all the time, lots of manual work), restart, etc. Can you do that with zero downtime? Probably. More scripts...

Load balancing and HA? Yeah of course we can wire that up. More scripts. More CloudFormation templates or whatever...

Eventually you arrive at an inescapable conclusion: At a surprisingly low level of scale, you are reinventing Kube and Kube-like systems. So why not just use Kube? You get config management built in. You get a lot of deployment power in kubectl. You get auto-provisioning load balancers on AWS. You get everything.

And I could not be more serious when I say: It "just works". We've had two instances of random internal networking issues between our ingress controller and a service, which were resolved by restarting the ingress controller. That's... it, in a full year.

Like Docker before it, I think Kube is a foundational piece of technology that is only going to get more and more popular. Its incredible. But I do think that there is room in the market to make it easier to adopt. Best practices around internal configuration stuff is hard to come by; even something as simple as "I want a staging env, do i use two clusters or namespaces in one cluster" doesn't have a clear answer. Getting a local development environment cycle up and running is a pain in the ass. Monitoring and alerting is still pretty DIY; GCP solves some of this, but there's no turn-key alerting piece that I'm aware of. Logging is a nightmare if you are self-hosting, including on Kops; we ended up installing the Stackdriver agent and we use GCP for it, even though literally everything else in our stack is on AWS.

I literally don't believe there's a level of "production scale" your organization could be at where you wouldn't benefit from Kube. It is far far easier to set up than a bare deployment of anything if you do GKE. Connect Github to CI to Kube... good to go.

Lots of folks have explained it well in the normal terms used for it. Here's the comparison.

Compare it to init. Systemd is currently the dominant init for most Linux flavors. When your system starts and the kernel has done its thing, init takes over. It launches daemons, sets up networking, that sort of thing. After startup, it will relaunch things that crash, possibly do other things.

Kubernetes is sort of an init for an emerging pattern of cross-machine systems. Like systemd, it uses configuration to figure out what should run in what order, relaunches things that fail and manages networking (albeit in a very different context), that sort of thing.

There are huge differences because the problems are very different. But as a high level comparison, there are worse ones. (Especially if you're very charitable when interpreting "that sort of thing".)

You have a bunch of servers with kubernetes software installed on all of them. 1 of them is master, the rest are workers. You talk to Master through API and it talks to workers with SSH.

You send the Master some YAML files that list what containers you want running, how many, and lots of optional details. Master makes it happen so it matches what you said should be running.

Master also constantly checks everything so it matches what you asked for, even if something changes like a server failing. That's it.

Full Disclosure: I work for Red Hat in the Container and PaaS Practice in Consulting.

As others here have said, it’s a Container Orchestration Platform and is a largely pluggable architecture that also manages at varying extents Clustered Computing Resources, Application Resource Management, SDN Networking, DNS, Service Discovery, Load Balancing, and other concerns of “Cloud Native Application Development and Deployment“.

You can try it out at http://kubernetesbyexample.com/ .

The key concept to understand is that containers are just regular operating system processes with additional isolation.

Most people are familiar with memory isolation between processes. The use of further isolation mechanisms for the filesystem, I/O etc. is generally termed a container [1].

Kubernetes adds an orchestration layer on top of containers to manage processes running across different operating system instances.

[1] https://jvns.ca/blog/2016/10/10/what-even-is-a-container

The simplest way to explain it is that it is google's version of docker swarm.

Aside from that, it's a way to manage the deployment of containers across one or more hosts. Getting more traffic than usual? You have a program that detects it and sends a command to kubernetes/dockerswarm that tells it to scale up the number of web containers. Stuff like that.

Kubernetes is a container operating system. A container allows is a light weight virtual computer. Which allows you to run applications in isolation. This is useful when you might have applications with conflicting libraries. When you have multiple applications packaged in containers. You need to decide on which computer to run it, you have to deal with the networking, the load balancing if scaling it out. Upgrades and such things. Kubernetes is designed to automate all that away. You build a cluster (group of computers) You tell kubernetes you wish to deploy/run an app, kubernetes figures out where to deploy it, how to network it, link em, load balance, etc.

Kubernetes is an open source system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications

Watch this video. It's literally K8s for kids (and quite awesomely well done): https://youtu.be/4ht22ReBjno

Kubernetes orchestrates clusters of containers in a data center.

It chooses where to run the containers, wires up networking between them. And keeps them running.

I'm on my phone so I can't grab the link for you but a good intro is Kelsey Hightowers Kubernetes The Hard Way on GitHub.

Kubernetes is a thing that makes it easier for one program to talk to several other programs, by tying them together with elastic string, and putting them in whatever room it thinks they should live, and makes them as big or small as they said they needed to be. It runs in the engine bay of the truck, not in the cab of the truck, or in the container.

The best way to get started is to follow the official guide https://kubernetes.io/docs/tutorials/stateless-application/h...

You will use a tool called a “minikube” which is “mini” version of kubernetes that runs on your local machine.

You should focus on to get familiar with kubectl(1) first, it’s a simple CLI tool to manage kubernetes cluster.

If you have any questions & issues you can ask on Stack Overflow #kubernetes and you can also join the Slack channel.

To follow this up, can anyone explain the benefits over Docker? I've used Docker before but am unfamiliar with Kubernetes terminology. I do understand it's an open source project by Google.

Katacoda has excellent tutorials/labs for Kubernetes. https://www.katacoda.com/learn

Its a tool for DevOps helping orchestrate container deployments and scaling, you probably want to play around with docker before diving into to kubernetes.

Kops has not released a 1.9 version yet. Even k8s projects can't keep up.

I think LTS type of implementations are down to container services providers and things like OpenShift.

We published a post about this a couple months ago which might interest you[0]. Here's the HN discussion in case you missed it[1].

[0] https://gravitational.com/blog/kubernetes-release-cycle/

[1] https://news.ycombinator.com/item?id=16285192

Agreed. This would also reduce mid/large enterprise worries regarding adoption (or investigating the tech for possible adoption). Being on the upgrade train several times a year is too probably much for them...

Yep at the moment (AFAIK) support is essentially for 9 months from release (current release + 2 previous) so without an LTS process, everyone has to upgrade at least that often to keep getting security patches...

Disclosure: I work on IBM cloud

IBM has been running kubernetes on baremetals internally for over a year and just recently announced it as a product: https://www.ibm.com/blogs/cloud-computing/2018/03/managed-ku....

I will admit that it isn't as straightforward to get it to work as you might imagine at first. But once you've got the automation humming, its been a surprisingly easy to maintain. Would highly recommend that route! (but of course I'm biased since I've worked on it)

One interesting use case is that its straightforward to access the underlying hardware on baremetal machines with `SecurityContext: priviliged` (you can do a much more fine-grained security permissions; I'm just giving an example). So for instance, you can access GPU's, TPM (trusted platform modules) this way.

+1 for "regular" Ceph. Don't bother with that rook stuff. Just setup a regular Ceph cluster and go. Kubernetes handles its stuff and a (much more reliable and stable) Ceph cluster handles blocks and files.

NIO, the self-driving car company is doing this. They did a pretty detailed interview on their use case which includes a 120 PB data lake, and Cassandra, Kafka, TensorFlow, HDFS. You can read here: https://portworx.com/architects-corner-kubernetes-satya-koma... . (Disclosure, I work for Portworx the solution they use for container storage, but hopefully the content speaks for itself).

Redhat probably has the best production quality deployment for self hosted Kubernetes. They support running glusterfs: https://github.com/openshift/openshift-ansible/blob/master/i...

Personally I wouldn't do it unless you have a RedHat contract (or already have a team that manages glusterfs), but its worth looking at.

1200 core k8s cluster + 1.5PB ceph (shared nodes to some degree) no issues with persistent disks etc, only "annoying" thing is to figure out RBAC initially

you just use a storagecontroller, then its no work whatsoever, ceph does the rest

I've been recently experimenting with getting lustrefs usable with kubernetes, and needed some way to natively integrate it into the cluster I had.

https://github.com/torchbox/k8s-hostpath-provisioner Proved to be useful, it allows you to use any mounted path on (all) a node(s) (hostpath method) to return satisfy a persistent volume claim and return a persistent volume backed by the mounted file system.

A similar set up could work using bare metal, are you using something like Openstack Ironic?

OpenEBS looks promising, but have not given it a try yet, ref: https://www.openebs.io/

Commercial: Portworx, StorageOS, Quobyte

OpenSource: OpenEBS, Rook (based on Ceph), Rancher's Longhorn

Commercial options highly recommended if you want safety and support as storage is hard, although people seem to be running all of these options well enough. Portworx probably most highly developed with Quobyte a good option if you want shared storage with non-kubernetes servers.

We experimented with both using cephfs via rook and glusterfs via heketi and ran into enough operational issues and speed bumps that we're just using hostPath volumes for now.

IOW, they're not production ready yet.

http://openshift.io is also built on Kubernetes (by virtue of using openshift.com for its deployment pieces)

Seems someone else was able to get it that working: https://blog.jessfraz.com/post/building-container-images-sec...

10 minutes? That sounds a lot. We usually have 1-2 minutes wait time for a changed ConfigMap to reach a running pod, or for a Deployment to roll over to a new ReplicaSet. (That's on k8s 1.4.)

I don't have much content on CoreDNS. But from the description in the README, https://github.com/coredns/coredns/blob/master/README.md, I guess it acts as a flexible and programmable entry service for underlying services and can do many applications such as Load Balance, depending on plugins.

We've been using CoreDNS for a bit now, and it's much better than KubeDNS. We found that KubeDNS would time out and drop requests from time to time. No such issues with CoreDNS. Would recommend (at least from a reliability standpoint).

> Isn't this going against what the Elixir/Erlang system already provides?

No?

So, I guess, let's dig into that a little bit. Erlang's always kind of left the health and safety of your deployed system as a whole up to you, being preoccupied with giving you tools for understanding and maintaining its internal operation. OTP provides two semi-unique things: supervision of computation with control of failure bubble-up and hot code reloading. Hot code reloading isn't used all that much in practice, outside of domains where it's _very important_ that the whole system can never be offline and load balancing techniques are not applicable. That's a specific niche and, sure, probably one that kubernetes can't service. With regard to supervision, there's no incompatibility between OTP's approach and a deployment consisting of of ephemeral nodes that live and then die by some external mechanism. Seems to me that kubernetes is no different a deployment target in this regard than is terraform/packer, hot-swapped servers in a rack or any of the other deploy methods I've seen in my career.

I'm reading through Joe Armstrong's thesis on Erlang, and one of his main points is that concurrency programming shouldn't need to care about locality, at least in terms of correctness. Processes can run locally, or on separate machines, because each process is isolated as if it's its own VM. Containerizing it just makes for simpler deployment units, in my view.

That said, there might be some pragmatic tradeoffs eventually made for the Erlang VM that I'm not familiar with yet that makes the container move more dubitable.

Kubernetes does resource management; all cluster scheduler systems do. The difference between Kubernetes and Mesos for resource management is that resource requests and limits are optional in Kubernetes and mandatory in Mesos. It is best practice to specify resources in Kubernetes.

I've run vanilla k8s for about 3 years now in prod but am also fairly familiar (and really like) Openshift Origin. I usually tell people asking this question the following;

OO comes with a bunch of really nice quality of life improvements that are missing (but in a lot of cases can be added via 3rd party TPR/CRDs/etc) in k8s but you aren't deviating so far from k8s that you won't be able to go work on vanilla k8s in the future. Not at all. Most of the additional stuff are annotations that simply wouldn't do anything and you'd remove them if you moved from OO to k8s.

I think that if you're brand new to the environment OO can really help you get running quickly. You just have to make sure that you do in fact dive in to the actual k8s yaml and deal with ingresses, prometheus, grafana, RBAC, etc at some point. I haven't used OO in awhile but I believe you could successfully do most of what I do day to day via yaml/json through the OO UI.

On the flip side a lot of people will probably tell you to start from k8s, whether that's GKE or AWS or minikube or wherever and go through the k8s the hard way. Personally, and I help people quite frequently on the k8s slack, I feel like that leads a lot of people down a path of frustration. It may be perfect for your style of learning or it may just scare you off.

Now when it comes to OO you are at the behest of their releases. Their most recent release was Nov 2017 and k8s 1.10 was just cut. I'm not sure what version OO is on now, evidently they changed their versioning numbers to not correspond to the k8s version.

Join #kubernetes-users and #kubernetes-novice on slack.k8s.io if you need any help. It's a vibrant community. You can message me directly @ mikej if you'd like.

edit: Ok OO 3.7 is k8s 1.9, that's perfect. I wait a few months before jumping into new major.

I'm not a production user, but I've had a look at both. From what I've seen OpenShift provides a more polished and complete package than base Kubernetes, so if you're looking to get productive quick, it's probably a good option.

Kubernetes base is more flexible of course, but with that flexibility comes more stuff you have to do yourself...

Either way they're both based on the same tech., so experience with one largely translates across to the other

Kubernetes is great for those that are ops-inclined, OpenShift is a much better experience for your typical developer though.

Personal anecdote: We've had OpenShift Origin running in production for a year and a half, I'm the DevOps guy and know how to poke around in the internals but the developers on my team are just that, developers. They just want to say "here's my code, go make it work", S2I, templates and Jenkins pipelines let them do just that - they go paste the URL of a git repository into the web interface and watch the project build and automatically deploy. It's a pretty magic experience watching a Junior developer deploy 13 applications over the course of a year with no supervision or hand-holding from a more-senior developer.

I deployed plain k8 to start with 2 month back and started adding more add-ons and plugins as we go. This made us be flexible abt what we use and control. So far no issues. Also our customers are inclined for k8 workload files not OpenShift.

Depends if all service discovery sources and targets are all within k8s. If so, k8s works well, if not, not so much.

I'm not so sure about that last part. We (Kinvey, www.kinvey.com) have been using it for customer-facing services in production for over a year. Results have exceeded our expectations.

We are using Kubernetes since 2016 to deploy OpenStack for SAP's internal cloud. We also offer the reverse, Kubernetes on OpenStack, as-a-service to our internal customers. [1]

We are all-in-all very satisfied with Kubernetes, except that it's quite messy to do right on bare metal.

[1] Also available as open-source: https://github.com/sapcc/kubernikus

Please check out the large and growing list of case studies of companies of all sizes adopting Kubernetes: https://kubernetes.io/case-studies/

Disclosure: I'm executive director of CNCF, which funds the case study write-ups.