> The unsupported product is as good as our (also unsupported! but luckily not End-of-Life) VSphere and VSan environment.
> It is a different product to me, in short because I am the one administering it. It runs on a different platform altogether
Distilling my previous comment, it's really this.
I navigated the waters of Fleet and Deis v1 to find an answer to "how can I make sure this does not go down with lots of lead time and half a dozen warnings well in advance before it does go down." (Aside, my datacenter at the time had famously reliable electricity on two grids that just does not go down.)
Now I have to renegotiate that position to get the same guarantees with Kubernetes. Before, I was worried about maintaining etcd quorum when N machines go down, and preventing split-braining. Now, I'm still worried about those things, but they're behind an abstraction layer of Kubernetes API and new suite of tools for managing it.
I'm not expected to manage my etcd quorum in the same way. I am sure that's good news. Or I am still expected to manage etcd quorum, but it's buried behind a mound of Kubernetes so it's hardly even clear that there is ETCd running at all on a basic cluster without multi-master. You couldn't have a Deis v1 cluster without running at least 3 instances of etcd. You were forced right away to get to know those failure modes.
If it's not clear yet, I am a really small-time consumer of high availability.
The new system also forces many best practices on me, in ways I'm not accustomed to. (More not bad things...) I was once advised to split my control plane from my data plane (and possibly also my routing plane) back in Deis v1, to ensure reliability. I never got around to it with Deis v1... but in Kubernetes it's already been done for me by kubeadm.
I had a 5-node fleet cluster with 5 etcd members that was "all control plane all the time" inside of what amounts to a single AZ, and it's pretty clear that this is a totally wasteful design now. You wouldn't build a K8S cluster with 5 masters and no minions. But what's 40GB of RAM in a private data center? To ensure reliable service, sure we had that kind of RAM just laying around. With Deis v1 we did exactly that. For a cluster the size of mine, I'd say it was a thoroughly researched and well-advised decision... at the time.
It's clearer to me from talking with you that, from a code perspective, there is just way too much code in common to call it a brand new product. For a cluster admin who doesn't get very deep into the code though, I feel like it's a much easier case to make that it is in fact a distinct and new product. The constraints have all shifted, and the guarantees are in quite a lot of cases not the same.
This all amounts to basically hand waving though, and I'll reiterate I am not passing judgement on the progression of Deis/Workflow and its place in the container ecosystem.
It's been a wild ride! Thanks for bringing along a community with you. The marching forward and continued progress of free software such as Deis and Kubernetes is a thing to behold.