I’m in the very unusual situation of being tasked to set up a self-sufficient, local development team for a significant national enterprise in a developing country. We don’t have AWS, Google or any other cloud service here, so getting something running locally, that they can deploy code to, is part of my job. I also want to ensure that my local team is learning about modern engineering environments. And there is a large mix of unrelated applications to build, so a monolith of some sort is out of the question; there will be a mix of applications and languages and different reliability requirements.

In a nutshell, I’m looking for a general way to provide compute and storage to future, modern, applications and engineers, while at the same time training them to manage this themselves. It’s a medium-long term thing. The scale is already there - one of our goals is to replace an application with millions of existing users.

Importantly, the company wants us to be self sufficient. So a RedHat contract to manage an OpenShift cluster won’t fly (although maybe openshift itself will?)

For the specific goals that we have, the broad features of Kubernetes fit the bill - in terms of our ability to launch a set of containers or features into a cluster, run CICD, run tests, provide storage, host long- and short lived applications, etc. But I’m worried about the complexity and durability of such a complex system in our environment - in the medium term, they need to be able to do this without me, that’s the whole point. This article hasn’t helped me feel better about k8s!

I personally avoided using k8s until the managed flavours came about, and I’m really concerned about the complexity of deploying this, but I think some kind of cluster management system is critical; I don’t want us to go back to manually installing software on individual machines (using either packaging or just plain docker). I want there to be a bunch of resources that we can consume or grow as we become more proficient.

I’ve previously used Nomad in production, which was much simpler than K8s, and I was wondering if this or something else might be a better choice? How hard is k8s to set up today? What is the risk of the kind of failures these guys hit, today?

Are there any other environments where I can manage a set of applications on a cluster of say 10 compute VMs? Any other suggestions?

Without knowing a lot about their systems, I suspect something like Oxide might be the best bet for us - but I doubt we have the budget for a machine like that. But any other thoughts or ideas would be welcome.