We have been testing out production clusters with Swarm 1.13 and I think its neat. But what do I know - I'm not a cloud hosting specialist. And the lack of success stories out there is making us nervous.
Depending on the size of your team, I would recommend Kontena (www.kontena.io) or Kubernetes (https://kubernetes.io/). While I haven't used the latter, a group of two devs is re-architecting our entire system to use Docker and Kontena. Nearing the end, the whole process seemed incredibly fast to iterate. While K8s seems far more proven in real production environments, it seemed much more daunting to us with more features than we needed for our use case.
secrets stored in plain text and (by default) transmitted in plain text between etcd services, does not fill me with confidence on their security.
If setting up a secure cluster is daunting, then use a distribution that handles it for you. OpenShift (https://www.openshift.org/) is built on kubernetes, and it's install is secure by default.
Disclaimer: I work for Red Hat, and spend lots of time on OpenShift consulting.
It's definitely something we'll fix in Kubernetes, but rooting workloads is the primary problem, and secondary acl defense in depth is good but won't block most attacks for long.
Default ACLs are clearly the most important line of defense in an orchestrator's security model, because whether a container escape can happen is not something the orchestration system has control over.
Being able to trigger node compromise should have nothing to do with being able to schedule.
there are tons of these niggling issues that are cropping up.
the complexity of using k8s goes up exponentially every day. I have bo doubt it is a great piece of tech.. but at this point, it seems tailor made for consulting.
On OpenShift I'd be very interested to see a list of the secure defaults that have been chosen, is there a list of the changes from base Docker/Kubernetes policies available anywhere?