Openshift sells itself as a more secure Kubernetes ... anyone knows how much of this it mitigates?
The biggest thing is that they don't allow containers to run as root, but they also have other features like blocking hostPath mounts and offering SELinux integration. Nearly all of this can be turned off (if you have permissions to do that in the RBAC) but they generally take the "secure by default" approach.
OpenShift also moves to CRI-O and rootless containers in later version, which further protects hosts.
OpenShift helps mitigate the following, by providing secure defaults. It is totally possible for someone (a cluster level admin) to un-configure these security controls, so keep that in mind.
These are partially mitigated by Red Hat providing trusted image content and having a well oiled machine for patching/CVE response:
* Compromised images in registry (use Red Hat base images, and in-cluster source or docker builds to lock down registry access. Registry not exposed by default.) * Application vulnerability/Application exploit (again, Red Hat base images for runtime, regularly patched by Red Hat. Images have to run as non-root user. Doesn't solve self-created app vulnerabilities)
These are mitigated by OCP's locked down default RBAC:
* Writable host-path (host-path locked down by default. Not sure why host-path is special here though, seems like any PV could be used for persistence) * Privileged container (also locked down, admin must enable for a user/service account) * host-path mount (specifically for getting access to the file system, OCP locks down this via RBAC by default. Layer on SELinux to prevent an escalation from allowing file-system access) * list kubernetes secrets (RBAC to limit scope, and additionally a node can only access secrets for pods scheduled to it, in case node is exploited) * access container service-account (default service-account has pretty limited permissions) * access Kubernetes dashboard (containers have no special access to the console, which requires auth)
These are just miscellaneous config/security controls:
* Exposed dashboard (OCP console has authentication by default, no default passwords) * clear container-logs (built in log aggregation would require an additional exploit to defeat) * cluster internal networking (depends on the sdn plugin, but most installs limit networking to containers in the same namespace by default) * access tiller endpoint (No Helm 2, support for Helm 3 without tiller coming soon. Driven by tiller security/RBAC concerns)
Prometheus does talk to the control plane, but via a service account that only has read and list permissions, so shouldn't be able to compromise it. The truly paranoid could have static configurations (scrape foo.bar.svc.cluster.local:1234, not "foreach pod, scape $POD_ID:1234"), but if reading/listing pods can compromise the control plane, that would probably be a P0 security bug in Kubernetes.
Obviously, anything running in your cluster has the potential to exploit a kernel bug to read files on the node that it shouldn't have access to, including other service accounts. The guidance there should be to be careful about what permissions you grant any application.
Once we turn off service account tokens (they will he generated for pods on demand on each host) then that attack gets much smaller, although at that point is an extension that reads secrets is still as privileged as any pod on the cluster.
Being able to create a pod in a namespace makes you also able to read any secret in that namespace.
I wrote my own thing as an alternative to Ingress. I have Envoy and the public TLS certificates in a namespace, then send it a list of services over xDS from another service (in its own namespace, with read/list access to services and endpoints): https://github.com/jrockway/ekglue
This is limited in the sense that you have to edit the route table and reload Envoy to add new routes, but at least you don't have to configure the backends. (And Envoy doesn't have to poll DNS every 5 seconds for every upstream.) I don't manage the route table because what I've learned from Ingress is that route tables don't compose nearly as well as people think. They are ordered, and that creates all kind of craziness. (ingress-nginx, for example, orders by the creation timestamp on resources. So if you delete and reapply your Ingress rules, the controller behaves differently! That seems like a disaster waiting to happen, so I just manage it as one chunk. I think that is what we did at Google; it was always Quite The Procedure to create a new publicly-routable DNS name.)