> BTW, are there out there any on premises multi-user HPC clusters with Docker running on the nodes?

The vendor ClusterVision (EMEA, went bankrupt then returned to life) used to offer a cluster deployment option based on OpenStack+Docker; OpenStack for defining the cluster boundary on top of your hardware then Docker serving as the "image" on each compute node layered on top of the regular OS. Not at all a typical use-case for HPC containers, and with hindsight it is fairly obvious how that Byzantine stack resulted in too much complexity for them to support cost-effectively.