Comment by jaytaylor | Hacker News Reader

Comment by jaytaylor | Hacker News Reader

Parent Full thread

    Since we have a highly available Kubernetes
    cluster with data replicated across all
    nodes ..

This sounds like one of the more cutting-edge bare-metal kubernetes setups I've heard of!

Curious what the kubernetes implementation / configuration looks like for the setup they've described. Particularly in terms of HA-cluster k8s configuration, networking solutions integrated, and replicated storage.

Considering there is no AWS / GCE / Azure block-storage primitive to build on, it'd be cool to understand the choices and tradeoffs made in this scenario.

Is all data being stored on something like HDFS or CockroachDB? Or perhaps the k8s minion slave nodes launch containers with Ceph volumes.

Well, it’s a 6 man team so most of our trade offs were for expedience, not the worlds greatest architecture. We cheated and sync’d data using a HA MongoDB setup amongst the cluster. RKE for K8s clustering was a life saver (nothing easier at this point on bare metal IMO), although RKE can be brittle at times.

Hi Jay -- this is Brian (I wrote the article). There are constant tradeoffs. I am leery to say we are the most sophisticated bare metal setup. I think all of us that are close to the work would love for many parts to be much more sophisticated. That said, the way we handle distributed data right now is using Mongodb. We have a cluster that spans the three nodes that we have in a restaurant. The primary use case is persistence for our MQTT topics. Most of this data is what I will call "mildly ephemeral". It isn't needed for the long term, but we do `hope` to maintain it long enough to exfiltrate it to the cloud and deliver it to any necessary recipients. If we were to have a global cluster outage, it is not the end of the world. Over time, it probably will be. Back to work! :)

Reply on news.ycombinator.com