So, we maintain labels `prometheus_replica` which prints the pod name of the Prometheus. Since the operator deploys it as a Statefulset, the pods are ordered, so the naming will be like prometheus-k8s-0,-1 and so on. Having these labels help to deduplicate the metrics on the ingester/federation level.

Although yes you're right, it was a bit difficult to get it running out of the box with native Federation capabilities. A secondary long term storage like Victoria/Cortex/Thanos is something you can check out.

Cortex was just released, and while it doesn’t solve HA right away, it does make it easier: https://cortexmetrics.io