I've recently managed to make Prometheus+Grafana stateless using https://github.com/cortexproject/cortex if anyone is having issues with scaling / backing up.