We tried this at a previous company I was at, it was incredibly tough to maintain and tune. We ended up using databricks which made it significantly more productive.