I've thought about solving this problem with an ML approach like you all are taking but as you say never had the bandwidth because I was focusing on my "core missions". I'm no longer a heavy spark user but am very happy to see you all working on this!
It always seemed so inefficient to me to spend all this time hand tuning jobs only to have the data change and need to do the same thing again.
Good luck!