That's the hard part of the problem, extracting signals from the noise. Have a look at the Google Autopilot paper (cloud scheduling, not self driving). There are a few interesting ideas that could be adapted. They generally have some good ideas if you peruse their research page, specifically the data center and related topics.