Model decision interpretation is a solved problem: https://github.com/marcotcr/lime