You might be interested in this approach to explaining the predictions for any classifier. https://github.com/marcotcr/lime