Something that is often left out when talking about interpretability is the relationships between the predictors and the dependent variable in simulations. For example:

- I fit a complex, difficult to interpret model to a dataset, attempting for forecast my sales (structure of the dataset largely irrelevant for this example)

- I take an entry from the training set and decrease the value of some price attribute by 15%, leaving everything else unchanged

- I try to predict the sales for the entry I just created using the trained model

- What happens if the model now predicts lower sales? There is a clear relationship between price and sales volume going in the opposite direction. Would lowering my prices by 15% really lead to a decrease in my sales? How do you track what's happening in the model to create this forecast? Did I use the wrong model? Was my training data incorrect? How do you explain this to a client or to a product user?