Thinking about it this way is interesting. It could be an issue with bias or unbalanced training data.