When we host competitions on Kaggle, a lot of work goes into asking the right question and structuring the problem. The domain expertise is incorporated in this step, as well as in putting the competition results to use in production.

This splits the "domain expertise" and "predictive modeling" components into two separate chunks. While domain expertise is crucial for asking the right questions, we've found that it isn't as necessary for the predictive modeling component. For example, in the essay scoring contest we hosted, none of the winners had touched natural language processing prior to the contest. However, they beat out many experts with decades of experience in NLP.

For an internal data science team, the "domain expertise" component is at least as important, as they are charged with asking the right questions as well. However, this does not mean competition winners cannot develop and learn this - they have already demonstrated their creativity and tenacity in one domain (applied machine learning), and this carries over nicely to other domains from our experience.