Engineering Practices in Data Science

I agree that setting up a dedicated pipeline early on in a project and committing yourself to work within its confines aids organization, but it also contributes to putting your creativity in the right place. We often enjoy building things for the sake of building and those of us with stronger proclivities towards engineering can sometimes get too jolly putting together new pipes when we should be training models. I have been guilty in the past of writing a Perl script that spawned shell scripts that cued Perl scripts on a cluster that ran R scripts that piped back into Perl scripts. Then again I always liked the boardgame Mouse Trap.

A (flawed) analogy: if data science is storytelling, stats is the story and engineering is the words you use to tell it. You need to do well at both to effectively tell your story.

>...good engineering going out the window quickly with elaborate ensembling.

This is one of my criticisms of data mining contests (sorry kaggle!). When I was in grad school I liked doing these contests to get practical experience - my last company actually recruited me through one. But as they got more popular I found the best engineers and data scientists stopped having as much of a chance of winning. Good modelers get 90% of the way there and then are beaten by impractical solutions that would get someone fired from a job.

Edit: And we're back up.