Hacker News Reader

Comment by rgejman | Hacker News Reader

Data cleaning/munging is a surprisingly pervasive problem in my field ('big data' cancer research). I spend a good fraction of my time hunting down bugs caused by dirty sources upstream of my scripts. One way to fix this is to become more rigorous about using tools and libraries that check and enforce data correctness and consistency... but it's hard to do this on an individual level. Even if I'm careful about ensuring the correctness of my data, other researchers who send me data may not be as scrupulous. As with any challenge that requires teamwork and constant vigilance (e.g. unit testing), perhaps the data science community needs to invest in tools and computational frameworks that constantly monitor data for correctness and consistency using DDLs and the like.

View on HN

My team at Airbnb is working on a tool called "Salus" - named after the Roman goddess of hygiene - that allows you to check the correctness of your data in a number of common, extensible ways. Perhaps this is something that the open-source community at large might be interested in?

Interesting. I'd love to hear the details.

I'd be interested. Any tool that checks correctness is one less thing I have to write before I begin my analyses.

yes.

Yes. This is something that could be useful to data journalists, who often have to quickly grip and rip new datasets.

I have rolled an extensive (purpose-built) general data quality system at work, and would love for there to be a general purpose tool that does the same sort of thing. I would love to use, contribute to, or otherwise make real an open source version of such a tool.

There is an enormous need in several industries for this sort of thing; but most people don't really know they need it yet.

Absolutely - please keep me updated on progress!

Definitely something I'd be interested in taking a look at, I imagine others would as well!

Yes.

Sounds intriguing... Is it something rule-based where I have to define things ahead of time, or can it infer stuff - for example highlighting outliers in an automatically detected numeric field?

Absolutely. Would definitely be interested in hearing more about it.

If cancer research is anything like high frequency trading and time series analysis (which I used to work on), then one heinous thing about corrupted data is inter-dependencies among different data processing tasks: if data coming in "early" in the data workflow topology is garbage, its errors are propagated throughout the dataflow, resulting in multiple days of data detective work, laborious undoing of everything that went wrong, etc.

Also, I observe that the data science community does not talk about these issues nearly as much as they should either because 1) it doesn't make a very inspirational topic 2) they are shielded from this kind of issue by a data engineering team.

Agreed. We need to talk about these issues as a community and come up with effective strategies to prevent data contamination. We've done this for web development and the result was a panoply of tools to validate form input. Now we need to do this for our back-end tools.

For instance, statistical programming languages like R, which generally operate on nice tabular data, should come with built in methods to enforce data validity. I should be able to tell R that I expect certain columns to only contain values from 0-1 and no NAs. This is an easy example because languages like R are somewhat dictatorial about how they want you input and process data, but one can imagine the same sort of methods built into the base libraries of general purpose languages.

On the other hand, maybe we need to be more rigorous on the whole about data validation. We accept that automated/continuous testing is an effective mechanism for preventing bugs. We need the same automated systems to check data files and flag them when there are issues.

Take a look at the testdat package which provides some tools in the direction.

Reply on news.ycombinator.com