data chimp automatically shows data visualizations, tables, and messages about your data as you work in your Jupyter notebook according to rules you set. The rules are specified in code via special “config” notebooks, and when the rules are triggered, your rule code has access to the data frame that’s currently being analyzed, so you can visualize or aggregate it however you like.
For example, you can write a code rule that says, “if I’m working with a data frame column that has more than 3% missing values, show a time series plot of the percentage of missing values for the column over time.” With rules like this, you can spot unexpected features in your data, catch buggy data wrangling code, get oriented in a new data set quickly, or encourage analysis best practices within your team.
I built data chimp because when I first joined the data science team at Heap (I came from an engineering team), I was shocked at how error-prone and repetitive exploratory data analysis was. On one occasion, we published an analysis that contained a bug that slipped past code review, and when our stakeholders got excited about the analysis, we had to tell them to hold off while we re-crunched it. Moreover, I was regularly frustrated by being forced to either
* write the same visualization and aggregation code over and over again
OR
* use a canned function (e.g., pandas profiling’s ProfileReport) that wasn’t flexible enough to show me what I needed to see in a particular scenario and that stifled my ability to iterate on the visualizations generated by the function
data chimp makes analysis less error-prone by borrowing some ideas from automated testing and linting, and it resolves the repetitive code dilemma by making it easy to customize what is automatically shown according to your own rules and by making it possible to paste the code that generated a particular result back into a notebook cell so you can easily iterate on it.
data chimp is currently “beta quality,” but I’m hoping to get some feedback by posting here, and more generally, I’m looking to understand the problems that data scientists tend to face as they’re working with data. If you want to tell me why data chimp sucks or complain about your job to me for a few minutes, I’d love to chat sometime. ;) Book a time here: https://meetings.hubspot.com/matt-dupree