This is extremely interesting. This paper does a few different things that are worth separating:
1. Structures knowledge about common statistical tests, the assumptions they need to satisfy, and the kinds of problems for which they are appropriate.
2. Models this as a constraint satisfaction problem.
3. Creates a DSL to write specifications for statistical testing.
4. Develops an output format for statistical testing based on (3).
5. Provides a python package that implements the above.
IMO (1) is by far the most fundamental of these, but the paper spends most of its time describing (3). I'll admit to not fully understanding (2), what the alternatives were, and why it's an appropriate choice - I wish there were more on that in the paper as well.
If the approach to (1) were described in more depth and published in a usable way outside of the implementation in (5), I could see it being broadly useful and leading to an ecosystem of different (competing) implementations. As it stands, the work on (1) is spread across a few functions in solver.py.
Looking more broadly, statistics-as-an-HCI-problem is fascinating, and I think this is a promising start. I'd also love to see more attention paid to (4), the output format. Most "doing harm with statistics", I believe, comes from misunderstanding/misapplying results, so looking at the full workflow / user story is critical. The focus of this paper is instead making the input easier, which is also obviously important.
emjun, so glad to see you in the comments here! What's the best way for interested folks to contribute to this project?