Doctests in R
hughjonesd.github.io
hughjonesd.github.io
I wonder if a challenge for doctests in R is they often have to test larger, more realistic outputs?
For example, in dplyr's mutate doc, one example is this:
starwars %>%
select(name, mass) %>%
mutate(
mass2 = mass * 2,
mass2_squared = mass2 * mass2
)
On the other hand in siuba (a port of dplyr to python), I often have to truncate the example output, because it's hard coded in the docstring:
(cars
>> mutate(
cyl2 = _.cyl * 2,
cyl4 = _.cyl2 * 2
)
>> head(2)
)
cyl mpg hp cyl2 cyl4
0 6 21.0 110 12 24
1 6 21.0 110 12 24
https://dplyr.tidyverse.org/reference/mutate.html#ref-exampl...
This idea (doctests) is one of those crucial "You can, you should, but you probably don't" things where the point is to make doing The Right Thing™ so easy that you actually do it rather than just nodding when people say you should do it. You clearly should check that your documented APIs match reality, that examples work, but in many languages that's not easy enough to do out of the box, so either a project invests time and effort to do this or they go without and most projects will go without.
Over a year ago I wrote some MS Teams integration code in C#. Microsoft publishes documentation with examples for how to use the APIs for this. The documentation is wrong. There are open bugs on long abandoned (as is usual for Microsoft) repositories, nobody cares. But chances are back when they shipped the documentation if that had failed, somebody would have fixed it, maybe it'd take a few minutes or even an hour, but it'd save a nasty experience for lets say conservatively, thousands of developers. Instead it's just a much upvoted Stack Overflow answer with a workaround.
If you're using pytest, then it's just a flag away (well two if you're using both docstring-doctests and document-doctests, but the latter seems unlikely).
But that it's opt-in makes sense either way, as doctest support was added later on, whereas in Rust it's been there all along.
- Example sections will be cluttered with unit tests.
- Doc tests asserting warnings or errors will produce examples of bad code. This might make sense for the example `safe_mean` function, where its only purpose is wigging out for improper input. But most functions should just show how to use them.
- Test scripts are still useful for setting up loops, creating helper functions, or other stuff. But then test code will be split between the roxygen comments and those test scripts.
I use doc tests in python scripts, because they're quick sanity checks that fit in the same file. I don't use them in packages. If R had doc tests, I'd rather use them in single-file scripts. Maybe a function that acts like `source` but also generates and inserts the tests.
The fundamental purpose of doctests is not to write unit tests, but to ensure your examples are valid. It's easy to write examples which don't work in docstrings.
Running a doctest system on your documentation doesn't preclude having actual tests, quite the opposite. Edge cases or complicated scenarios often don't make for great examples, but are usually valuable tests.
For instance in Rust most methods of Vec have an example, which is doctested, and yet Vec still has an extensive suite of unit tests: https://github.com/rust-lang/rust/blob/master/library/alloc/...
Technically you could use doctests as a literate-ish test framework (assuming that's even supported, which it may not be), but the oddball environment tends to make that not great, and the "literate" part is not very useful when unit testing. It's way more valuable to ensure docstrings and standalone documentation are valid.
I use roxygen example help file generation as well for R packages, but have mixed feelings relative to python documentation.
Unit tests are primarily intended for the developers of the library. They do help users, sometimes, when you're trying to work out some fundamental misconception about what the library DOES, but generally speaking the granularity of unit tests is too fine unless you're REALLY digging in.
Much better, I think, to spend effort on writing clear documentation. R has a problem with that. Docs typically have overwhelming terse detail followed by anemic examples. Couple this with that fact that R users tend to always be in the middle of something urgent and completely unrelated to writing libraries (like, analyzing data and making decisions based on that data) and you get a recipe for frustration.
I find myself referring over and over again to the tidyverse "cheatsheets" [https://posit.co/resources/cheatsheets/]. These show, explicitly and clearly, what the things actually do. I wish someone put that kind of, often graphical, content into the docs for all functions.
And granted, this is a complaint for base R and other older spaces in the R world...the tidyverse packages (and modern ones inspired by it) tend to have pretty great examples with lots of iron
"Tests and documentation are similar, but not identical. Tests need to cover difficult corner cases. Examples need to convey the basics to the user. I like the following advice:
… write the best possible documentation, and [R] makes sure the code samples in your documentation actually compile and run [and do what they are supposed to do]
In particular, use doctest as an addition to manually created tests, not a substitute for them. Use doctest to make sure your examples do what they expect, and for simple tests of basic functionality."
Disclaimer: I'm an R programmer but not deeply familiar with authoring packages.
>Couldn't people just add expect_* tests to their examples?
Users can run examples with the `example` function. So if you use the `testthat` package in examples, then you should add it to your package's imports. Which means more to load with the package, but only for a small benefit that's rarely used.
Also, raising warnings or errors in examples and not catching them is a no-no. The CRAN package repository will not accept a package like that.
Edit: I originally wrote that this wouldn't create any examples in the final manual pages, but I was wrong.
The magic makes sense now too. You already need a roxygen2 header to set up the auto-generated tests, so why not call it @expect and then write equal instead of expect_equal, so as not to repeat yourself?
But surely the user wants to see what the result of the call is, if it's relevant? That's why rust examples (which are also doctests) include the corresponding assertions.
You can hide them, but usually you don't, because e.g. showing what the result of `str::len` is is the point of having an example: https://doc.rust-lang.org/std/primitive.str.html#method.len
Unless roxygen or Rd independently runs the code and embeds the output independent of the doctests succeeding or failing?
Wouldn't R still be the primary language in RStudio, with Python being made available as necessary? Or is the idea that RStudio will turn into a proper Python IDE? Curious what makes you say that RStudio is putting its 'focus' on Python.