Don't Enforce R as a Standard
timotheepoisot.fr
timotheepoisot.fr
Most scientists have little or no quality training in software development, but scientific research is increasingly reliant on software. At present, software is a glaring black box in a great deal of research, because very few reviewers have the skills to thoroughly scrutinise it.
A software monoculture always has negative consequences, but fragmentation can be equally problematic in many cases. A reviewer saying "This would be better in R" usually means "I have no chance of understanding your code, because it's not in R". For better or worse, R is currently the statistical computing lingua franca in most fields.
I believe that the scientific method is in real trouble, due largely to the immense complexity of much modern-day research. Scientists have access to immensely powerful analytical and statistical tools, but most lack the training and infrastructural support to use them in a rigorous manner. Bad practices in software and statistics are the norm, rather than the exception; I'm sure most of this is just an honest shortcoming, but I'm equally sure that the lack of CS and stats experience amongst reviewers is a gift to would-be Bogdanovs and Obokatas.
Science has always been a collaborative effort, but I think most fields are in desperate need of greater support from computer scientists and statisticians. Ultimately I would like to see those professions become deeply integrated into all scientific fields, with the expectation that all papers should credit a statistician and (where applicable) a computer scientist. Likewise, editors and reviewers need far closer ties to CS and statistics professionals. Of course, these issues are intertwined with many other problems in funding and peer review.
Until then, the hegemony of R may simply be a price we have to pay for better research in the short-term. A software monoculture at least gives reviewers a fighting chance of spotting issues with software that might affect the validity of results.
The point, then is to supply code with simple instructions so that anyone can run it. Secondary goal: write it in a language a human can actually read.
If you're doing science and you want people to understand your code, you use Python, not R.
(It's a pet peeve of mine. The language is called "C++". "cpp" is commonly used as the suffix for C++ source file names, but the name "cpp" also refers to the C preprocessor.)
Wouldn't that depend on which language "people" know?
> Its the PHP of Data Science.
Some might disagree with that statement. Some might even argue that languages where for loops are idiomatic and strongly promoted by the BDFL are the PHP of Data Science.
PHP is the canonical reference for code which can be understood but not interpreted, because you don't know the values of various obscure global flags.
R isn't horrible, and if you are in the repl, you can print any function's code you can access, which is awesome.
Further, the vast majority of R modules are at least partially implemented in R and tend to be very readable.
python: X.dot(Y)
matlab: X*Y
R: X %*% Y
pandas has a lot of functionality, but the interface is far inferior to R's data frame, where you have
df[row predicate, column predicate]
The api to sklearn, while definitely more consistent than R's apis, has a lot more programmer nonsense interjected: imports of random packages, the difference between pandas and numpy that still peeks through the second you step outside of statsmodels, etc.
numpy has a serialization format, while pandas uses pickle or hd5. R just uses save/load, and unlike pickle in my experience, it reliably works.
matplotlib is an ugly api, particularly compared to ggplot.
- python 3 (.5?) is adding the @ operator for matrix multiply. Of course this only helps you if your company uses python3
- There are 2 packages (seaborn, ggplot(clone)) which sit on top of matplotlib and provide a much nicer interface for statistical plots
FWIW, I've done both Python and R and they both have their pluses and minuses, but beauty is in the eye of the beholder and I've seen some R packages that do, indeed, make R beautiful, case in point, the dplyr package from Hadley Wickham. I'd love to see the future of R be inspired by dplyr and ggplot.
[1] -- http://technology.stitchfix.com/blog/2015/03/17/grammar-of-d...
The language matters. One is optimized for ... honestly I have no idea what R is optimized for. I don't think R works that way. It just is. Python though, its optimized for readability.
R is optimized for statisticians
R and Python really are very similar as languages. R is more functional, but that shouldn't impede your ability to understand code (once you master some of the basic idioms of FP)
If I understand it correctly, this observation shows just what a short distance we've come in programming languages and software development since the 1960s. Verifying the correctness of a program shouldn't depend on good naming or being able to execute it in your head.
Our battle should be more with SAS.
This is very true, but it will be difficult to realise within the university centred system of research composed of many small labs, none of which can really offer any sort of career path for non-academic professionals.
On the one hand, attitudes like the third reviewer's are a primary reason for the state of HPC today, where new advances in research are just as often finding a way to run 30 year old code on a modern supercomputer as they are writing new software to take advantage of the fantastic array of new hardware. The number of times you hear "sorry, we can't change that. Livermore wrote that 15 years ago, and nobody knows how to change it anymore" is enough to drive a rational person over the edge.
On the other hand, the state of academic peer review being what it is, I don't fault the reviewer for suggesting the paper be resubmitted using more common methodology. A conscientious reviewer has a lot of papers to review, and spends a good amount of time on the ones that aren't written by crackpots. While the author was of course fully in their rights (and may even have advanced the field) to use their own software written in a relatively uncommon language, for the results and methodology to be understood, that's asking a lot of a reviewer.
I'm genuinely unsure as to how I would have responded to the review request. I have much sympathy for both people involved.
Also, it was rejected, with an invitation to resubmit, which makes quite a bit of difference. Especially when you take into consideration that the reviewers expressed why they can't publish it in the rejection.
I got the strong impression that the editorial decision was based on other factors, with these comments only being minor ones.
Reviewer comments are not a list of absolute requirements; they're thoughts and suggestions that help both the authors and editors improve the manuscript. They will include a recommendation for publishing, and they can make that recommendation contingent on particular issues, but that's somewhat unusual.
It's not completely clear what's happening in this case, but I'm strongly inclined to think this was a 'minor comment' from a reviewer.
The three reviews were helpful and constructive, but these two comments infuriated me.
It's very common to address reviewer comments without actually changing anything. Basically, you say you disagree for reasons X, Y, and Z. The editor can disagree, ask for further clarification from the reviewer, or simply accept the argument. Nothing is set in stone.
As for the implication that there are other reviewers waiting in the wings with suitable experience, good luck.
I was attacking a theoretical review to make a point I wanted to make. The blog post doesn't say the reviewer recommended rejection on the basis of implementation in Julia, or that the editor's decision cited the language choice. My point was that if those things were true (which is not clear from the post), that would be bad. I make that point because in my experience, it's not uncommon for reviewers to take similarly unreasonable positions.
Where are you publishing? I have had reviewers block if certain experiments were not done or things were not done a certain way.
I know there must be a balance between bleeding-edge and stability, but if in research you cannot use the new tools, then there will not be any progress at all.
Fast forward it seems like the most widely used tool in computer data analysis. Did something fundamentally change?
They can be built by a single prof/grad student.
They can be used by anyone based on R's package-manager GUI.
Research in most fields is going to move more quickly than any software team can add functionality to monolithic programs like SAS. Therefore, most functionality becomes available only in R. Users - people who want to run specialized-domain-function-X on their dataset are best suited by learning R, then using their colleagues packages.
Also, R Studio has made the entry barrier to R very low...
2) CRAN, nothing like it for SAS & SPSS.
3) Early adoption by book authors. A whole generation of data scientists were taught in R.
I think this has been a big barrier to any sort of package ecosystem to rival CRAN. Even if your entire target audience has SAS or SPSS, if your new package depended on one or more of those modules, it's going to cost some of your prospective users hard cash to run it. Whereas R just installs the dependencies for you.
2) developing new statistical methods and other innovations (knitr, etc) is much easier in R or S-Plus than SAS, SPSS, etc. So R's improved a lot faster than the others.
Data scientists, on the other hand, largely rose from tech companies that hired software engineers and preferred open-source software. When I started in the field ~10 years ago there were very few reasonable open-source options outside of R. Matlab and SAS were prohibitively expensive, Octave was too immature, Python didn't have basic functionality like data frames. In short, it was our only viable option, so we used it. A lot of people started using it for the same reasons, so R's library support became the best. This is what keeps me semi-locked into R - I'll probably eventually move to Python or Julia, but R hit the critical mass first and in a huge way.
edit: I may be using the word library imprecisely here, which may be causing miscommunication. I mean 'packages', to be clear.
I believe An Introduction to R specifically notes that one should use other software to provide R with appropriate data.
But the bigger point I was making is that CRAN (and, as pointed out by bbgm, bioconductor) and R's package system have been around a long time before that. And, unfortunately, was built with _very_ little input from software engineers. :)
edit: incidentally, this is a case where the opensource ideas really help a project. SAS and Stata don't exactly encourage any motivated grad student to rewrite their graphics interface or report generation tools.
----
1 - very high usability, particularly (contrast to python) for users who aren't developers, don't think like developers, and don't want to be developers
2 - very high expressibility
3 - it's free, so you can install it on lab computers and your laptop and everywhere else without forking out tons of money (see the price of matlab)
4 - a generate of grad students used it and it, in turn, was taught to a generate of undergrads
5 - academic statisticians tend to like it and tend to implement stats techniques in it, so there's a wide body of shared code on cran implementing most analyses you can think of
The typo is too good. Cobold is the sprite that plays its tricks on us even 56 years after its first appearance.
However, I believe that moving to R is not the answer, as you will still be riddled with a lot of the problems that you faced in Stata such as a quirky syntax (except backticks. I hate those backticks). Maybe Python will catch up (I doubt it, what we need is a DSL, not something too general that requires loong commands to do a simple regress). My best bet would be Julia, but it's still a long way to go regarding things like missing values.
Also, I think you should choose equipment which best suits the experiment, not other people.
While I think that's true, a paper can potentially be much more impactful if its' results can be easily understood by others, and isn't that really the point of science? In that sense, it's better for the author to stay away from niche methods and tools unless absolutely necessary.
Reviewing research papers is really, really hard work to do well. It takes a massive time commitment to truly digest and understand novel work to the level where you can provide a quality critique and review.
It's much easier to throw out facile critiques like the ones these reviewers provided. It shows that you know your stuff and did your job as a reviewer, without requiring you to actually understand the authors' innovations or lack-there-of.
Most methods papers in most fields are ignored by the vast majority of researchers. Just like the author doesn't have the time to rewrite his code in R, his prospective audience doesn't have the time to learn a new software package just to try out his proposed method.
If the author actually wants to change the way research is conducted in his field, he needs to make it easy for others to try his method out and possibly change the way they do research. As of today, that means an R package.
Or, the author can dig in his heels, refuse to write an R package, and be ignored. Maybe that isn't fair, but as I've learned through my own experiences, that's the way it is.
And it's probably taken them so long to adjust to R, that they don't want to change again any time soon.
And you're right, the tool shouldn't matter, and I definitely agree with your stance on open source, but keep in mind those attitudes do exist. Speaking of attitudes concerning open source, here's one of my favourite blog posts about the subject: http://www.catuhe.com/post/Le-syndrome-du-Puppy.aspx
The program is not just "a tool". It's your proof. You may have written a proof in the paper, but that was just a practice run. It's the one you wrote in the code that is the real proof, because there is little garaunteeing that seasoned software developers write code that matches the spec, say nothing about" amateur" programmers in the sciences.
Now, that does not preclude innovation. People still invent new mathematical notation systems. It's usually with the express purpose of solving problems that can't easily or at all be stolved in current systems. It is the bleeding edge of the science of math. But physicists are still expected to use the math their colleagues will understand, or spend a lot of time explaining their new system (QM anyone?).
There is, of course, the issue that math is significantly less rigorous and specific than code. Math doesn't compile and doesn't run. Or rather, it gets compiled and ran by people. Part of avoiding new, arbitrary notations is so that your work is accessible for verification. That gives running code, especially coffee with a comprehensive set of unit tests, a distinct advantage over math. But that doesn't obviate the bed for verification.
If you are using any nontrivial piece of custom software in your science, it needs to be scrutinized just as much as the rest of the math in your paper--if not more so, since it does the actual work.
The first reviewer is almost certainly right. If they can't understand your code, then they aren't the equivalent of lay-users complaining about open source software. First of all, it is still an issue of much debate whether or not releasing source code to user's for software products is necessary. If you are writing code for your science, you have an absolute duty to open source that code (even if in a non-extendable way, we still need to see the code). You wouldn't make a claim without providing the math. And you wouldn't provide math other people couldn't understand, arguing "some math is better than no math".
I don't think that standard should be R, or that it necessarily has to always be the same thing. It can adapt over time. But until you manage to release a few papers on how Julia improves over R, in explicit detail... "when in Rome, do as the Romans."
Something like:
Reviewer 1: Review scientific theory (domain expert)
Reviewer 2: Review scientific methods and conclusions (senior scientist)
Reviewer 3: Review code for scientific accuracy (senior programmer)