Learn R Through Examples (2020)
gexijin.github.io
gexijin.github.io
Certainly R has its quirks, but most of this comes from being one of the oldest continuing existing programming languages there is. It derives from S which was written 46 years ago. Because of this it has multiple object/class systems reflecting the changing standards for OOP. It's most dominant one, S3, predates Java and therefore uses the Generic Function paradigm of OOP similar to Common Lisp's ClOS. If you're experienced but have never worked with non-Java style OOP you're going to be a bit confused.
R's most important feature, which is well worth studying and mastering for any serious programmer, is that it is a completely vectorized programming language. It borrows this style from APL (though is a million times more readable). Every value in R is a vector and for the vast majority of operations the best approach to solve your problem is by thinking in vector operations. This makes simple things like string formatting with `paste` seem like a confusing nightmare, but there is a real logic there. Functions like `ifelse` can seem strange, and writing C-style code in R, while possible will result in horrible performance.
Once you do learn to think in vectors you realize that R isn't just popular in the stats world because most statisticians haven't seen a "real" programming language, but because you can very rapidly iterate on models. Translating mathematical notation into R, for the experience R programmer, is easier than any other language I've worked with by a long shot.
My advice to any experienced programmer approaching R is to have some respect for the language. Most of the frustrations you'll have aren't because R is a bad language, but because you have less experience than you think and learning R well can expand your programming views in a similar way to Haskell.
And R excels at doing statistics and data science. If you keep that mindset, I believe many will find that its an excellent programming language.
We were essentially told "Install R Studio, then just copy and paste these library imports and you're good to go". Your description of a vectorized language makes total sense, and that single paragraph is more of an intro to R than we ever got in class.
That said, I think the reason this happened with this class in particular is the viewpoint of (what I perceive) as the majority user's of R. Mathematics focused researchers who never learned the language, they just have done enough to get by and don't really appreciate the underpinnings, or the nuances of running an environment on a machine that isn't theres.
I can't entirely absolve myself of blame though, once I realized what was happening I should have gone and done some more foundational R learning, but at that point I just wanted to be done with the class.
But for "the majority of professors who tangentially use R code in classes on statistics/bioinformatics/economics/finance (anything not explicitly about R and/or Data Science best practices)"? Absolutely.
The R code you see in industry (or academic labs where someone cares about modern R) looks vastly different from those script examples in college that are most people's first impression of the language.
Haskell is an advanced functional programming language. Most R stuff seems to be incoherent, hard to verify correctness, hacky. It does not seem built on a solid foundation like Haskell. Truly everything being a vector is not a huge take away.
As for "here's just a bunch of examples", well that seems sort of a brute force way to learn something. I agree examples are important, but they are usually to back something up. Having to reverse engineer some fundamental ideas out of just examples is more work. Seems like this is just promoting more hackyness. Seems like its training a neural network instead of actually understanding something.
If you define a language that "will expand your programming views" as one that can verify correctness easily then yeah, R is terrible at that. But so are many languages that are as flexible as R. Would you have the same opinion of FORTH? Or LISP? or TCL? I think these languages definitely count as "hacky" languages and yet they don't seem to draw the same derision as R (in my experience)
* Multiple dispatch. Before learning R, I knew about polymorphism in Java and C++, and multiple dispatch in R broadened my mind and turns out to be very handy.
* The idea of "frames". In R, when you invoke `lm(height~sex*age, data=mydataframe)`, the first argument (the formula) doesn't get evaluated until the lm command asks it to be evaluated, and lm can set up the "frame" for that evaluation, i.e. the place where variables are looked up, however it likes. In fact, lm sets it up to include variables from both the scope in which you invoked lm, and also from mydataframe. This is what makes R so wonderfully concise for modelling in data science, compared to e.g. Python + pandas. I knew about frames from interactive debuggers, but until R it never occurred to me that the programming language could manipulate them.
* "Held" arguments. In R, when you invoke `plot(x, y1+y2)`, it doesn't just evaluate the arguments and then call the plot function -- it leaves the arguments unevaluated, and invokes plot. Plot then (1) decides when to evaluate them, (2) gets access to the language expression `y1+y2`, which means that it can print "y1+y2" on the plot label, (3) it can even define extra variables to include in the scope when y1+y2 gets evaluated. (I knew about held arguments earlier, from Mathematica, but they only clicked when I read the R documentation.)
I've read that R is a descendent of Scheme, and that that's where it gets all its "manipulate language expressions" from. I don't know any Scheme, nor Lisp, and I should definitely learn them -- but in the meantime, my experience has been that R's ability to manipulate language expressions is what makes it such a wonderful sweet spot as a data modelling language. I mostly use Python + pandas nowadays, but it feels such a slog in comparison.
A. Understanding how to implement functions using syntactic forms is a steep learning curve. I remember running out of dplyr and having to implement a udf. Fairly unpleasant experience (enquo, !!, perhaps other unusual constructs). Felt like programming C macros.
B. "A function can decide where variables are looked up however it likes" is a significant obstacle in understanding how even basic constructs like function calls actually work. There is a non-trivial amount of hard-to-debug dark magic lurking behind every corner.
A middle ground has never been achieved. For example, `plot(expr(x), expr(y1 + y2))`, where the system limits the dark magic to explicit uses of the `expr()` construct, and `expr(x)` always means `{vars: vars(x), expr: (vars(x)) => x}`. Instead of patching interpreter environments, simply call a lambda function.
That's the weird thing about R. All this dark magic is hiding under the hood, but the core R team hid it so deftly that to the casual statistician it's a straightforward data modelling language that "just works". I'm not sure that it's possible to get rid of the dark magic and retain that data-modeller friendliness.
The same could technically be argued for Python; The current approaches to dealing with high-performance compute workloads either rely on JITing (e.g. Numba, Tensorflow/JAX's XLA) or bridging over to giant binary blobs through the CPython's well-supported C interop.
If you want to see the performance benchmark of R against major programming languages for data analysis please check this excellent keynote speech on the R compilation or more accurately on failures of R compilation effort [2].
As mentioned in [1], the main reason people do not focus on providing industrial strength compiler for R is probably because most of R program’s time are spent in the library codes that are written in a compiled language (e.g., C or Fortran). As you can see from [2], even though this is the case (computation intensive being delegated to proper compiled languages), the R programs running time still suffers due to the "impedance mismatch" because of the look up overheads, indirections, etc.
Perhaps someone should try to compile, transpile and/or embed R inside D language similar to the efforts provided in [3][4]. D now supports C compiler internally, has DasBetterC and D interface to C++ (DPP) is second to none. Additionally, since D can also perform better than Fortran codes for numerical computing perhaps Fortran codes can eventually be replaced by existing high performance D library like Mir[5]. I can foresee this symbiotic relationship can be beneficial for R and D (R on D?). D can becomes very popular with extra humongous data analysis and statistics libraries from CRAN, and R can get the run time and compilation performance improvement it badly needs.
[1]https://www.r-bloggers.com/2021/05/where-are-the-industrial-...
[2]https://www.youtube.com/watch?v=VdD0nHbcyk4
[3]https://dlang.org/blog/2018/06/20/how-an-engineering-company...
[4]https://theartofmachinery.com/2021/01/01/djinn.html
[5]http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/...
It’s not just vectorizing mathematical operations a la numpy.
S3 appeared a few years before Java but there were other OOP languages like C++ around at the time.
[0] There's an old article from Robert Gentleman named something like "S4 objects in 5 pages, more or less" but I can't find it. However, there's a mention of Dylan and CLOS here: https://genomebiology.biomedcentral.com/articles/10.1186/gb-...
EDIT: Here's the document I was looking to cite: https://www.stat.auckland.ac.nz/S-Workshop/Gentleman/S4Objec...
https://arxiv.org/pdf/1409.3531.pdf
Object-Oriented Programming, Functional Programming and R (John M. Chambers)
"Chambers and Hastie (1992), in the discussion of classes and methods, noted that S differed from other OOP languages because of its functional programming style. In fact, this version of functional OOP finessed the resulting distinction from encapsulated OOP in two ways. First, the methods were dispatched according to a single argument, the first formal argument of the generic function in principle. As a result, the methods were unambiguously associated with a single class, as they would be in encapsulated OOP. Methods were actually dispatched on either argument to the usual binary operators, but a number of encapsulated OOP languages do the same, under the euphemism of operator overloading.
"Second, the question of whether methods belonged to a class or a function was avoided by not having them belong to either. Methods were assigned as ordinary functions and identified by the pattern of their name: “function.class”. In any case, there were no class objects and generic functions were ordinary functions that invoked UseMethod() to select and call the appropriate method. Neither the function nor the class was able to own the methods."
Instead I would propose using R in some capacity, encountering it's weird quirks (why sometimes evaluating an expression prints something to console and sometimes it doesn't? How is dplyr using my column names as variables? Why do I get warnings when using & instead of &&?) And then turning to Advanced R as a source of sanity
Another source I would recommend is in fact the R language definition. It's very approachable and you quickly realize that R is pretty simple at the core, buried under piles of cruft
R’s future is inseparable from the tidyverse. We need to just lump them together in any serious discussion of R.
I teach a graduate level R course mainly for economics and statistics majors. (My educational background and career are computer science and technical; while that may stereotype me into Python, I just love R.) I spend the first three weeks on base R, to convey language-essential concepts, like vectorized objects, then the rest of the course is tidyverse-centric.
https://www.tidyverse.org/ There is also a free ebook that’s a good reference.
ggplot2 the plotting package included is pretty awesome
I took a biostatistics class and after the basic examples in R using the tidyverse to analyze data for projects was very helpful.
I spend the first three weeks on base R, to convey language-essential concepts ...
R by itself is nice. But the tidyverse is creeping in and bringing dependency hell with it.
at a push, base + data.table + ggplot2 get everything done. tinyverse ftw.
R is a fantastic DSL for data manipulation and statistical analysis, with both traditional and modern tools, on datasets up to the gigabyte scale. It has great, easy-to-use data structures and unparalleled APIs in the tidyverse. It is not the thing for the latest deep learning implementation on petascale data, but most data science work doesn't need or benefit from that. Surprisingly, some machine learning methods have their nicest APIs in R, because that's where the users of those methods are.
It has its warts, but so do JavaScript and SQL, and I think few people dispute that these are very powerful DSLs. Statistical analysis is just as legitimate a computing task as building webpages or querying databases. It is not the same as general-purpose programming, and it needs a good DSL.
Edit: I found an interesting quote from a guy named Martin Fowler about the subject.
“Languages can have a domain focus but still be general-purpose languages. A good example of this is R, a language and platform for statistics; it is very much targeted at statistics work, but has all the expressiveness of a general-purpose programming language. Thus, despite its domain focus, I would not call it a DSL.”
There are issues with R: lots of weird quirks; many different ways of doing things, often just supported for legacy reasons; poor error messages; to name just a few. But on the positive side, it really encourages functional-style programming through the use of apply functions and many of the new tidyverse packages. The resulting code can be very neat and less error-prone than equivalent python code.
The S3 system of OOP initially struck me as very weird, but now I see it as essentially just single dispatch, not really much of OOP at all. It works quite well and is extremely simple.
I guess my thoughts are: approach R as a functional language, and I think you'll find much to like. Try to write C++ or python-style OOP in it, and you'll just find it very strange.
I use only Python in my work. I learnt R and honestly, it's the same thing as using Python scientific packages. It's mostly vectorized operations, spaghetti functional, if people know how to write functions, code just to get it done. To make graphs, web dashboard(no, we are not doing web dev, it's dark magic frameworks), build machine learning and eventually some reports. Stuff like that.
I do some software engineering but that s optional and I do it because I can. Most data scientists/ ml engineers can't. So you guys are not fair. R and Python in these environments are not even being used for building stuff. This language is not build for that. Hence it's not good from the perspective you look at it from(software engineers).
Unlike Python , R is solely for statistics, data science and probably some basic ml( I haven't tried tho). Also Shiny for building web dashboards. But don't look at the code for dashboards, it's bad, with 'get it done and forget' approach.
That being said. Good luck scrapping, mining, cleaning data with something not called R/Python. Good luck with data engineering. Exploring and visualizing trends. Creating dashboards even. Machine learning. Monitoring and reporting in scientific manner.
Try This type of work with your favorite languages. Then see how quickly and easily it's done with R/Python . Come back and say it's bad language.
It's the same thing as embedded dev complaining about how bad js is for his job. You just totally ignore the context.
I do however recommend picking up data.table along the way because that is easily one of the best reasons to continue using R today.
Apart from that, I think there's a bigger challenge which still needs to be addressed is that analysis/modelling projects tend to be worked on by individuals and/or thrown away after the initial value is pulled out of them.
Going forward, I think we need to start identifying design methodologies that would make collaborating on this sort of work pain-free and more agile. Doing so should give us more value and sooner and for longer.
IMHO, you really have to embrace dispatch-oriented programming, and that includes being scrupulous about avoiding type instability. You also have to be a bit conscious about allocations, since it's easy to write Julia code (especially if you're trying to write in a "vectorized" style as is common in R, Python, Matlab) that generates absurd numbers of allocations, which must then be garbage-collected. But also easy to avoid those allocations if you know.
It took about two years, but after picking up more of this, I was eventually able to switch everything my group does from a two-language solution of matlab for scripts and plotting and C (with MPI) for HPC to all-Julia. This [1] was originally targeted at academics making the same switch, but much of it could be relevant to those with an R background as well.
[1] https://github.com/brenhinkeller/JuliaAdviceForMatlabProgram...
That said, about halfway through my master's about two years ago, I found myself in an intro to data mining course that was sold as an "we will teach you R". I had heard non-programmer math friends talk about what they had accomplished in R, and was excited to dive in.
Now, it didn't help that the class ended up being heavy on the statistics side (which despite a math/cs double major, stats was never my thing), but the actually learning R part was 99% left as an exercise to us alongside of the classwork required.
I can say without a doubt, learning R is the worst programming experience I've ever had. Our assignments would give some high level direction on which libraries to use, but getting the right libraries setup and in the environment was just an absolute nightmare. All I remember from that class is hours every week googling unreadable python pukes from R studio (because apparently everything data mining/ML related in R is actually just python), and then spending an hour or less actually doing the statistics work.
I feel bad because I feel like I was setup to not be able to give it a fair chance, but if that's what non-programmer math types are subjected to when told "you need to do some programming for your job", I can understand the apprehension.
I’m curious what libraries you were using, I’ve had to go into the source of quite a few popular libraries and I don’t think I’ve ever encountered Python. Lots of C and it’s derivatives, lots of Stan and FORTRAN, but I don’t think I’ve seen Python yet.
readr, caret, lattice, ggplot2, RColorBrewer, mlbench, ElemStatLearn, klaR, dplyr, arules, arulesViz, tensorflow
That's not Rs fault though
There are people that know how to code but those are few and in-between. Usually they are pushed out of academic positions because there are very few ways to fund work to develop scientific code.
Whilst it's certainly got some runs on the board. I think having data science folk work in more standard languages would actually make supporting them and their needs easier.
What in R made it feel academic for you?
Secondly, it seems optimised for producing one off figures and results, not production systems generally desired by industry.
Everything? I'm just wondering what you've been doing exactly. I know that Tensorflow in R is just a wrapper on top of Python, not sure what else. If you're doing any deep learning, then going straight to Python is certainly much better than using R. For most other things it's not quite as clear of a decision.
I think it is apparent only to you. The most popular package for ML in R is `caret` and it has nothing to do with Python. Similarly, `mlr` also has no Python. In fact, except for tensorflow and keras, I can't think of any major ML package that needs Python. Even torch package which brings pytorch to R doesn't need Python (https://cran.r-project.org/web/packages/torch/index.html).
What confuses me even more is that you were learning statistics but using R packages that use Python as backend. In my experience, almost all the new statistics and econometrics methods are first released as R packages by the researchers. Can you name any data mining R packages that you used that required Python? I am really curious to know.
One other thing I would mention is knowing SQL well is the most translatable skill. A lot of dplyr and pandas are doing SQL like operations (in fact dbplyr will generate SQL equivalent commands for your dplyr code for various backends).
In summary know how to manipulate data in SQL then pick a language (because you will need to do some IO/reporting stuff outside just data work) where the ecosystem of packages feels user friendly to you and your work flow and roll with that.