A newcomer’s (angry) guide to R
arrgh.tim-smith.us
arrgh.tim-smith.us
I understand some of this stuff is actually seen as a positive for Python in some contexts (production usage) and I agree. Just pointing out the woke take is the languages are both good, but good at different things. If I need to run a quick analysis on a dataset, I'm grabbing R 9/10 times. If I'm building a production pipeline, I'm using Python 9/10 times. This is perfectly fine.
I'm also coming from more of an office setting where everything is in Excel. I've used R to reorganize and tidy up Excel files a lot. Ggplot2 (part of the Tidyverse) is also fantastic for plotting, the grammar of graphics makes it really easy to make nice and slightly complex graphs. Compared to my Matplotlib experiences, it's night and day. Though I'd expect my experience with programming to be quite different from others' though, mainly because any code I write is basically an intermediary step before the output goes back in Excel.
That said, if anyone's interested in learning R from a beginner's level, I can recommend the book R for Data Science. It's available freely at http://r4ds.had.co.nz/ and the author also wrote ggplot2, RStudio, and several of the other Tidyverse libraries.
EDIT: I'm also currently writing my master's thesis in RMarkdown with the Thesisdown package. It's wonderful, it allows for using Latex without really knowing Latex which is great for us in business school.
What is probably not a good idea (which the article unfortunately does) is to introduce people to R by talking about data.frame without mentioning data.table. Just as an example, the article mentions read.table, which is a very old R function which will be very slow on large files. The right answer is to use fread and data.table, and if you are new to R then get the hangs of these early on so that you don’t waste a lot of time using older, essentially obsolete parts of the language.
Detrimental to the runtime performance; if you happen to be reading and processing tabular data from a csv (which is all I've ever used R for, I must admit), then you get real performance gains as a programmer. For one thing, it allows a functional style where it is much harder to introduce bugs. If someone is trying to write performant code they should be using a language with actual data structures (and maybe one that is a bit easier to parallelism than R). The vast bulk of the work done in R is not going to be time sensitive but is going to be very vulnerable to small bugs corrupting data values.
Tidyverse, and really anything that Hadley Wickham is involved in, should be the starting point for everyone who learns R in 2018.
> languages like R and MATLAB that were designed for data frames and matrices
Personal bugbear; the vast majority of data I've used in R has been 2-dimensional, often read directly out of a relational database. It makes a lot of sense why the data structures are as they are (language designed a long time ago in a RAM-lite environment), but it is just so unpleasant to work with them. R would be vastly improved by /single/ standard "2d data" class with some specific methods for "all the data is numeric so you can matrix multiply" and "attach metadata to a 2d structure".
There are 3 different data structures used in practice amongst the R libraries (matrix, list-of-lists, data.frame). Figuring out what a given function returns and how to access element [i,j] is just an exercise in frustration. I'm not saying a programmer can't do what I want, but I am saying that R promotes a complicated hop-step-jump approach to working with 2d data that isn't helpful to anyone - especially non-computer engineers.
For attach metadata to an anything, why not use attributes()/attr() or the tidy equivs? Isn't that what it is for?
It might not make you feel much better, but data.frame is just a special list, c.f. is.list(data.frame()). So, if you don't want to use the connivence layers for data.frame you can just pretend it is a list and reduce the ways of accessing data structures by one.
You can paper over the distinction between data.frames and matrices if it comes up for you often enough. E.g.
`%matrix_mult%` <- function(x,y) { if("data.frame" %in% class(x)) { x <- as.matrix(x) stopifnot(all(is.numeric(x))) } if("data.frame" %in% class(y)) { y <- as.matrix(y) stopifnot(all(is.numeric(y))) } stopifnot(dim(x)[2] == dim(y)[1]) x %*% y }
d1 %matrix_mult% d2
... but I'll grant that isn't the language default.
The issue is that I learn these things /after/ R does something absolutely off the wall with its type system. And a lot of my exposure comes from using other people's libraries.
For my own work I just use tidyverse for everything. It solves all my complaints, mainly by replacing apply() with mutate(), data.frame with tibble and getting access to the relational join commands from dplyr. I'll cool with the fact my complaints are ultimately petty.
> For attach metadata to an anything, why not use attributes()/attr() or the tidy equivs? Isn't that what it is for?
I've never met attr before, and so am unaware of any library that uses attr to expose data to me. The usual standard as far as I can tell is to return a list.
> It might not make you feel much better, but data.frame is just a special list, c.f. is.list(data.frame()). So, if you don't want to use the convenience layers for data.frame you can just pretend it is a list and reduce the ways of accessing data structures by one.
Well, I could. But data frames have the relational model embedded into them, so all the libraries that deal with relational data use data frames or some derivative. I need that model too, most of my data is relational.
The issue is that sometimes base R decides that since the data might not be relational any more it needs to change the data structure. Famously happens in apply() returning a pure list, or dat[x, y] sometimes being a data frame or sometimes a vector depending on the value of y. It has been a while since I've run in to any of this, because as mentioned most of it was fixed up in the Tidyverse verbs and tibble (with things like its list-column thing).
> `%matrix_mult%` <- function(x,y) { if("data.frame" %in% class(x)) { x <- as.matrix(x) stopifnot(all(is.numeric(x))) } if("data.frame" %in% class(y)) { y <- as.matrix(y) stopifnot(all(is.numeric(y))) } stopifnot(dim(x)[2] == dim(y)[1]) x %*% y }
I have got absolutely no idea what that does in all possible edge cases, and to be honest if the problem that is solving isn't actually one I confront often enough to look in to it.
It just bugs me that I have to use as.matrix() to tell R that my 2d data is all made up of integers, when it already knows it is 2d data (because it is a data frame) and that it is made up of integers (because data frame is a list of vectors, which can be checked to be integer vectors). I don't instinctively see why it can't be something handled in the background of the data.frame code, which already has a concept of row and column number. Having a purpose-built data type only makes sense to me in the context that at one point they used it to gain memory efficiencies.
I mean, on the surface
data %>% select(-date) %>% foreign_function() and data %>% select(-date) %>% as.matrix %>% foreign_function()
look really similar, but changing data types half way through is actually adding a lot of cognitive load to that one-liner, because now I have to start thinking about converting data structures in the middle of what was previously high-level data manipulation. And you get situations that really are just weird and frustrating to work through, eg, [1].
But they are some absolutely amazing features to use. After helping my wife learn R, and learning about all the dypler features, going back to other languages sucked. C#'s LINQ is about as close as I can get to dypler like features in a main stream language.
Of course R's data tables and data frames are what enable dypler to do its magic, but wow what magic it is.
I keep trying to type something that is actually pronounceable! ;)
I also base this on my own experience. I typically work with 2-3 million row datasets. I found that doing certain data operations was quite slow in plyr but a lot faster in data.table. It’s possible that if I had spent time reordering my plyr pipelines and filtering out unneeded columns or rows, then it would have worked better. However, data.table doesn’t require such planning ahead and thinking about what columns/rows you need to send to the next operation in a pipeline, because multiple operations can be executed from a single data.table call, and the underlying C library is able to make optimized decisions (like dropping columns not requested in the query), similar to an in-memory SQL database. So between dealing with slow code while doing interactive analysis, and/or having to spend time hand-optimizing dplyr pipelines, I found data.table to be a significant improvement in productivity (other than the one-time effort of having to rewrite a few internal packages/scripts to use data.table instead of dplyr)
BTW one thing that always made me avoid DT (I even preferred sqldf before dplyr was created) was its IMHO weird syntax. I always found the syntax of (d)plyr much more convenient. ATM it seems to me that dplyr has won the contest of alternative data management libraries. I cannot remember when I last read a blog post, article, or book that preferred DT over dplyr. I'm old enough to have learned that wrt libraries, it's wise to follow the crowd.
# Given a dataset that looks like this…
> head(dt, 3)
mpg cyl disp hp drat wt qsec vs am gear carb name
1: 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4 Mazda RX4
2: 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4 Mazda RX4 Wag
3: 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1 Datsun 710
# What's the mean hp and wt by number of carburettors?
> dt[, list(mean(hp), mean(wt)), by=carb]
carb V1 V2
1: 4 187.0 3.8974
2: 1 86.0 2.4900
3: 2 117.2 2.8628
4: 3 180.0 3.8600
5: 6 175.0 2.7700
6: 8 335.0 3.5700
# How many Mercs are there and what's their median hp?
> dt[grepl('Merc', name), list(.N, median(hp))]
N V2
1: 7 123
# Non-Mercs?
> dt[!grepl('Merc', name), list(.N, median(hp))]
N V2
1: 25 113
# N observations and avg hp and wt per {num. cylinders and num. carburettors}
> dcast(dt, cyl + carb ~ ., value.var=c("hp", "wt"), fun.aggregate=list(mean, length))
cyl carb hp_mean wt_mean hp_length wt_length
1: 4 1 77.4 2.151000 5 5
2: 4 2 87.0 2.398000 6 6
3: 6 1 107.5 3.337500 2 2
4: 6 4 116.5 3.093750 4 4
5: 6 6 175.0 2.770000 1 1
6: 8 2 162.5 3.560000 4 4
7: 8 3 180.0 3.860000 3 3
8: 8 4 234.0 4.433167 6 6
9: 8 8 335.0 3.570000 1 1
You can see that the interactivity is great at helping you compose answers step-by-step, molding the data as you go, especially when you combine with tools like plot.ly to also visualize results.
I still think R has an atrocious design as a programming language (although it also has its beautiful side - like when you discover that literally everything in the language is a function call, even all the control structures and function definitions!). It can be optimized for this sort of thing, while still having a more regular syntax and fewer gotchas. The problem is that in its niche, it's already "good enough", and it is entrenched through libraries and existing code - so any contender can't just be better, it has to be much better.
Scientific applications require things to fail hard and often, to aggressively fail whenever anything is potentially behaving incorrectly. R does the exact opposite of that in several different, pernicious ways. IMHO, Python is more dangerous than a scientific computing language should be, but at least it will stop when it hits an error. R has undoubtedly cost humanity millions of dollars in wasted research costs and caused untold confusion, from otherwise perfectly-performed studies reporting corrupted statistical results. The world would be a noticeably better place without it.
I simply cannot articulate my opinion about R without sounding grossly hyperbolic. I'm sad that HN, a place which is typically enlightened in the ways of the programming arts, is so confused what this article is on about. If we tolerate such blatantly hostile design in something as important as the language of scientific statistics, where do we expect to get?
It's unfortunate that you've gotten to a _terrible_ feeling about R without realizing that many of the 'silent' failures are easily configured away (some examples, https://github.com/hadley/strict). That R isn't noisy about things that CS majors might think it should be by default is, BTW, entirely appropriate. Many of what one might call 'silent' failure modes in R are for the express purposes of making exploratory data analysis easier... and that was one of the original purposes for R.
To do the first really well means automating away many of the issues that would crop up in the second allowing R to 'just work'. Because of that nothing beats R for getting to an answer as fast as possible (not even Python) at the cost of making it more difficult to productionise a solution in pure R.
Given its huge popularity and free-nature the benefits clearly outweigh the costs by a large factor.
And it really is insane and horrible.
Even so, if you want to use R as the production system, you shouldn’t implement the jumbled spaghetti code an iterative analysis involves just for your own sanity's sake. A rewrite is always required at which point hello Python
Yes, R is not my preferred language for anything heavy-duty, but I would guess ~95% of R usage is on datasets small enough to open in excel, and that is where the language truly shines (aside from being fairly friendly to non-programmers).
So yes, there are some problems with R, but what are your proposed improvements? Because if I have to analyze a .csv quickly, I'm going for R most of the time.
I have very quick flows for data processing: load data, make long form, add meta data as categories, plot many things with seaborn one-liners. I use Jupyter lab and treat it Like a full lab notebook, including headers, introduction, conclusion, discussion. Works very wel for me.
Julia?
Things that are OO based like C++ and Python are pretty bad at representing math because they put forward an idea of the actual representation (the object) as what matters, instead of the actions it performs (the function overloads). This may be good for some disciplines, but in a mathematical algorithm I really don't care what kind of matrix you gave me for `A`, I just want you to do the efficient `Ax=b` solve and have the action of the solver choose the appropriate method to abstract away the data. In Python you'd have to tell it to use the SciPy banded matrix solver, in Julia your generic ODE solver will automatically use the banded matrix solver when it's a banded matrix. This then allows for a composibility where the user overloads the primitive operations on their type, and your generic algorithm works on any data representation. This matches the workflow of math where an algorithm is proven on L2 functions, not on functions represented with column-wise indexing and ...
Case in point, the bug I raised about TZ handling (which is also an example of silent failure):
I'm one of the original authors of Presto, a distributed SQL engine for analytics on big data. From the beginning, we've been careful to follow the SQL standard and do everything possible to either return the correct answer or fail the query. For example, an addition or sum aggregation on an integer will fail on overflow rather than silently wrapping.
Returning an incorrect answer or silently corrupting your data is the worst thing a database can do.
For many years before you mounted your high horse, scientists were writing equally shitty code in Perl. When they've moved on from R, they'll write shitty code in some other language.
Now it's certainly showing its age and the limits of its design, but it's still best in class for a certain kind of user. We could do better for software development, but it's not clear that doing so would actually make data analysis easier.
To me, the opposite is true. People with no CS background would benefit the most from a simple design.
> in R, data types are pretty fungible, everything is a vector, coercing things generally "just works".
Things just work until they don't, and then you need to understand all the weirdness of R.
I don't know what's the typical experience of a non-programmer with R, but as a programmer, I had some headache trying to understand R semantics (apparently I'm not the only one [1]).
That said, R can be goddamn frustrating at times because of the way the documentation is written. It would be nice to simply be able to query about a function and get a cogent help file that explains THE BASICS of how to use the function for the most common use-case(s). Instead, the help files try to be "canonical" and front-load a bunch useless technical detail-- like that something is an "S3" object. Still haven't figured out what that really means and, I expect, that knowing something is "S3" will NEVER help me out when I am in a jam and need a little help to do something simple because I forgot some data manipulation detail.
Instead, I end up googling all the time connecting the dots all over the internet to get very simple stuffs done. At least now we have stackoverflow which, as vicious as it is, seems like Mister Rogers' Neighborhood compared to the old R mailing list.
Yes, well, from the outside it doesn't seem like Python programmers have any better grasp of those either.
I don't understand the difficulty I've often seen voiced against this. Why would a newbie or someone who just wants to get analytical work done need anything beyond installing Python and doing `pip install library`? It's certainly orders of magnitude easier and faster than, say, using a C library. The only trouble I can see a newbie running into is if they want to install a library which doesn't have precompiled wheels and they need some dependencies to build it, but that's rarely an issue for popular packages.
Also, they can't easily search through the packages in a nice GUI and just click on the one they want to install.
Hmm, not really. It's actually advised against [1].
[1] - https://askubuntu.com/questions/802544/is-sudo-pip-install-s...
pip install --user
So you have to install extra Python packages, as root. You have to have that Python experience that guides you to install as few of them as you can, just enough so you can get started with a virtualenv, so you don't end up relying on your system Python environment.
And this is really hard to explain to people who aren't deeply familiar with Python. "Never use sudo to install Python packages! Oh, you got errors. We obviously meant use sudo for two particular packages and never again after that."
In the terrible case where you don't have root, you have to ignore Ubuntu's version of Python and compile it yourself from scratch. Hope the right development libraries are installed!
Maybe I'm wrong and there's a method I've overlooked. If there is: please show me how to install a Python package on a fresh installation of Ubuntu 16.04, without ever using sudo, and I will happily spread the good news.
On Windows, meanwhile, the standard Python installer gets all this set up properly in like three clicks. Better yet, because it installs per-user by default, "pip install" just works. And if you still choose to install it globally, it will fail, but it will tell you exactly what you need to do to make it work:
Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: ...
Consider using the `--user` option or check the permissions.
Then again, with Python in particular, I have often had errors either with pip-install, or after "successful" installation, for various reasons.
Not sure if there's support for upgrading all packages at once, though.
Except when it isn't. For instance, because some wheel fails to build because you're lacking the VC++ redistributable (or it's not where pip thinks it should be):
C:\Users\YeGoblynQueenne\Documents\Python> pip install -U spacy
Collecting spacy
Downloading spacy-1.2.0.tar.gz (2.5MB)
100% |################################| 2.5MB 316kB/s
Collecting numpy>=1.7 (from spacy)
Downloading numpy-1.11.2-cp27-none-win_amd64.whl (7.4MB)
100% |################################| 7.4MB 143kB/s
Collecting murmurhash<0.27,>=0.26 (from spacy)
Downloading murmurhash-0.26.4-cp27-none-win_amd64.whl
Collecting cymem<1.32,>=1.30 (from spacy)
Downloading cymem-1.31.2-cp27-none-win_amd64.whl
Collecting preshed<0.47.0,>=0.46.0 (from spacy)
Downloading preshed-0.46.4-cp27-none-win_amd64.whl (55kB)
100% |################################| 61kB 777kB/s
Collecting thinc<5.1.0,>=5.0.0 (from spacy)
Downloading thinc-5.0.8-cp27-none-win_amd64.whl (361kB)
100% |################################| 368kB 747kB/s
Collecting plac (from spacy)
Downloading plac-0.9.6-py2.py3-none-any.whl
Requirement already up-to-date: six in c:\program files\anaconda2\lib\site-packages (from spacy)
Requirement already up-to-date: cloudpickle in c:\program files\anaconda2\lib\site-packages (from spacy)
Collecting pathlib (from spacy)
Downloading pathlib-1.0.1.tar.gz (49kB)
100% |################################| 51kB 800kB/s
Collecting sputnik<0.10.0,>=0.9.2 (from spacy)
Downloading sputnik-0.9.3-py2.py3-none-any.whl
Collecting ujson>=1.35 (from spacy)
Downloading ujson-1.35.tar.gz (192kB)
100% |################################| 194kB 639kB/s
Collecting semver (from sputnik<0.10.0,>=0.9.2->spacy)
Downloading semver-2.7.2.tar.gz
Building wheels for collected packages: spacy, pathlib, ujson, semver
Running setup.py bdist_wheel for spacy ... error
Complete output from command "c:\program files\anaconda2\python.exe" -u -c "import setuptools, tokenize;__file__='c:\\users\\yegobl~1\\appdata\\local\\temp\\pip-build-7o0roa\\spacy\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'ex
ec'))" bdist_wheel -d c:\users\yegobl~1\appdata\local\temp\tmpypkonqpip-wheel- --python-tag cp27:
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-2.7
creating build\lib.win-amd64-2.7\spacy
copying spacy\about.py -> build\lib.win-amd64-2.7\spacy
[217 lines truncated for brevity]
copying spacy\tests\sun.tokens -> build\lib.win-amd64-2.7\spacy\tests
running build_ext
building 'spacy.parts_of_speech' extension
error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27
----------------------------------------
Failed building wheel for spacy
Running setup.py clean for spacy
Running setup.py bdist_wheel for pathlib ... done
Stored in directory: C:\Users\YeGoblynQueenne\AppData\Local\pip\Cache\wheels\2a\23\a5\d8803db5d631e9f391fe6defe982a238bf5483062eeb34e841
Running setup.py bdist_wheel for ujson ... error
Complete output from command "c:\program files\anaconda2\python.exe" -u -c "import setuptools, tokenize;__file__='c:\\users\\yegobl~1\\appdata\\local\\temp\\pip-build-7o0roa\\ujson\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'ex
ec'))" bdist_wheel -d c:\users\yegobl~1\appdata\local\temp\tmp8wtgikpip-wheel- --python-tag cp27:
running bdist_wheel
running build
running build_ext
building 'ujson' extension
error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27
----------------------------------------
Failed building wheel for ujson
Running setup.py clean for ujson
Running setup.py bdist_wheel for semver ... done
Stored in directory: C:\Users\YeGoblynQueenne\AppData\Local\pip\Cache\wheels\d6\df\b6\0b318a7402342c6edca8a05ffbe8342fbe05e7d730a64db6e6
Successfully built pathlib semver
Failed to build spacy ujson
Installing collected packages: numpy, murmurhash, cymem, preshed, thinc, plac, pathlib, semver, sputnik, ujson, spacy
Found existing installation: numpy 1.11.0
Uninstalling numpy-1.11.0:
Successfully uninstalled numpy-1.11.0
Running setup.py install for ujson ... error
Complete output from command "c:\program files\anaconda2\python.exe" -u -c "import setuptools, tokenize;__file__='c:\\users\\yegobl~1\\appdata\\local\\temp\\pip-build-7o0roa\\ujson\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, '
exec'))" install --record c:\users\yegobl~1\appdata\local\temp\pip-ibtvwu-record\install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_ext
building 'ujson' extension
error: Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat). Get it from http://aka.ms/vcpython27
----------------------------------------
Command ""c:\program files\anaconda2\python.exe" -u -c "import setuptools, tokenize;__file__='c:\\users\\yegobl~1\\appdata\\local\\temp\\pip-build-7o0roa\\ujson\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --recor
d c:\users\yegobl~1\appdata\local\temp\pip-ibtvwu-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in c:\users\yegobl~1\appdata\local\temp\pip-build-7o0roa\ujson\
Note that this is just one case where I was trying to install one particular package. I got a couple more examples like this in my installation diary, notably one when I tried to install matplotlib, this time on Windows Subystem for Linux, a.k.a. Ubuntu, and hit a conda bug that meant I had to use an older version of QT until upstream fixed it and other fun times like that.
Unless the package needs a native component like libcurl of particular version then it can turn into couple of hours of blindly trying everything you can think of.
> Another great example: in R, data types are pretty fungible, everything is a vector,
Unless it's a dataframe or factor or string or s3, s4 or s5 or a couple of other things.
And the documentation will tell you the reference paper that you can read and some completely impractical example.
Ugh, feels better now, sorry for the rant.
This implies we strive for good design in languages just because it appeases some ideal we have about how languages should be. But really we strive for good design in languages because it makes them more powerful, more expressive, easier to use, etc. Sure, maybe Python doesn't have all the right abstractions to be perfectly suited to statistical tasks, whereas R has more natural abstractions for that kind of stuff. But that doesn't mean that R doesn't also have many objectively bad design decisions even for statistical uses.
I think R-Studio (an R based IDE that turns it kinda into a more excel like experience) where you can inspect the data in memory (including matrix data) and graph making is where it really helps bring people into the R language. And with a set of instructions anyone can go load the analysis packages and do their data analysis.
Compare this to python, where they have to go the unix shell set up the environment, load the libraries. When they come back reset everything and get back to where they started.
I've come to really like the way environments work in R, as well.
I think the reason is that R is not just a badly designed language, but in particular its design is inconsistent. That’s as confusing to newcomers as it is to people who care about PL design.
I used R for almost a decade. Last year I switched to Python and Jupyter, never looked back. Can’t recommend the switch highly enough. R has great stats packages, but struggling with the language is just not worth it.
There is that. Matlab has the same problem.
One problem with "real programming languages" is that programmers who grew up with C don't see any need for built-in multidimensional arrays. This is one reason FORTRAN is still around, and why array work is straightforward in Matlab and R.
Anaconda is becoming to python what Chrome is to browsers, particularly as Jupyter matures. Drop it in, and a huge amount of what you want to do is ready to go. Sure, there's lots of libraries/extensions available, but most of the time you can do real work with a default userland, non-privileged install.
It is just ridiculous to call the first row in a data set as 0th row, and the last row as (n-1)th row. It does not make any sense for data analytic work.
I find zero indexing logical: zero is the first natural number and is thus a fine candidate for being the first ordinal.
In my experience most mathematical series lose nothing in terms of elegance or readability by being indexed from zero instead of using more traditional indexing from one.
I've actually found the opposite, though admittedly it doesn't come up often enough to be a major nuisance.
The reason for this is the set {1...n} has order n, but the set {0...n} has order n+1, so you get lots of off by n+1 inelegancies or errors when order is important. It's better to be explicit at the set level when you need an {0...n-1} set, because usually the order gets passed around to later expressions and not the set element, so there's less algebra.
Zero indexing is great when your index is an offset, as it is for true arrays.
My ideal calendar has 12 30 day months, days 0 to 29.
If it's 10th June and you have an appointment for 0 August, that's 50 days from now.
At the end of the year, a 5 or 6 day 'month' called Holiday. 30th December becomes Christmas Day (observed) and 0 January is still New Years. New Years Eve is either 5 or 6 Holiday.
6 Holiday would be a lot of fun. ^_^
There are a few times where I’ve had to formally write out a zero-based indexing scheme of a given expression because python indexing can seem so weird in this case. For example, “lag zero” just sounds like a funny way of talking about lag 1 (to me). Of course, if you’re predicting y_t+1 then it sort of makes sense that y_t[+0] would be paired with B_0.
Then there is the whole thing about how, say, range(5) will return 0,1,2,3,4 and NOT 5.
All of this makes sense once you use python for a while but if you spend most of your time writing with pencil then it will probably take some adjustment.
These who tried to do modeling using vanilla R just dropped midway because of R performance and switched back to fortran
ggplot2 just should be baseline instead of default gfx.
My company has tons of python code producing reports with reportlab, making UIs with PyQt5, as well as a multitude of small scripts to interact with MySQL.
We’ve been nothing but happy with Python in the years of using it.
I'll add that concepts like data frames are not really intrinsic, and you get needless complexities like "length", "nrow", "dim", each of which does the wrong thing in 90% of the scenarios of interest. The confusion of lvalues is another strange quirk -- a <- 0; length(a) <- 20 is totally valid, and you get things like class(a) <- 'foo' being preferred over the equivalent a$class <- foo. It has all sorts of odd concepts between lists and data.frames -- the double-bracket syntax, etc. The object model is very confusing, though most people seem to have converged on the S3 system, which is the oldest one.
If you discipline yourself to learning "the good parts", especially by learning either data.tables or tidyverse or becoming a master of split/lapply/aggregate/ave, then it is very powerful. The modelling tools and plotting (both base graphics and ggplot2) are excellent.
I'd love to see a NeoR arise at some point that fixes the strange historical inconsistencies (like what happens when you refer to vec[0], as noted by the author) in non-backward compatible ways.
[1]That is, top-level elements. If a list L contains 3 vectors, each with 9 elements, length(L) is 3, not 9 or 9*3 = 27
Microsoft (or perhaps the Apache Foundation) could pull such a trick, and win a huge user base by doing so.
My guess is that several top R developers must have also thought of doing so, esp. those who write C code for R.
I always feel bad when I resort to a for loop in R. Do you have any recommended ressources about mastering those specific manipulations?
Sometimes a loop is vastly more performant if you count the amount of time it takes to get the "idiomatic" way working and working in a way that allows easy troubleshooting.
getting comfortable with purrr has been enormously helpful in consolidating my code and avoiding unnecessary loops
R may not be the most "beautiful" language in a general perspective, but it certainly is more beautiful than Python when it comes to actual data analysis. There is nothing in R that is as ugly as even the best implemented pandas, numpy, and matplotlib code. All of the options in Python, which is generally pointed to as the "superior" language to R, feel tacked on and hackish.
The real story behind most of the complaints is that they come from software developers who only rarely need to do data analysis that would require R, and therefore use it infrequently and mistake their unfamiliarity with the language with the language being bad.
I also groaned at the part where the author struggled to google questions about R because of its "stupid name". I have literally never, ever had issues Googling anything about R they same way I haven't ever had issues finding answers to my questions about "Python". The author is grasping at straws, and this is a programming blog's equivalent of clickbait.
R is a terrible name and it is not up for a debate. Whenever you name a product, company or in this case a language as a letter "R", you're literally asking for trouble.
Just to be fair, C is also a horrible name. On the other end of the spectrum - Julia and Rust are excellent names for a programming language because they're unique in the context of programming.
Funny enough I most often get wrong results when googling something Rust related, because there's a town called Rust (Germany), so Google pushes the location based results up, and rust is also, well, oxidized metal, so sometimes I get DIY pages as a result.
I love pretty much every other design decision about Rust, but it's the one "hard" to Google language in my experience.
Maybe they should start using Rlang. Although it's going to be annoyingly close to erlang
> Index vectors like a[1] … a[4]. All indexing in R is base-one. Note that no error is thrown if you try to access a[0]; it always returns an atomic vector of the same type but of length zero, written like numeric(0)
That's serious WTF right there.
In general a lot of the complaints revolve around the language making error handling unnecessarily difficult which is something that will drive me up the wall with a language. I'm a fairly defensive programmer and if your language is fighting me when I'm trying to do error checking I'm not going to be happy. I can kind of understand the thinking of "just write a perl script to verify/reformat your data before passing it to R", but that doesn't help me find my own errors.
You can also access values by negative indexes, which have another special purpose in R that make it easier to quickly manipulate/analyze the data in the array, rather than causing an error.
It's just like how SQL won't throw an error if you try to do a SELECT...WHERE ID=non_existant_value, instead it will just not return any rows (or return NULL, depending on the exact query). But you wouldn't call SQL bad, you would just acknowledge that it serves different purposes and acts differently than something like Java or C.
Your comment is a great example of what the parent commenter was talking about: experienced "programmers" tend to dislike R simply because they are unfamiliar with it, and it therefor does things that they do not expect. But again, that doesn't mean the language is bad, it just means the users should probably become more familiar with their tools rather than trying to use a wrench to bang on a nail.
It's not the same: `a[90000]` would be `NA` (which is a 1-element vector, not an empty vector).
Having 1 based indexing is a design flaw that R shares with SQL. SQL's error handling also leaves much to be desired.
That works great for him, he can "not worry about it" and things work because he knows all the quirks.
If I just "don't worry about it" my programs don't work for mysterious reasons.
It seems likely that R could have been designed to have the same strengths without having so many weird and arbitrary quirks.
Not really... at least not practically. R is a successor to S which was written in 1976 (fun talk on the history: http://blog.revolutionanalytics.com/2016/07/rick-becker-s-ta...).
Many of the arbitrary quirks started out for the sake of backward compatibility with S. All that being said, yeah, you could probably design a 'modern R' without the weird and arbitrary quirks.
At any rate, at least it's not Pandas and matplotlib...
There's a certain yin and yang to the space, isn't there? You get to choose between a hacky language that has pretty good tooling built on top of it, or a pretty good language with hacky tooling built on top of it.
I think that Python is probably winning because being a decent language gives you a decent escape hatch, whereas no amount of great libraries can save you from having to go through the bizarro language.
That said, R may be bizarro, but at least, once you learn it, it's predictable. Whereas I'm not sure even Pandas really knows whether a given call to .loc will copy or refer to the original data.
1. R and Lisp are hardly alike even if it was inspire by it. It's like saying Erlang and Prolog is very similar. If you want learn FP do it in Erlang, Lisp, Haskell, etc.. Don't do it in R, it's half baked.
2. R syntax is ugly with warts. But built in datatype like dataframe, factor type, NA (missing value notion) value, make this language much better than many languages out there for dealing with data. Subsetting dataframe is a breeze even in base R.
3. There are many many advance statistical packages only in R. GLMnet was was in R for 4-5 years before someone decided to port it to Python. You can argue that there might be alternative package. But the statistician that created ridge, elastic, etc... method made GLMnet. There are many statistician out there that just implement their latest method in R. If you want to learn a subject in statistic there is probably a book out there and it'll have an R package and code to come along with it. Next to that will be SAS. There are very few stat book with python packages. You want to learn bayesian statistic? Social Network Analysis? There's a book for it with R code and a package to do that. Good luck finding one in Python for these subfield of statistic. There's a bayesian hierarchical analysis in Ecology and that book is in R.
4. ggplot2 is amazing for static graphic. R doesn't have good dynamic graphic out there and I kinda meh with Shiney. If you hate the syntax then you may learn to appreciate it by reading it from the creator https://www.r-bloggers.com/a-simple-introduction-to-the-grap...
They are very alike in the underlying core design, not in how you use them.
In R, everything is an expression, and every expression is a function call. Even things like assignments, if/else, or function definitions themselves, are function calls, with C-like syntactic sugar on top. You don't have to use that sugar, though! And all those function calls are represented as "pairlists", which is to say, linked lists. Exactly like an S-expr would - first element is the name being invoked, and the rest are arguments. And you can do all the same things with them - construct them at runtime, or modify existing ones, macro-style.
So in that sense, R is actually pretty much just Lisp with lazy argument evaluation (which makes special forms unnecessary, since they can all be done as functions), and syntax sugar on top. Where it really deviates is the data/object model, with arrays and auto-vectorization everywhere.
And yes, it's easy to break stuff that way. Just as easy as it is with macros (esp. non-hygienic ones).
Because of much better performance and predictability of code.
> as.list(quote(if (1 > 2) 3 else 4));
[[1]]
`if`
[[2]]
1 > 2
[[3]]
[1] 3
[[4]]
[1] 4
> `if`(1 > 2, 3, 4)
[1] 4
> `if`(1 > 2, cat(3), cat(4))
4
> f <- as.list(quote(function(x, y=1) { x + y }));
> f
[[1]]
`function`
[[2]]
[[2]]$x
[[2]]$y
[1] 1
[[3]]
{
x + y
}
[[4]]
function(x, y=1) { x + y }
> as.list(f[[3]])
[[1]]
`{`
[[2]]
x + y
> as.list(f[[3]][[2]])
[[1]]
`+`
[[2]]
x
[[3]]
y
> `function`(pairlist(x=1, y=2), quote({x + y}))
Error: invalid formal argument list for "function"
> eval(call("function", pairlist(x=1, y=2), quote({x + y})))
function (x = 1, y = 2)
{
x + y
}
> eval(call("function", pairlist(x=1, y=2), quote(`{`(`+`(x, y)))))
function (x = 1, y = 2)
{
x + y
}
> eval(call("function", pairlist(x=1, y=2), quote(`{`(`+`(x, y)))))(123)
[1] 125
I use python for most things, but there are so many packages that can only be found in R (especially in the bioinformatics world), so it becomes a necessity.
This is a very bad example of what factors are for in R, because it makes it seem like factors are for defining variables or keys in key value pairs. You can use them for that, but it isn't the intended use. A better example would be:
suppose you were comparing the amount of sugar in fruits based on several growing locations, and you had three columns:
| Fruit | Location | Density (g/L) |
Fruit would be a factor variable (let's say it takes the possibilities of apple, banana, orange), and location could be too, if it were a discrete set of possibilities (as opposed to lat/lon coords)
This author seems to forget that R was built for working with data in an analytical setting, unlike all of the languages he's comparing it to. It has creeped into other areas, but that seems to be because in the hands of a skilled user it is far easier to implement a data analysis solution. I'm sure someone will come in and say how much better pandas is, but on the small datasets, I'll stick with R, especially with how brittle and buggy matplotlib is.
That is the approach for tidy data, which is used a lot in the R tidyverse (http://tidyr.tidyverse.org/articles/tidy-data.html)
> That is the approach for tidy data, which is used a lot in the R tidyverse (http://tidyr.tidyverse.org/articles/tidy-data.html)
Do you have a reference to where Hadley et al. suggest using factors in a key-value system? I'm reading Wickham's books at the moment and have not seen this assertion. Indeed, I believe he would not state this, as he explains the utility of factors explicitly:
A factor is a vector that can contain only predefined values, and is used to store categorical
data... Factors are useful when you know the possible values a variable may take, even if you don’t
see all values in a given dataset...
HN was started by someone who wrote a book on, and flipped a startup using, Lisp. Python and Go are both used a lot by a buyer of startups, and HN exists to deliver startups, or their IP, to those buyers. R is more of a language for helping its users do data analysis, perhaps in corporate offices, and hence doesn't have as much use for HN's business purpose. Submissions on Python and Go are more likely to stick to HN's front page.
I use lisp and R. While R evolved from lisp, which let's me understand how it does certain things, I don't know if I'd describe it as more like lisp than python.
Indeed, the analogy I use to describe to friends why I have a strong emotional distaste for R is to use the following analogy:
Imagine you grew up as a heterosexual male. In your early years, you have fond memories of a young girl whom you had a fling with.
She drops off your radar, and you run into her 30 years later. She's gotten breast implants, botched her face with plastic surgery, and went through a rather traumatic divorce and reinvention of herself.
To your friends who lived on an island where there were no women and kept in basements and were regularly beaten by other stats programs, she might even be beautiful, and she certainly pays them attention to their base desires that they crave.
To you, she'll always be a mangled shadow of her former self and what could have been...
It absolutely is more like Lisp than Python, once you strip the syntax away:
I write a script. It doesn't work. I don't know why. I look at the error message, and then google for 30 minutes to understand what it really means - which parts of the code broke, why, how to fix them. Because none of the 3 things (which, why, how) is easy to get to.
OK, I fix it, having learned something new (like that there are infinite special cases with almost any functions).
I commit it to repo, go for coffee. In the afternoon, a colleague asks how to run that code. Well, it was a simple script, half a page, what's the problem?
I take a look, and on their machine it doesn't run. We don't know why. An hour later we discover she has some R profile file with a setting that changes behavior of some standard library... and she also has different encoding set as default, and so on, and so forth... whatever. I don't know why runtime environment encoding changes behavior of code that only deals with numbers, but hey! It's interesting at least. We fix it, we are happy.
A few days later I run the script again. It works. The result doesn't look right though. It's mostly zeroes. Hmm.
I run it a few more times, playing around with input, trying to figure out what's up.
OK, after a few minutes I realize there's lots of red color that flashes on running the script on my screen - just so fast I barely see it.
It turns out half the code isn't really running, the script just ignores it though (errors do NOT stop the code from running), and keeps going. It produces partial output happily announcing it finished.
That is the most serious mindfuck. Everything is OK, says the prompt, here's your 1 megabyte result of the calculation, oh, just don't look at the numbers, because I havent' really run any of the code... I couldn't find one of the functions.
I sit there wondering. Which is worse: the fact that every time I try launching the script something else is happening, or the fact that the runtime environment by default will return garbage with NO warning at the end (which is the only thing you see on screen) but with a million warnings in between (which you won't see unless you have really good reflexes...).
Which is worse?
I decided at some point, that I want a language to fail, and to always give me the same result. An error, an exception, this should kill the program and shout as loud as possible "Won't give you anything". Also I want code that ran yesterday to run today, and to run on my colleague's machine, and on a newer version of R. This was never our experience.
The other part is that it sounds like you don't have a standardized R environment. I admit that R's tooling there isn't the best, but there are options, e.g. {packrat} & {lockbox}... or better yet a Docker image.
I keep thinking someday I’ll try and build some of the missing stats infrastructure in Racket. The problem at present is time and that this stuff is real work!
Notably the use of + instead of chaining operators, the use of a custom "ggproto" object system instead of S3 (which makes extensibility a nightmare), and the superfluous presence of the aes() function (rendered unnecessary by better lazy evaluation tricks not really well-explored at the time).
I find this statement interesting - the aes() function helped distinguish attributes that were bound to values as opposed to being bound to constants. What would you propose as an alternative if the aes() function were eliminated?
EDIT:
Looking at ggviz [1], the presumptive successor to ggplot2, it seems that two new syntactic features will be used to distinguish constant vs. bound data.
But it is flexible, and it's fun. I have enjoyed following other's examples of recreating chart themes mimicked from the Economist and FiveThirtyEight. I think the latter often does employ ggplot2 with heavy customization to create some nice looking visualizations.
Because it's the "grammar of graphic" see the link below.
https://ramnathv.github.io/pycon2014-r/visualize/ggplot2.htm...
If you don't like that syntax use another library. But ggplot2 how you code graphic like that is amazing for me and many other people. I've tried SAS and Matlab and ggplot2 is the best.
...I am confused here. If you're testing for equality, R requires you to use == and not =. If you try to test for equality with =, it throws an error instead of treating it as an assignment. That's good. But who is trying to test for equality with <-?
if(a < -b)
if(b < -9)
if(b <- 9)
I clicked because I was a programmer for 15 years before I used R, and I have subsequently developed and shipped R packages, so I feel like I'm in a pretty good position to get the visceral, cathartic, "argh" the writer here was going for.
If you want "=" to (more or less) mean what it used to mean "long before computer programming", try Prolog.
?- a = a.
true.
?- a = b.
false.
?- A = a.
A = a.
?- A = a, A = b.
false.
?- A = a, A = B.
A = B, B = a.
?- A == B.
false.
?- A = B, A == B.
A = B.
(hint: "=" is not assignment, neither is it equality and "==" is just a stricter version thereof).
Of course I probably would have preferred just keeping the lispy syntax of scheme without all the infix stuff!
The second time, I had a purpose and found enough code to copy to achieve it. I was rewarded.
The third time, I had a more complex problem demanding use of JSON from Elastic Search, and found that the two packages out there in git are basically orphanware, use dplyr in extremely confusing ways, and offer little or no advantage to simplistic HTTP fetching and direct to JSON parsing. Which is a huge shame, because the idea of an elastic search abstraction is very attractive. But. "it just didn't work out of the box"
I am very clear I am an R "consumer" not an R developer. But, at this point, absent Shiny and a gui, I think that Python and Numpy has as much to offer me basically.
Some people say the syntax is FP friendly. I have been trying to learn FP in Haskell and I think R is about the worst notation you could invent to sell FP.
R is such a bad language to learn FP in.
99% of the time I use tidyverse with no noticeable impact on the performance. For that occasional 1%, I must admit datatable package works out really well. tidyverse pipes are so unixy that makes it easy to transition to command such as cut, head, sort and column if needed without any mental contortion.
I have used Python occasionally and with method chaining, it can almost simulate the "dplyr" like syntax. However, it is hard to find some obscure statistical test out of the box which is easy in R.
To me, this is all the encouragement I need to use R.
A few will deconstruct the criticisms and look for the small documentation or even language changes to solve something.
I applaud the final group.
Hmmmmm, okay after 20 minutes of squinting at the script I see there's a lowercase letter at the header name of the last column somewhere in the middle of the 200 lines...now we can move on to debugging the next "Error"...
R worshippers: "To a man with a hammer, everything looks like a nail." Interactively manipulating tables and producing visualizations is not what software engineers/programmers on HN would call "programming". Try learning a compiled programing language like C and creating a tool or three.
R haters: "It doesn't matter whether the cat is black or white, as long as it catches mice." Remember, R was designed by statisticians for statisticians -- nothing more, nothing less. Try manipulating a tabular dataset and producing visualizations in R, and see how easy and painless it is. There's a good reason behind all the hype.
And that guy talks about computer science? "<-" is a single token, what is so hard about that? It is a lexing issue, not a parsing issue!
Surely one cannot accuse OCaml of being written by laymen:
# type r = { mutable x : int };;
type r = { mutable x : int; }
# let y = { x=10 };;
val y : r = {x = 10}
# y.x <- 100;;
- : unit = ()
# y.x < -10;;
- : bool = falseIronically, you complain about wanting to learn statistical concepts, and not a language, but by focusing on a single language, they can minimize the amount of time spent on agonizing over lines of code trying to get to the same answer, and instead focus on the actual statistical problem.
Its also important to note that there are lots of potential pitfalls for people who don't know about them. For example, scikit-learn is the most popular python package out there for most data analysis work. But I would bet my ass that at least 50+% of its users don't realize that the logistic regression implementation implements L2 regularization by default, and in fact there is no (non-hackish) way of implementing non-penalized logistic regression. So you will be getting completely different answers than someone who would be using R, SAS, SPSS, etc. And the only way for someone to know this is for someone to understand the implementation of the function in all possible languages.
With a library and language I know, I can look though, advise on what went wrong and see how close you got. If I have to start supporting every language in the world, that's simply not reasonable.
You're funny!