Comment by scottlocklin | Hacker News Reader

Comment by scottlocklin | Hacker News Reader

Parent Full thread

Its definitely a n00b criticism; R is absolutely the shizznits for cleaning and transforming data. You have to use data.tables for larger data sets, but (God Bless Wes) it can't really be beat for this; it does an amazeballs job and has every doodad you can imagine for making your life easier.

Cleaning a broken csv or whatever: no, it is crap for that. You use awk/sed/tr and all that for such problems.

If you're the type who don't want to deal with R, I guess you can use it from the CLI. A couple of the R deploys I've done work like this.

The real problems with R are .... oh man .... so many. R inferno covers a lot of them as a language/environment. Weak database connectivity is another one. The thing which makes me batshit is the nodejsbro-ification of the package management system. Aka people chaining together things like node works; R's package manager isn't designed for this. But also the way code, packaged and otherwise simply rots between the many, many upgrades.

You could probably run and deploy scikit learn/pandas based code from 5 years ago without much problem. In R, you have to make a build with the salted package dependencies ... and for all I know stuff it in docker.

Anyway unlike python, it basically has every data transformation and statistical tool under the sun. I guess this is the price we pay.

It the little bits of R programming I've done, the biggest pain for me was debugging. If something fails, I had no idea what line of code the error happened. Since some of the functions were rather long, step-wise debugging was laborious.

Yeah, I had this problem until I discovered options(error=recover) which drops you into a nice lispy stack. And ya, someone told me about it: I don't know where you'd read about such things. There's probably a dozen alternatives I also don't know where to read about.

Theres a pretty nice chapter on debugging in the great book Advanced R [1]. Hadley for the rescue! (As is commonplace in the context of R)

[1]: https://adv-r.hadley.nz/debugging.html

It's got tryCatch, and the browser function works wonders.

Besides the aforementioned browser(), try() and tryCatch(), you would check the packages assertive and assertr to guarantee the expected inputs, and testthat to create tests for your scripts. All awesome.

Is R really that good for data cleaning and transformation? It's slow, single threaded (yes, even for a lot of real-world use-cases with data.table) and memory-hungry. People only ignore this because code is generally written from the top of a notebook down to the bottom without ever being re-run.

It's pretty reasonable for medium sized data sets. When it falls down, I use J.

I have had this issue as well. Although to be fair, I would say this isn't the fault of R. Educators in the field of data science love notebooks because they can pair documentation, visualizations, and code all in one document. However, heavy reliance on notebooks produces a class of programmers who have very little clue how the code they are writing actually runs.

A popular counterpoint in the R community is that in many data cleaning tasks, the bottleneck is human understanding / coding time, not comptutation time. In other words, we'd rather spend 1 hour writing up a script that runs in 10 minutes and needs to be run a handful of times at most, than spend 6 hours writing something that takes 10 seconds.

Edit: This of course goes hand-in-hand with the claim that it is easier/faster to write R scripts. If you're not familiar with it, the tidyr and dplyr packages in particular (part of the tidyverse) are fantastic in the verbs they provide for thinking about data cleaning.

That is not true at all.

R has inbuilt great parallel tools (check for example the doSnow and future frameworks);

the best packages for data manipulation are mostly written in C (for example data.table and a good part of the tidyverse);

and with frameworks like Drake you can easilly create a Dag out of it that can process complex iterations millions of times. Check the uses of the Rcpp package that makes interfacing C code to R a breeze.

But of course, if you were comparing R to a pure compiled language, you are out of luck.

data.table is amazing. after getting over the learning curve i pretty much never touched dplyr again. pandas... lol. Hopefully https://github.com/h2oai/datatable lives up to its counterpart

managing python dependencies is no fun either, tbf

This take is pretty good on the R issues https://www.talyarkoni.org/blog/2012/06/08/r-the-master-trol...

I am not sure there is a language where the median user's code looks more different from 2012 to now than R's. RStudio and the tidyverse have basically created an all-new "standard" library for the language that has gotten extremely wide adoption, and it tends to smooth over a lot of R's warts if you stay with code that has been written to fit with the style of code that the tidyverse encourages.

This sounds like something that in most other languages, would have become R v2 (non-backwards-compatible)? Why does R retrofit instead of enforce best practices?

The short answer is that "tidyverse" is a set of third-party packages that was developed by a different team than base R. It's not a modification of the language itself. This is possible because of certain features of R that make it particularly syntactically flexible (see, e.g., https://adv-r.hadley.nz/metaprogramming.html).

I know of exactly nobody who was happy with how the Perl 5 to Perl 6 transition went, or the Python 2 to Python 3 transition. I think not ever having a firm cutover like that was beneficial to R.

EDIT: And yes, as citrate05 says, it's just an additional set of libraries. There's no changes to the language itself.

Python wasn’t pretty but I’m happy it was done. Than if we were still on python 2 with only a portion of the changes. There would be a lot of discontent if that was the case too.

Reply on news.ycombinator.com