Hacker News Reader

Comment by AuthorizedCust | Hacker News Reader

I teach a data science graduate course to non-tech majors, and it's mostly on R. I welcome these changes.

I teach base R first for a few weeks, then I teach the TidyVerse as I introduce data science concepts, like text mining. I convey the TidyVerse as like an overlay on R, improving shortcomings and adding great functionality, syntactic sugar, etc.

This context switch is jarring to some students. It's great to see some of the TidyVerse's strengths--part of the tibble--be moved back into base R.

Now pardon me: I need to go. I start teaching dplyr in 51 minutes!

View on HN

I must admit to not really understanding the TidyVerse attraction, or why I should be doing "tidy data evaluation", rather than using base R.

If I want to use an advanced data manipulation library, I'd typically reach for data.table. If I want to use a verb based approach, why not SQL rather than dplyr.

I have tried dplyr and code I've written a few years ago still imports that package (hopefully it still works), but I just didn't find it was particularly useful or helpful compared to the alternatives.

I studied statistics, so R was the first programming I ever learned. I didn't know what it was at the time, but the tidyverse's chaining of operations on a data frame (via magrittr's %>%) was a really cool introduction to functional programming

The pipe operator in R also sort of does my head in.

I learned R in a way that everything went right to left. The variable on the left was manipulated by whatever (functions or other code) on the right.

The pipe operator reverses that flow, with everything now moving left to right, which I find difficult to follow and debug.

The best use of the pipe operator, in my opinion, is when every pipe followed by an end line, like this:

  data %>%
    filter(thing > 4) %>%
    mutate(new = fun(old)) %>%
    group_by(var) %>%
    summarize(new_mean = mean(new))

Then you are reading the code top-to-bottom, just like any other R script but without all of the temporary variables.

This only really works if nothing ever fails.

It's great for data analysis and terrible for programming.

The reason it sucks for programming is because you can't debug it or inspect intermediate variables, which is really really annoying.

It's great for one off transformations and plotting, but a really, really really bad idea for programming.

Then you add NSE, which makes it hard to functionalise procedural pipes (especially for people who learned tidyverse) and it's a recipe for unmaintainable and profoundly annoying legacy code.

That bring said,I love it for interactive analysis.

>The pipe operator in R also sort of does my head in.

As someone who taught myself base R from scratch in 2013, I (used to!) agree with this. When the pipe operator was first introduced, I’d roll my eyes whenever I saw a script that used it and move along.

But I forced my brain to adapt, and now it’s probably my favorite feature of the R language. Data science is full of sequences of transformations, and in my opinion it’s more readable and bug-resistant to phrase these long chains as:

  f(x) %>% g() %>% h()

rather than:

  h(g(f(x)))

or certainly:

  foo <- f(x) 
  foo <- g(foo)
  foo <- h(foo)

I can comprehend and modify others’ (to include past versions of me) R code much more quickly with this paradigm. You can quickly debug a chain by commenting out functions sequentially (i.e. first test: “f(x) # %>% ...”). It also becomes much faster to plug new transformations into the chain, when needed.

One thing that helps to keep track of the input x as it moves through the chain is using the “.” placeholder (especially when you need to specify function arguments), like so:

  f(x) %>% 
    g(., n=100, param=“baz”) %>%
    mean(.$column_name)

Here, the . stands in for “whatever is coming out of the pipe” from the left.

Tidyverse is more than dplyr. It also includes libraries like ggplot2, for which there really is no peer. Also stringr, for string manipulation, tidyr, which has a wide variety of very useful utility functions for working with data. dplyr is simply the backbone that connects them all and allows you to move between them without switching thought processes.

dplyr also offers functionality that many don't realize beyond its basic data manipulation. tibbles allow for arbitrary data types for columns, meaning you can have data frames containing your typical strings or floats, but you can also have data frames with columns consisting of nested data frames, or fitted models, or other atypical objects. Once you start working in this way, it can really streamline a lot of complex analytical processes, make your code much cleaner and easier to work with, and allows you to integrate them directly into other tidyverse packages, like ggplot2.

> If I want to use an advanced data manipulation library, I'd typically reach for data.table

Data.table is powerful, but from a usability/readability perspective, most people find it inferior to dplyr. And there are now packages for using the dplyr API and having it run data.table on the backend, so I personally see little to no use for using data.table by itself anymore.

> If I want to use a verb based approach, why not SQL rather than dplyr.

This seems like an overly simplistic reduction. There are bigger differences between dplyr and SQL than both being a verb based approach. And even were there not, dplyr is directly integrated within R. It is still much easier to do your data manipulation directly in R and handing it back and forth between dplyr and whatever other libraries you are using, than it is to do the same in SQL (even utilizing SQL queries within R).

The beauty of the tidyverse is that it is unifying the most important aspects of the data science process under a single approach and API philosophy. It integrates in a way that is pretty unprecedented not just in R, but in any programming language (where data analysis tasks are concerned).

I don't want to say that the Tidyverse is inferior, because it's not and it works for a lot of people. Hadley will also know much more about R and always be a better programmer than me.

However, there are other tools which predate Tidyverse, which I think so a stellar job at their core competency.

ggplot2 is great, but it was also around long before the Tidyverse concept. But at the same time base R plot() can also be pretty powerful[1] and look great.[2] As an alternative to ggplot2 I would also propose that Vega Lite[3] could be a contender, with an excellent cross language ecosystem.

There are also libraries available for applying SQL on dataframes from directly within R if that's what you want to do. sqldf[4] has been around for a long time, and now there is also the new duckdf[5], which is a bit quicker. Or one can use the DBI[6] library, which requires a bit more coding. Learning SQL is also a great skill which has a lot of value outside R.

Tidyverse may be useful, helpful and convenient for a lot of people, but I think we shouldn't lose sight of the wide R ecosystem which has provided a lot of alternative packages for a long time, perhaps without the marketing and profile of RStudio and the Tidyverse.

[1] http://karolis.koncevicius.lt/posts/r_base_plotting_without_...

[2] https://github.com/KKPMW/basetheme

[3] https://vegawidget.github.io/vlbuildr/

[4] https://cran.r-project.org/web/packages/sqldf/index.html

[5] https://github.com/phillc73/duckdf

[6] https://cran.r-project.org/web/packages/DBI/index.html

> includes libraries like ggplot2, for which there really is no peer

lattice is largely ignored but it is quite similar to ggplot2 in terms of features, and (as a matter of opinion) the plots it produces are aesthetically more pleasing, also great for making subplots using conditioning variables

SQL is incredibly verbose compared to dplyr. Modern non-SQL query languages coming out tend to be more more similar to dplyr, based on method chaining or piping data table objects through function calls, like UNIX pipes. It's much more composable than SQL and it just makes intuitive sense as a sequence of data transformation steps in a pipeline.

My favorite feature of dplyr that makes it stand out compared to SQL is that a window function is just a group_by() without a summarize(). There's no separate syntax for "PARTITION BY".

For data analysis, whenever possible, I don't even write SQL anymore, I use the dbplyr package, which is a dplyr-to-SQL compiler.

Thanks for sharing.

I do exactly the same, and I always find it hard to justify base-R in front of my students, after they learn about the Tidyverse ;)

I find tidyverse a bit slow, and I don't like having to memorize all the verbs. I typically reach for data.table these days as a nice middle ground between base R and tidyverse. Your thoughts?

There’s like 3 verbs.

Select, for columns Filter, for rows Mutate, for adding rows or changing values

On top of that there are a bunch of sugared versions of these, like rename, instead of select or transmute instead of mutate.

When you want to do something advanced you either combine those or go out to see if somebody else has already made that function for you. I don’t think that having functions for joins or removing NA’s is something special to tidyverse. Everybody needs to put such operations somewhere.

I find tidyverse to have a simple, expandable vocabulary that is easy to learn, read and grow.

But it is slow.

I never found any kind of logic behind data.tables API, and I find it hard to read. I haven’t found an introduction to it that makes anything click.

But it is fast and it has these absolutely amazing rolling joins

don't forget the crucial group_by() / summarise()

https://atrebas.github.io/post/2019-03-03-datatable-dplyr/#i...

This is how I learned dt.

Thank you for the link. I still end up confused.

How does selecting the 3rd column,

    dt[, 3]

jive with

>`DT[i, j, by, ...]` which means: “Take DT, subset rows using i, then calculate j ...

I don’t understand how column selection and calculating an expression is the same thing.

So now I can’t figure out if

    DT[, sum(V1)]

Will return the column indexed at the sum of V1 or if it will return the sum of the column called V1. How can I tell, from the syntax?

j is always for columns.

So for your first example it grabs the third column. The calculation you’re performing is a column selection.

For your second example, you’re returning all the rows, selecting column V1, and summing it.

I really prefer to name the operations

    dt[rows, columns, groups]

Thanks too. I had not heard of the "TidyVerse".

The TidyVerse is a opinionated group of R packages that all function similarly. ggplot2 is in there, and I already use that one.

I do like R for analysis. (I choose to use it over excel or Stata for my data analysis/ classes), though I wish the help pages were better.

Its a hard language to grok, even the data type names are unclear (data frame, vs matrix, vs vectors..). I'll take a look at the tidyverse packages.

https://www.tidyverse.org/learn/

any additional learning resources you'd recommend?

The “R for Data Science” book by Hadley Wickham (creator of tidyverse, and I believe he is chief data scientist at R Studio) is hands down one of the best introductions to data exploration and analysis.

As the other commenter said, probably the best resource out there is "R for Data Science" by Hadley Wickham, the architect of the tidyverse. It will get you up and running with the most important parts of the tidyverse (dplyr for data manipulation, ggplot2 for data visualization, tidyr for general data cleaning utilities, etc). Its available online for free here: https://r4ds.had.co.nz/

To add onto others’ answers, Advanced R from the same author dives more into the inner wirkings of R (such as lists X vectors, what are classes etc). It helped me understand why my code works the way it does.

Reply on news.ycombinator.com