Hacker News Reader

Comment by wodenokoto | Hacker News Reader

Does anyone know how dataframe in R compares on these 10/11 points?

I'd say native data frames in R aren't great at these (maybe #4, maybe #9). I'm excited to see how Arrow can perform, and hopefully we'll see solid bindings to R as well.

The data.table package (https://github.com/Rdatatable/data.table/wiki) does make progress on some of these - I'd say #1, #3, maybe #7, #8. Dplyr has a query planner too, fwiw.

I'm not an R-guru, but I did my thesis with 5k lines of R, so I'd call myself proficent:

Point 10: R has lazy evaluation, which means here that a function will not be evaluated when you define it, it will be evaluated when you call it (maybe not quite the same as some other language's lazy eval). I'm not aware of any built in feature for query planning, if you ask for nrow(some_func(myframe)), it will evaluate the some_func(myframe) function and then count up the rows. You could always write your own query planning function I suppose.

Point 11: R has several multicore/cluster libraries, and some are actually decent. If you are like most R users, you use StackOverflow a lot, and you'll end up with one algorithm that uses the snow package and another algorithm that uses multicore, one that uses parallel, and so on. A few very well written packages have hooks that make going to multiple cores easy, but most do not and you typically have to roll your own.

Hi, Wes here. R data frames have limited to no query planning and are not multithreaded in general, so Problems 10 and 11 are problems in R also.

And R’s syntax leaves much to be desired: it’s ugly compared to python’s.

To each their own - I find the grammar of dplyr and the tidyverse is quite elegant.

Agreed! Sadly it is difficult to fully emulate in Python without some form of delayed evaluation in the language.

R is one of the few language that popularize data frame.

Out of SAS, R, and Python or even C++ and Java. Data frame are native.

So personally, the syntax especially for data frame is beautiful compares to python.

Python don't even have missing value built in or subsetting data frame.

Native data.frames don't but data.table does to some extent. It's parallelization is still relatively new/nascent and requires OpenMP which doesn't have good support on MacOS but it is present.

> R has lazy evaluation, which means here that a function will not be evaluated when you define it, it will be evaluated when you call it.

It seems unlikely you meant this as stated. How is it possible to "evaluate a function" when you define it? Certainly you need to give it arguments, and that can only happen when you call it.

A clearer explanation is the lazy evaluation here: http://adv-r.had.co.nz/Functions.html#function-arguments

I can pass an argument but R won’t try to evaluate unless it needs it. This can be beneficial when only some of a function’s branches need the argument. You can pass a solver for the traveling salesman problem but R won’t waste CPU cycles until it reaches a point where it has to solve the TSP to get an answer. Maybe the first branch of the function is a feasibility check, and the TSP will be skipped for something else.

This is a little harder to explain for data frames, but you can create R functions that act a little like generators in python. This can help with memory management where instead of a gigantic matrix you have a function that generates the part of the matrix that you need.

Wow, if I'm understanding that correctly R's function arguments can have default values that depend on the values variables within the function have when the argument is first used. That seems insane!

It's odd but it enables a lot of useful things (e.g. magrittr's pipe operators). It's possible to write functions that change their behaviour depending on what name they were called by too.

"That seems insane!"

Welcome to R!

It's a sort of lisp-y idea. Arguments passed to functions get "quoted", so the function can get or change data about the expression and its scope/environment before evaluating it. Like others have mentioned, it's what make R quite good for developing DSLs like R's formula language or dplyr. (And other conveniences, like auto-labeled plots, etc.) But similar to lisp macros, it can make for unpleasant surprises if not used wisely.

If you look at attempts to do this stuff in python---e.g. patsy, which emulates R's formula DSL, and there's another project that emulates dplyr I don't recall the name of---you see they have to resort to parsing and eval'ing strings instead of working on expressions (language objects that represent ASTs), which is not nearly as nice or safe.

Edit: But just to emphasize your surprise -- yes, you can definitely be surprised by delayed evaluation in many contexts if you're used to more traditional languages.

    > y <- 10
    > wat <- function(x=10*y) { y = -y; x }
    > wat()
   [1] -1000

But (1) good library writers don't play these kinds of tricks, so it doesn't come up too often in practice; and (2) when writing/debugging my own code, I've not found it too hard to reason about, anticipate, and avoid these effects.

    > y <- 10
    > less_wat <- function(x=10*y) { force(x); y = -y; x }
    > less_wat()
   [1] 1000

What's even crazier is how (according to this snippet) R thinks that 10*10 is 1000 :)

R data frames have excellent support for missing and categorical data. Everything else is just as much of a problem.

Disagree. Factors (categorical data) in R only support strings

But they can be ordered and are represented as integers. So you can turn your numerics to factors and keep them in order and do whatever goofy math you want to do on your categorical data.

Right, but I don't think it's fair to say that R's support for categorical data is "excellent" if only strings can be category labels/levels. Categories (aka dictionary-encoded data in other systems) semantically may be any type in practice (strings, timestamps, numbers, etc.). The function as.factor in R is lossy because the input type is coerced to string.

> the input type is coerced to string.

Wow. Didn't knew that.. strings is the one thing I always minimize in my datasets, due to speed and memory considerations..

BTW, I wanted to thank you for your 2012 slides on how you used hashes to group and join data. It led me to learn more about categoricals and I ended up implementing a Factor() object [1] in the other tool I use (Stata) that ended up being a life saver. In fact, once you have a powerful and fast categorical type, with a set of key functions, you can do anything from group the data, to count distinct categories, to run fixed effect regressions in no time.

[1] http://fmwww.bc.edu/repec/scon2017/Baltimore17_Correia.pdf

Memory isn't the issue. Both factors and string vectors only store each unique string once in memory. The issue is the limitation that factor levels can only be strings and not any other data type. Also, I think testing strings for equality is O(1) since it should simplify to a pointer comparison.

That's not right. Factor _levels_ can only be integers, factor _labels_ can only be strings since labels are the printable representation of a level. Maybe you could store the "real" values of a level as attribute (I.e. metadata) of the factor. Anyway, I think that's as solvable problem.

Well, I don't know if there's a standard terminology used elsewhere, but what you call labels, R calls levels.

Erm ... Well, I admit it's somewhat confusing.

    factor(x = character(), levels, labels = levels, exclude = NA, ordered = is.ordered(x), nmax = NA)

    [...]

    levels
    :   an optional vector of the values (as character strings) that x might 
        have taken. The default is the unique set of values taken by 
        as.character(x), sorted into increasing order of x. Note that this set 
        can be specified as smaller than sort(unique(x)).

    labels
    :   either an optional character vector of (unique) labels for the 
        levels (in the same order as levels after removing those in exclude), or 
        a character string of length 1.

This way, you can do something like that:

    > x <- 1:3
    > factor(x, levels = 1:2, labels = c("foo", "bar"))
    [1] foo  bar  <NA>
    Levels: foo bar

But this actually is:

    > factor(as.character(x), levels = c("1", "2"), labels = c("foo", "bar"))
    [1] foo  bar  <NA>
    Levels: foo bar

I'm trying to think of an example where you would want to use a factor with levels that are timestamps rather than just a vector of timestamps. And likewise for other data types, even strings. The only case I can think of would be if you wanted to limit the possible values to specific set.

My impression is that factors in R are borderline-deprecated, especially in the tidyverse, in favor of just using the equivalent non-factor vector.

I think you're mixing up scales. Timestamps and numbers can be recoded to categories but when doing this you're throwing away information -- the same way you lose information when converting an integer to a string. When you work with factors you're no longer interested in qualities the data would have on its original scale like distance etc.

Could you please provide a real world example where this actually is a problem.

Reply on news.ycombinator.com