Hacker News Reader

Comment by minimaxir | Hacker News Reader

I hadn't used data.table or plyr because the native R functions were giving me good performance even at tens of thousands of rows.

But now that I'm doing analysis on hundreds of thousands of rows, doing aggregation takes awhile. This article convinced me to give those packages a try. If data.table and plyr aggregate functions are indeed paralellizeable, that's a big deal, especially when implementing bootstrap resampling.

View on HN

You should know that dplyr, plyr's replacement is already fairly stable and worth using. For basic tasks, it's as fast or faster than data.table in my experience, with the caveat that it is more likely to copy for some methods than data.table which is very strict about this.

And being able to combine it with an RDBMS means that you can potentially do plyr-ish things on datasets that can't fit in memory. I think the SQL generated by dplyr tries to be smart / efficient too.

Wow, if the benchmarks are true, then dplyr sounds perfect.

http://blog.rstudio.org/2014/01/17/introducing-dplyr/

I use data.table and like it. It's fast (and I think memory efficient). plyr is IMO so slow to be unusable on just hundreds of thousands rows. dplyr, as mentioned, is the replacement that fixes it, but it was literally first released a couple months ago this year. I have no doubt Hadley will produce something brilliant, but I'm guessing it will undergo a lot of change to get there.

Reply on news.ycombinator.com