Hacker News Reader

Comment by banachtarski | Hacker News Reader

3 packages:

plyr ggplot2 reshape

Learn those 3 and use them for everything you do in R.

plyr, while having awesome syntax, is really slow for anything beyond even the most modest dataset. Try using it with 50k rows, not to mention true "big data".

data.table is much faster. It's an extension to data.frames that adds some additional constrains/rules that allows for much faster operations including aggregating, subsetting, and merging data.

Cleaning data, however, does not have a steep learning curve or high difficulty level in R-- it has a steep learning curve and high difficulty level period. Implementing good procedures for data munging is 80% of the job.

Yeah - for the small data sets, Ruby or Python is often easier easier to use to whip a data set into a form that can be simply slurped into R as a dataframe.

With pandas / statsmodels / patsy, it's getting easier to stick with python for everything.

Don't forget Bokeh[1] - a replacement for ggplot2

[1] https://github.com/ContinuumIO/Bokeh

Has to be said I cheated a bit on the Coursera course, I used Ruby to clean up some of the data. Seemed a lot easier, and it's a lot more likely I'd use some kind of scripting language (be it Perl, Ruby, Grep, ...) to do a first clean up than head directly to R to do the job.

This is a good point. Usually, I use haskell or something to do large scale transformations prior to a plot or something. However, I still use plyr from time to time to do quick one offs on smaller samples.

With 50,000 rows plyr should be fine, and I've tested it up to 1,000,000 and it's reasonably. The problem is more that it doesn't scale terribly well in the number of groups.

I really like ggplot2. I always had a hard time getting on board with the other packages. sqldf allows SQL Queries against dataframes. Perhaps plyr and reshape feel familiar to folks from other backgrounds (science/stats) but SQL is the lingua franca among a good portion of the developer world (and explains innovations like LINQ in .NET).

Reply on news.ycombinator.com