Scaling PageRank with R on Rescale

blog.rescale.com

41 points·gpoort··5 comments

These days a cheap way to get attention seems to be to Hadoop'ize a well known but expensive operation. "Bogo-sort too slow ? no worries, we will run it on the cloud." These approaches are buzzword compatible. They keep the marketing guys happy. Managers do not get fired for embracing the cloud/Hadoop, even if it is done in a grossly inefficient way.

I realize I bring unwelcome rain, but the post really begs the question if using R here is the best use of resources: technological and human

Hey srean,

This is Amey from Rescale here. I definitely agree with your larger point about R not being the most efficient tool to use in this case. But if you read the post closely, that isn't the point we are trying to make.

R its other array-based cousins provide a much more natural and expressive framework for linear algebra/matrix operations. I'd say it is still worthwhile to try and scale your existing R code to moderately large datasets. Come to think of it, most "big data" isn't really that big.

If you want to take a look at scaling page rank on commodity clusters with R, you should take a look at the newly released SparkR (http://amplab-extras.github.io/SparkR-pkg/) and possibly just call into GraphX (https://github.com/amplab/graphx).

Thanks for the heads up etrain!

I wonder why they call the algorithm PageRank when their example is academic citations? The academic citation index existed long before PageRank took that concept to the Web.

People even gamed it the same way (citation clubs == linkbuilding networks.)

Reply on news.ycombinator.com