Hacker News Reader

No, but I frequently see people implying that you can do your data science in Python and R as long as you can fit the data in RAM. As you mention, it's not RAM that's the limiting factor for larger data volumes, it's finding tools that exploit parallelism.

View on HN

I have done plenty of parallel work in python and R, so I'm still not sure what you mean.

Yes, that's my point. It's too simplistic to say "well, the data fits in RAM", you have to add parallelism to make the workload tolerable. In the past, some people have done that using MapReduce or Spark, GNU parallel or just writing parallel code in their favorite language. But RAM by itself isn't the only limiting factor to whether a problem is solvable in a reasonable amount of time.

If you are using python and R, can’t you make your own parallelism?

Reply on news.ycombinator.com