Hacker News Reader

Comment by goatlover | Hacker News Reader

I know R is great for that, but are Python stats libraries lacking to the point of needing 10-100x more code? My understanding is that Python has caught up with R in the data science field in terms of popularity and usage.

You can also use R inside Python using the rpy2 library. And Julia embeds R with the RCall package.

View on HN

It really hasn't. Python has pretty good support for machine learning on data with no missing values.

It doesn't really support things like survival analysis, complicated time series models or (the most important) missing data.

Also, R's formula interface for models is a thing of beauty, while Python has nothing similar.

So, if you have data with no missings and are only interested in a small subset of statistics (that used by computer scientists), then Python can definitely suffice.

If you are in almost any other discipline, you're going to be using R.

Don't get me wrong, Python is a wonderful language, but R is so well adapted to analysis that it will be a very long time before it gets replaced (Julia looks really promising, but its not at 1.0 yet, and hence hasn't seen the broad scale adoption it would need).

So why has nobody tried to implement something similar for Python? Is there some fundamental limitation of the language at play?

Academic statisticians know and use R. They publish research in R. Python doesn't have the same mindshare, so somebody would have to specifically go out and re-implement it in Python. So far, it seems like nobody has been sufficiently motivated to do so.

You'd basically be implementing a new interpreter (and presumably trying to do so in python). And hauling in a bunch of Fortran bindings along for the ride.

It's not just that. R has a few things going for it:

1. It's a functional language, and functional programming style lends itself really well to matrix operations (and the engine of stats is all about matrices).

2. Matrices are baked really deeply into the language, which makes the functional style even more powerful. (my understanding is that what makes the formula interface work so well).

3. R can be viewed as coating of syntactic sugar on a lot of very useful fortran libraries.

On the other hand I once saw some open source toolkit for natural language processing in R. I had two issues with that. Firstly WTF was the author thinking doing all his general purpose programming in R. And two I didn't want to investigate further because as NLP is largely about general purpose operations and I didn't want to get sucked into that rabbit hole.

One thing I don't like about Python libraries, from the top of my head: That there is a distinction between functions and methods, e.g. between nansum(a) and a.sum() where a is a numpy array. The distinction is illogical and just has to be memorized, much like the difference between different categories of verbs in French.

In contrast, R uses an exotic form of object oriented programming called multimethods, which doesn't make a distinction between object.method() and method(object). And if you like chaining methods together like object.method1().method2().method3(), then you can do that in R using the piping operator from the Dplyr package: object %>% method1() %>% method2() %>% method3(), which is rightly equivalent to method3(method2(method1(object)))

The reason there is a distinction in Python between functions and methods is because Python supports multiple paradigms. It doesn't force the programmer into using OOP. Sometimes you want a function that doesn't belong to some kind of data type, and sometimes you want to create a new type of data and define methods that work only for it.

You make it sound like perl (only perl is more flexible - both its major strength and weakness)

Things (IMO) that R has going for it:

* RStudio IDE - none of python equivalents are anywhere near as mature

* Functional programming - I can't fully articulate why yet, but to me, FP seems like it "fits" most DS programming challenges better than OO.

* Tidyverse - I'm biased (being the author) but having a broad ecosystem of tools with shared underlying programming philosophies makes solving data science challenges particularly easy/painless/fun.

* RMarkdown - makes it so easy to intermingle prose + code and then (via pandoc) render to a very wide variety of outputs (html, word, pdf, html slides, dashboards, ...)

* Shiny - for cases where you don't want a single report but an app, an R user can quickly create an interactive app without knowing the details of html/css/js

* Very broad ecosystem of statistical packages. Most statistics researchers use R, so you're more like to see cutting edge stat research in R first.

Reply on news.ycombinator.com