However, Python is better for nearly everything else in the field (namely, working with nontabular data, external APIs, deep learning, and productionization).
It's about knowing which tool to use.
Basically if you were to go down any unbeaten path when it comes to statistical models, you're better off using R. But if your main goal is pushing to something prod, then you're better off with Python. The only exception being Shiny, they've put a lot of effort into making it production-ready.
What does R offer?
In Python there's SARIMAX and Prophet, interested in what R has to offer.
Also interested in a decent Grid Search for time series.
Just checked Prophet source code and it looks like there are actually 2 implementations in the same repo: one for R and one for Python. I thought it would have been the usual C/C++ implementation with 2 bindings. I wonder why they chose to develop it this way.
However, because the rest is easier in python, and my mental gears grind when I switch from one to the other, I end up using Python for the adhoc EDA and viz, and with Spyder, it is a pretty decent experience.
I agree with all of that except for productionisation. I would have agreed before dealing with issues around getting consistent versions of python + libraries to run.
The issues I see with Python are as follows:
- pip doesn't actually check to make sure your dependencies are compatible, which causes real problems with numpy et al
- conda isn't available by default, and running it on remote boxes is non-trivial (I spent a whole week figuring out how to get it running in a remote non-login shell).
- This makes it really, really difficult to actually get a standard set of libraries to depend upon, which is really important for production.
R, on the other hand, actually resolves dependencies in its package manager, and the R CMD BUILD for packages, while super annoying helps you produce (more) portable code (did you know that conda doesn't provide cross-platform yml files unless invoked specifically?).
In terms of handing it over to engineering/non data science people though, Python is much much much better.
tl;dr Python's an ace language with a terrible production story.
And you have packages like renv which also help isolate specific versions of packages to make portable environments even more reliable.
I strongly prefer R for data science, but its dependency management story is poor, even compared to Python’s (which, in turn, is poor compared to Rust/Ruby/…).
Python will silently upgrade numpy as a transitive dependency and break everything, which is much worse. MRAN also has daily snapshots which is normally how i handle stuff that will never be updated.
I also specified building an R package which does handle dependencies versus the python equivalent which does not.
I'm not saying R is good, I'm just saying Python is way worse.
Compared to Rust? Sure. Compared to Ruby? Maybe in the way that a lockfile isn't automatically generated when using pip.
Hating on Python's dependency management is a meme at this point. You could do a lot worse than the current pip + venv, and upgrading to something like poetry or pipenv is pretty painless. I'm pretty sure 99% of problems occur because people don't pin stuff.
Model building is done in AWS with access to Anaconda.
Usually we have an environment.yml for the REST API and one for model building.
This makes modeling -> deployment cycle fairly easy, if not perfect.
You can also use pip and env, but you have to make sure that all important dependencies are specified sufficiently specific. But that's also the case for Anaconda. (For instance, we had a problem in the API with a x.x.y release of greenlet or gevent since we only specified x.x)
For R, well use packrat. R IMHO has the problem of many different algorithms with different APIs. Yes, there are tools like caret, but 'you' will run into problems with the underlying implementations eventually. sklearn makes things easier here, at least most of the time.
I would also prefer R for EDA. But I don't like splitting eda and modeling that way, since there can be subtle differences in how data is read which can lead to hard to find problems later on. (Yes, you could use something like feather)
I also thing that tooling for python is much nicer, pytest, black, VSCode python integration just seem more mature.
I think all of those things you mentioned are JVM stuff, right? There's a version of R called renjin that could be used in that scenario.
Don't get me wrong, I'd love if this was better in python but right now it is far more difficult than it needs to be.
Maybe we mean different things by "productionising ML applications" but building a docker container with an R runtime and the correct package versions is not all, or even half, of what's required for production.
Set up a separate API gateway, which covers all your points (REST endpoints, monitoring, security) - there's plenty of off-the-shelf options. Route authenticated requests to the backend that runs your model.
Logging is pretty available in both (though better in Python to be fair).
I don't really see how building my model in Python would make it easier to add this API functionality either, so it's a bit irrelevant. Like my docker container (which appears to be almost essential in Python but nice in R) can call predict in any language, and then pass through to the API using the tools noted above.