I started learning it because I want to make an attempt to do some projects on Kaggle. Most people use Pandas, Seaborn, etc, which I will also use.
However, to me R appears like a little better Swiss Army Knife to do initial analysis. ggplot2, tidyverse, ...
Any help leveling up would be appreciated.
However, Python is better for nearly everything else in the field (namely, working with nontabular data, external APIs, deep learning, and productionization).
It's about knowing which tool to use.
However, because the rest is easier in python, and my mental gears grind when I switch from one to the other, I end up using Python for the adhoc EDA and viz, and with Spyder, it is a pretty decent experience.
I agree with all of that except for productionisation. I would have agreed before dealing with issues around getting consistent versions of python + libraries to run.
The issues I see with Python are as follows:
- pip doesn't actually check to make sure your dependencies are compatible, which causes real problems with numpy et al
- conda isn't available by default, and running it on remote boxes is non-trivial (I spent a whole week figuring out how to get it running in a remote non-login shell).
- This makes it really, really difficult to actually get a standard set of libraries to depend upon, which is really important for production.
R, on the other hand, actually resolves dependencies in its package manager, and the R CMD BUILD for packages, while super annoying helps you produce (more) portable code (did you know that conda doesn't provide cross-platform yml files unless invoked specifically?).
In terms of handing it over to engineering/non data science people though, Python is much much much better.
tl;dr Python's an ace language with a terrible production story.
And you have packages like renv which also help isolate specific versions of packages to make portable environments even more reliable.
I strongly prefer R for data science, but its dependency management story is poor, even compared to Python’s (which, in turn, is poor compared to Rust/Ruby/…).
Python will silently upgrade numpy as a transitive dependency and break everything, which is much worse. MRAN also has daily snapshots which is normally how i handle stuff that will never be updated.
I also specified building an R package which does handle dependencies versus the python equivalent which does not.
I'm not saying R is good, I'm just saying Python is way worse.
Compared to Rust? Sure. Compared to Ruby? Maybe in the way that a lockfile isn't automatically generated when using pip.
Hating on Python's dependency management is a meme at this point. You could do a lot worse than the current pip + venv, and upgrading to something like poetry or pipenv is pretty painless. I'm pretty sure 99% of problems occur because people don't pin stuff.
Model building is done in AWS with access to Anaconda.
Usually we have an environment.yml for the REST API and one for model building.
This makes modeling -> deployment cycle fairly easy, if not perfect.
You can also use pip and env, but you have to make sure that all important dependencies are specified sufficiently specific. But that's also the case for Anaconda. (For instance, we had a problem in the API with a x.x.y release of greenlet or gevent since we only specified x.x)
For R, well use packrat. R IMHO has the problem of many different algorithms with different APIs. Yes, there are tools like caret, but 'you' will run into problems with the underlying implementations eventually. sklearn makes things easier here, at least most of the time.
I would also prefer R for EDA. But I don't like splitting eda and modeling that way, since there can be subtle differences in how data is read which can lead to hard to find problems later on. (Yes, you could use something like feather)
I also thing that tooling for python is much nicer, pytest, black, VSCode python integration just seem more mature.
I think all of those things you mentioned are JVM stuff, right? There's a version of R called renjin that could be used in that scenario.
Don't get me wrong, I'd love if this was better in python but right now it is far more difficult than it needs to be.
Maybe we mean different things by "productionising ML applications" but building a docker container with an R runtime and the correct package versions is not all, or even half, of what's required for production.
Set up a separate API gateway, which covers all your points (REST endpoints, monitoring, security) - there's plenty of off-the-shelf options. Route authenticated requests to the backend that runs your model.
Logging is pretty available in both (though better in Python to be fair).
I don't really see how building my model in Python would make it easier to add this API functionality either, so it's a bit irrelevant. Like my docker container (which appears to be almost essential in Python but nice in R) can call predict in any language, and then pass through to the API using the tools noted above.
Basically if you were to go down any unbeaten path when it comes to statistical models, you're better off using R. But if your main goal is pushing to something prod, then you're better off with Python. The only exception being Shiny, they've put a lot of effort into making it production-ready.
What does R offer?
In Python there's SARIMAX and Prophet, interested in what R has to offer.
Also interested in a decent Grid Search for time series.
Just checked Prophet source code and it looks like there are actually 2 implementations in the same repo: one for R and one for Python. I thought it would have been the usual C/C++ implementation with 2 bindings. I wonder why they chose to develop it this way.
Millions and millions of users that have no idea what this blog post is technically about (but is interesting nonetheless)
It's true that they target different use cases overall (obviously with some overlap), but Excel tends to be used for lots of things that would be better handled with a different tool, because it's what people know.
R is far superior for interactive exploration/analysis and report writing. However Python is far superior if you are writing a program that does other things too.
My rule of thumb is that if a Python program is 70% or more Numpy/Pandas/Matplotlib etc then it should be R. Whereas an R program does comparatively little analysis and a lot of logic and integration, it should be Python. No one size fits all.
Calling R from Python: https://pypi.org/project/rpy2/
I'm not sure I'd ever mix the two directly myself; I'd compose my application as separate R and Python communicating somehow. It seems cleaner.
And sure enough it has something to tie in R too, truly connecting everything in a big nest: https://doc.sagemath.org/html/en/reference/interfaces/sage/i...
it works best if it's used semi-interactively, as a glue language between statistical packages which may be written in other languages. or to write simple "batch" scripts that basically just run a bunch of procedures in a row.
RStudio makes the whole experience much nicer in terms of plotting, and RMarkdown is great for preparing documents.
of course like shell scripting you can write fairly complicated programs in it, and sometimes people do, but due to backwards compatibility and weird design choices meant to make interactive use easier, programming "in the large" can get weird.
the analogy works for Python too -- it is definitely reasonable to use Python for shell scripting, but using Python interactively to pipe things from one program to another is slightly more frustrating than doing it in the shell, although might be preferred due to its other advantages.
But being so flexible makes it really expressive for doing ad-hoc analysis where you really don't know what you're looking for yet.
For data analysis, R is in my opinion better than Python. It's when you have to integrate it in existing workflows that Python quickly becomes a better choice.
Are you saying that you trust the code more because the domain knowledge make it more likely to get the right answer then? Has general knowledge increased such that scientists' code isn't as painful as it was 20 years ago?
For a while R was being pushed pretty heavily as a SAS alternative. My org paid R training courses etc. I found R and SAS pretty comparable at least the R packages we looked at (dpylr, ggplot2 etc).
I know about Python, the programming language I used PyGTK back in the day to build GUI apps. But it would not be my first thought for doing data analysis work. Does Python even offer something like R studio/ SAS Enterprise Guide and does it have a trending package?
Not that pandas/scipy/numpy don't make an admirable job. You can do something like this, but it's nowhere near as ergonomic as it is R. At the end of the day, R is fundamentally a language for data exploration, whereas with python those facilities are bolted on top of a general purpose environment.
Big mistake, btw. It took me years to unlearn all of the terrible habits I picked up from the R world. Do yourself a favor and start with python, if only to learn proper programming practices and techniques before diving into R.
One of course is allowed to learn more than one thing. Maybe play with a bondage and discipline language to expose yourself to the concepts the parent comment is advocating for.
As a student I constantly complained that we were being taught these useless languages. As a grownup I realize that while some of the Comp Sci faculty may’ve been out of touch, their goal was not teaching us commercially viable skills. They were endeavoring to teach us how to think. Once you know how to think you can express those thoughts in nearly any language, no matter how hostile to those thoughts it may be.
But maybe you just want to get things done, and if that’s so, the answer for data problems is basically one or more of R, Python, Julia, etc.
flights.iloc[0:10, flights.columns.get_indexer(['year', 'month', 'day'])])
flights %>% select("year", "month", "day") %>% head(10)
flights[['year', 'month', 'day']].head(10)
head(flights[c("year", "month", "day")], 10)
flights[1:10, c("year", "month", "day")]
flights[["year", "month", "day"]].head(10)for the data.table fans
Which is arguably the superior way to handle tabular data in 2020.
SELECT year, month, day
FROM flights
LIMIT 10Biologists like it for single cell analysis. They use Seurat and save the data as an object and load it up/ pass around around for analysis. Its actually kinda neat.
R's ggplot2 library is top tier in making graphs.
RStudio makes it very accessible.
Very popular. To the point of even having quite a lot of Microsoft support, lots of books, etc.
For reference, the authors of that book (the best book about ML in general) were all involved in the development of S and R.