Learning Python for Social Scientists (2015) | Hacker News Reader

Learning Python for Social Scientists (2015)

nealcaren.github.io

134 pointskome12 comments

One of the reasons R has been popular among social scientists is that you have libraries dedicated to statistical methods that are less common in Python. Examples include Mixed Effects Models, Hierarchical Linear Models, Psychometrics/Measurement Theory, Structural Equation Modeling, Survival Analysis, and a wide range of other techniques that shine when you have small amounts of data and lots of practical/physical barriers around data collection and experimentation.

Even for methods such as linear and generalized linear regression, there's just a range of libraries and diagnostic capabilities in R that aren't present in Python libraries.

Granted, a lot of these statistical methods can be expressed in Stan or some other probabilistic programming engines that have interfaces in Python. However, when you are a social scientists and not a statistician there's a big difference between having a few functions you can call and expressing a likelihood function.

In social science research, "getting the model right" is often the goal (i.e explaining why something happens), and "making good predictions" is of secondary importance (i.e. good predictions only matter insofar as they indicate that your model is good). That's one reason you end up doing involved probability modeling, compared to just slapping it with machine learning like you'd do in an industry setting. The flipside is that, as you said, many social scientists don't have the programming chops to do it themselves.

The other big advantage IMO is that it straddles the space between "traditional stats package" (SAS, Stata, SPSS, Minitab, etc.) and "programming language". We here know that it's more or less the latter, but the "batteries included" standard library with first-class support for a "data-frame" type makes it feel much like the former, especially if that's what you're already familiar with. It's a gradual transition from a stats package to general-purpose programming, and you don't have to jump through hoops, install third-party packages for basic stats work, or deal with ugly/counterintuitive syntax.

A problem with Python is the lack of a "best practices" style for scientific computing.

I find it amazing that Linear Regression in scikit-learn doesn't return standard errors/t-scores for the coefficients. (These can be useful for model selection, for example). I know about statsmodels and pysal, but the workflow with these is already slightly different.

>I find it amazing that Linear Regression in scikit-learn doesn't return standard errors/t-scores for the coefficients.

It's less surprising when you see how some people work with these types of models in Python. They minimize the prediction errors, and if another method has lower error than Linear Regression they use that. There's very limited thinking around diagnosing why a model is not working or modeling at all.

As nerdponx pointed out, this is completely opposite to how social scientists or statisticians do modeling work for research. The goal is to support or provide evidence against a hypothesis that has a theoretical basis. You cannot do that by just focusing on minimizing prediction error.

Linear Regression in R vs Python is great example of how simple functionality like print(lm) and plot(lm) exposes the different ways of thinking about how to approach a problem. R's lm() provides strong diagnostic functionality out of the box with a few commands, which support people who are focused on diagnosing a model and making adjustments to how the model is defined, not just doing basic transforms (e.g. log) and adjusting hyperparamters to minimize prediction error.

I find it amazing that Linear Regression in scikit-learn doesn't return standard errors/t-scores for the coefficients

Case in point! You don't typically need those for machine learning, so why bother with them in a machine learning toolkit/library?

You don't need them if you don't need to diagnose your analysis. But the hard part is of course thinking about why your model doesn't work, which is the strength of R.

I agree. Even if someone wants to swim against the current and use Python instead of R in these context, it is very hard to do and get publication quality results unless you have plenty of time and really know what you are doing. Since scientists are busy people, it makes sense to just follow the crowd and use R, which is a shame.

Why is it a shame to use R?

R is not inferior to Python. Perhaps for general programming, but for stats and visualization, R is the gold standard.

R's functionality for cleaning, reshaping, and otherwise manipulating data is also far ahead of all other languages I've tried, including Python. (As far as I'm aware, Python's current data manipulation functionality is pretty much in line with the 8-year-old R packages reshape2, which is officially retired, and plyr, which has mostly been supplanted by newer packages.)

> Social scientists don't really analyze images much, but that might be the next big thing.

Satellite images have been used in research in economics to estimate poverty and deforestation.

That all depends - there are some geography departments that do a lot of data driven analaysis, as do some economists. There really is a lot of overlap - with various disciplines doing data driven work. Would also expect that criminologists might be doing similar things.

Link to very first notebook by Julia Evans is dead.

Reply on news.ycombinator.com