(Who came up with R squared anyway? Seems like someone wanted to remove the possible minus sign from R but didn't know about the ABS function, so he used the square which has this side effect of making the numbers too small.)
c) The R^2 used with a linear model requires a constant term, in this case the constant term or bias explains a lot about preferences (almost 50/50) so there is less information available for the slope term.
Hope this helps.
This explains the paradox, basically. When you take the null model "preference = 50%" (i.e. intercept-only model), there simply isn't much residual variance left for the linear model to explain.
That's why you get an R^2 = 1 if you use the "R^2 = rho(state, preference)^2" formula (you are ignoring the role of the intercept in explaining most of the variance, and exploiting the translation-invariance of the Pearson correlation) vs. you getting an R^2 = 0.01 when you use the (more correct) "R^2 = explained variance / total variance" formula.
TL;DR: It makes sense to get a very low R^2 when it is the intercept and not the predictor that is explaining most of the variance.
I'd say that there is still quite a lot of residual variance to explain. You need a baseline - the worst choice would be to predict 0 (or 1) and the mean squared error would be 0.5. Using 0.5 as baseline halves the mean squared error to 0.25.
> data <- data.frame(state = c(0, 1), pref = c(0.45, 0.55))
> sum(residuals(lm(pref ~ 0, data = data))^2) # null model
[1] 0.505
> sum(residuals(lm(pref ~ 1, data = data))^2) # intercept-only model
[1] 0.005
> sum(residuals(lm(pref ~ state + 0, data = data))^2) # predictor-only model
[1] 0.2025
So, it seems clear that you only get a "perfect" prediction with the full (intercept + predictor) model mostly because of the intercept (which explains (0.505-0.005)/0.505 = 0.99 = 99% of the variance).
Thus, it makes sense that the predictor is only explaining the rest (i.e. 1%) of the variance... hence, the R^2 = 0.01
Maybe I'm completely missing your point but the calculations in the blog post are, adapting your code (I think you meant mean where you wrote sum):
> data <- data.frame(state = rep(c(0, 1), each=20), pref = c(rep(0, 11), rep(1, 9), rep(0, 9), rep(1, 11)))
> mean(residuals(lm(pref ~ 0, data = data))^2) # null model [NOT IN THE BLOG POST]
[1] 0.5
> mean(residuals(lm(pref ~ 1, data = data))^2) # BASELINE intercept-only model
[1] 0.25
> mean(residuals(lm(pref ~ state + 0, data = data))^2) # predictor-only model [NOT IN THE BLOG POST]
[1] 0.34875
> mean(residuals(lm(pref ~ state, data = data))^2) # MODEL
[1] 0.2475
> summary(lm(pref ~ state, data = data))$r.squared # MODEL
0.01
Either way, the point stands... the improvement in using a full linear model (that predicts 0.45 or 0.55, depending on state) is marginal compared to the baseline model that always predicts 0.50, as you demonstrate with your code.
To me, this doesn't seem paradoxical... the predictor is indeed providing little information over the "let's flip a coin to predict someone's voting preference" null/baseline predictor, since people's preferences (in aggregate) are almost equivalent to "flipping a coin".
note: I meant "sum", but it's the same, since the ratio between sums of squares is equivalent to the ratio between mean squares
Yes, I think we don't disagree. I was just puzzled by the "little variance left to explain" remark.
> note: I meant "sum", but it's the same, since the ratio between sums of squares is equivalent to the ratio between mean squares
You're right, sum of squares made sense if it was just for the ratio.
It's not that there is "little variance left to explain", but actually that (no matter what) there will always be too much variance left to be explained, when the response is Bernoulli-distributed and the parameter is not too far from 0.5 (i.e., the data generating process is like flipping a slightly loaded coin).
If you use the expected value to predict the Bernoulli variable, you will always be somewhat wrong (0.45 and 0.55 are both far from 0 and from 1, which are the only possible responses).
If you use a binary response to predict, you will quite often be very wrong, even if you are right on average, and even if your prediction is to generate Bernoulli-distributed samples from the exact same distribution (i.e., you know exactly how the coin is loaded/biased and you can exactly replicate its data generation process).
So... yeah... no "paradox" ;)
But, anyway, seems like I interpreted things incorrectly.
The reason we do that is because we are assuming the errors are normally distributed and finding the slope that gets the best possible R2 is equivalent to getting figuring out how to fit the line with the maximum likelihood estimator of the error (aka mean of the distribution).
So ultimately it’s about curves. If you wanted to get a sense for why this is strongly desirable you should try to fit a linear regression using absolute error instead of squared error.
A perception engineer took one look at the controller and saw linear error terms for distance left + distance right (distance to walls), changed it to distance left^2 + distance right^2, and the whole thing magically worked beautifully. Exercise for the reader: What position in the hallway minimizes sum of distances squared, vs what position(s) in the hallway minimize sum of distances without square.
This is essentially the same problem you pose.
Edit: no, derp, just walked into the same problem lol. Maximize the min should work though
> if X,Y,Z are "generic" random variables such that ρ(X,Y)=a and ρ(Y,Z)=b, we should on average expect that ρ(X,Z)=ab.
https://www.lesswrong.com/posts/vfb5Seaaqzk5kzChb/when-is-co...