Logistic regression from scratch
philippmuens.com
philippmuens.com
The idea behind binary regression ( Y is 0 or 1) is that you use a latent variable Y* = beta X + epsilon.
X is the matrix of indenpent variables, beta is the vector of coefficients and epsilon is an error term that sums the rest of what X can't explain.
Y thus becomes 1 if Y* >0 and 0 otherwise.
Seeing how Y is binary, we can model it using a Bernoulli distribution with a success probability P(Y=1) = P(Y* >0) = 1 - P(Y* <=0) = 1 - P(epsilon <= -beta X) = 1 - CDF_epsilon(-beta X)
Technically you can use any function that maps R to [0, 1] as a CDF. If its density is symmetrical then you can directly write the above probability as CDF(beta X). The two usual choices are either the normal CDF which gives the Probit model or the Logistic function (Sigmoid) which gives the Logit model. With the CDF known you can calculate the likelihood and use it to estimate the coefficients.
People prefer the Logit model because the coefficients of the model are interpretable in terms of log-odds and all and the fucntion has some nice numerical properties.
That's all there is to it really.
- this is a very basic thing
- that's all there is to it
because although it is a basic thing to you, it's not a basic thing to someone who hasn't spent the same time studying all the concepts beforehand.
This is generally why you have to take a class to grok stats rather than just read some reference material and definitions.
It's similar to programming environments when the senior dev says some thing that requires a lot of context is just really simple.
Monads are Simply Just monoids in the category of endofunctors, after all, they are really Simple, that's all there is to it! (what the subject hears: This is so simple to me, why aren't you smart enough to get this simple concept? You should know this already—I shouldn't even have to make this comment! Why haven't you been properly educated, like I have? )
also, mansplaining generally refers to assuming people (women, usually) have less pre-existing knowledge than they actually do.
(i.e. > 0.5 equals 1, and < 0.5 equals 0).
This seems to be a "rarely-questioned assumption".
Is there a reason why this is considered reasonable? And is there a name for the cut-off (i.e., if I were to want to change the cut-off, what keyword should I search for inside the software's manual?)?
In the software, you can get classification models to output class probabilities instead of class labels. You can then use whatever threshold you like for to transform those probabilities to labels.
You may see it refered to as "discrimination threshold". Varying that threshold is how ROC curves are constructed.
Almost all statistical models give you probabilities and it is up to the domain to determine the cutoff. You can clearly see this in logistic regression it's giving you a probability not a 1 or 0.
When a clinician gives me a dataset I build the model and leave it to them to do the cut off. It is not my domain or expertise to tell them where the cut off is. Plus it gives away the responsibility. I'm not responsible for it unless they don't know and are asking me for a reasonable cut off.
An example is people opting for artificial in semenation because of problem. I believe the cut off is above 60% because of the trade off.
You can read more about it in Regression Modeling Strategies by Dr. Frank Harrell.
1) agree with others comments that stating with 'very basic' ain't going to work
2) logistic regression actually isn't a common topic in an undergraduate statistics course. I cover it on the second to last day as an 'here is where you can extend it'
3) this isn't a very effective explanation.
4) a big background factor is the misconceptions about statistics that are ingrained in human beings. Learning statistical tests is hard, especially for many engineers and similar STEM types because it is typically the first (and sometimes only) class that deals with concepts of error and variance. Everything else presents an absolute theoretical model. On average, it breaks their brain...
log p(y=1 | x; beta) = beta * x - log Z(x; beta)
where
Z(x) = p(y=0 | x; beta) + p(y=1 | x; beta)
Thus, you can think of it as linear regression, but with an additional term log Z(x; beta) in the log likelihood.
Linear regression has a closed form solution of X projected onto Y: \hat{\beta} = (X'X)^{-1} X' Y
It is equivalent to the Maximum Likelihood Estimator (MLE) for linear regression. However, for logistic regression, MLE would estimate different from MLE for the log odds output.
Linear regression on {class_inclusion} = XB gives the linear probability model, which has limited utility. The required transform is covered by another commenter.
In that framework, they are literally the same model with different "settings" - Gaussian vs Bernoulli distribution.
Linear regression is extra-special because it's a special case of several different frameworks and model classes.
I should have written that it's better (in my opinion) to think of logistic regression in the context of GLMs, at least while you're learning.
Edit: yes logistic regression is a special case of regression with a different loss function. But it's not nearly "as special" as linear regression.
Joke aside, the truth is that logistic regression can be understood based on several assumptions.
Above we have the latent variable explanation. Then there is a Bayesian version. There's even a "random utlity" formulation, where one models explicitly choices of an agent with a probabilistic error. That one is good to explain hierarchical logit models and many of the "issues" with logit such as IAA.
GLM on the other hand I don't feel like it adds much except parameterizing the procedure, which ain't even a good thing. Nowadays we appreciate the semi parametric nature of regression a lot, which is why GLM has declined in use.
The point about the probability distribution is reasonable, but I am not sure if it is taken seriously by everyone applying GLM either. And again, if it is not necessary to assume such a distribution, then I would prefer a semi parametric approach, such as in linear regression.
Almost - logistic regression assumes that the function is linear in the log odds, i.e. log(p/(1-p)) = Xb + e. The problem is that you can't compute the log-odds, because you don't know p.
From this point of view linear regression would be using GLM with identity link function, logistic regression uses the logit function as the link function.
Its not just the underlying model, but the loss function is also different.
The claim is that “evidence” in Bayesian reasoning naturally acts upon the log-odds, mapping prior log-odds to posterior log-odds additively. To see this, calculate from the definition,
Odds(X | E) := Pr(X | E) / Pr(¬X | E)
= Pr(X ∧ E) / Pr(¬X ∧ E)
= (Pr(E | X) × Pr(X)) / (Pr(E | ¬X) × Pr(¬X))
= LR(E, X) × Odds(X),
So when we take the logarithm of both sides, we find that new evidence adds some quantity—the log of the likelihood ratio of the evidence—to our log of prior probability, in this phrasing of Bayes’ theorem.
I sometimes tell people this in a slightly strange language, I say that if we ran into aliens we might find out that they don't believe the things are absolutely true or false, but instead measure their truth or falsity in decibels.
So another perspective on what logistic regression is trying to do, is that it is trying to assume linear log-likelihood-ratio dependence based on the strength of some independent pieces of evidence. You can weakly justify this in all cases, using calculus and assuming everything has a small impact. You can further justify it strongly for any signal where twice as large of a measured regression variable ultimately implies twice as many independent events at a much lower level happening and independently providing their evidences for the regression outcome. So like, I come from a physics background, I am thinking in this case of photon counts in a photomultiplier tube or so: I know that at a lower level, each photon is contributing equally some small little bit of evidence for something, so when I count all the time up together, this is the appropriate framework to use.
A benefit of logistic regression is that the resulting model really fast. Furthermore, it's linear, so you can do incremental updates to your prediction. If you have `n` classes and `b` input features change, you can recompute in `bn` time, rather than doing a full matrix multiplication, which can be a huge time saver.
For comparison Leela Zero gets around 60% accuracy on predicting its own next move.
With this sort of accuracy you can reduce the search part of the algorithm to an effective branching factor of 2-4 rather than 40, nearly for free, which is a pretty big win.
If the prediction (policy) net had a 100% accuracy, you wouldn't need the tree search part at all.
Part one of Leela ranks several chess moves. Part two picks among those.
60% of the time part 2 chooses the #1 ranked move.
One addition: The second part can be run for an arbitrary amount of time, gradually improving the quality of the returned move.
The 60% figure comes from the training games, which are played very quickly, and so don't have a lot of time for refining, thus increasing the prediction accuracy.
In real games, tcec-chess.com/ this "self accuracy" would probably be a bit lower.
Often this is a good 'first cut' model for a binary classifier on tabular data. If feature interactions don't have a major impact on your target then this can actually be a tough benchmark to beat.
https://github.com/oli5679/WeightOfEvidenceDemo
https://www.listendata.com/2015/03/weight-of-evidence-woe-an...
Logistic regression is clearly a classifier. And you need data to train it. So it's a supervised learning algorithm.
More generally, I recommend Breiman's two cultures article for some insight into the similarities and differences.
If you need a really simple explanation, then machine learning is a tool for generating predictions, while statistics is a method for performing inference about causes.
Machine learning is typically defined as a way to enable computers to learn from data to accomplish tasks, without explicitly telling them how.
Both fields can use logistic regression, regularization, and gradient descent to accomplish their goals, so in that sense there's no distinction.
But IMO there is a difference in their primary intention: econometrics typically focuses on inference about relationships, machine learning typically focuses on predictive accuracy. That's not to say that econometrics doesn't consider predictive accuracy, or that machine learning doesn't consider inference, but it's usually not their primary concern.