Hacker News Reader

Comment by tdb7893 | Hacker News Reader

I was checking if there was a response and upon reading your comments again, looks like your method is less a model and more just some ad hoc procedure. Based on what your conclusion is it sounds like a linear model based on demographic data would work and linear models are easy to train. Though by your summary there are probably going to be issues with the model and you have the same issues with your procedure, you just can't tell because it hasn't been rigorously specified.

There's a channel on YouTube called Statquest that teaches statistics in a pretty accessible way if you're interested in analyzing this sort of data.

View on HN

I learned my statistics in university, not Youtube. You are confusing yourself about residuals.

To make my point in the original post, I held the group scores fixed and extrapolated combined scores based on changing population sizes. This is straightforward. Simple even.

\(\widehat S=\sum_g p_gS_g,\)

Why did I call this "linear"? (Not a "linear model", your term, because it is not). The model function actually tracks the population function. Over the 25-year period, however, this trend is in fact linear because the White-to-Hispanic population trend is linear on that timescale. Over a longer timescale it would not be.

What you're describing is a linear model, it just has an ad hoc construction instead of something like a least squares training (https://en.wikipedia.org/wiki/Linear_model). Also linear models are about linear associations and work even with non-linear predictors. Your assumption in the model is explicitly a linear association so how the predictors vary shouldn't affect it.

I don't mean to disparage amateur statistics since often there are interesting things that experts miss that amateurs have insight into but this sort of data with few data points (the data you have doesn't sound independent) and a lot of confounding factors is not trivial to accurately model. Not that everyone needs to learn statistics but this sort of data (and also this topic) probably deserves a bit more of a rigorous approach.

As a very simple example, picking a time range for the model because the model doesn't work when you go farther back adds a lot of potential bias into the model and that along with the non-independence issue (and without looking at the data myself I don't know if there are other issues) are going to lead to overconfidence in conclusions.

You remain confused.

Reply on news.ycombinator.com