1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town
And not sure how anyone can argue the dataset is worthy of being included. It is pretty offensive and misguided at minimum to argue that having more black people in your neighbourhood will depress housing prices. And for it to be solely because they are black and not to do with a range of other factors e.g. socio-economic.
It is pretty offensive and misguided at minimum to argue that having more black people in your neighbourhood will depress housing prices.
It's the purging of a dataset, a toy dataset in this context, for a reason of political correctness, that I don't support. If you look hard enough at anything, you can probably find a way to call it racist or some similar slur. If we start applying this lens to tools like scikit learn, we go down a path I don't agree with, that's completely performative in terms of actually addressing any wrongs, and is a continuing distraction from what could be useful work. Debating if and how racist this is is immaterial imo to whether or not we should erase everything doesn't align with modern hypersensitivity about political correctness
So even if you want to ignore the fact that the data was outright used to discriminate in the past, the data itself is actually flawed in several ways...
Why are people singling out this one dataset?
Which is such an insane premise that it's hard to take the rest of your point seriously.
The data is the data. The data isn't suggesting that "having more black people in your neighbourhood will depress housing prices." That's your take on what a racist causal interpretation would look like.
The correlation is very real and turning a blind eye to it is worse: https://www.brookings.edu/testimonies/how-racial-disparities...
> At low to moderate levels of B, an increase in B should have a negative influence on housing value if Blacks are regarded as undesirable neighbors by Whites.
Isn't that true? It's almost a tautology.
(Bad things can be true.)
For example ignores the fact that on the whole black people are significantly poorer.
In Boston the average white family has a net worth of $200k+ while black families in the same city have a net worth of <$10 (that's not a typo, it's less than ten dollars). Poorer people by nature can not afford houses in more expensive neighborhoods, so naturally you have a concentration of black people in poorer neighborhoods... it's not because white people find black neighbors undesirable, it's that black people are disproportionately poorer.
This kind of oversimplification tends to perpetuate a lot of negative stereotypes about black people while hand-waiving away the chronic issues black people are faced with that creates this kind of disparity.
> At low to moderate levels of B, an increase in B should have a negative influence on housing value if Blacks are regarded as undesirable neighbors by Whites.
Either way, it's a dramatic oversimplification. Black people are also significantly poorer than white people, so it's also likely that black people can only afford to move into a neighborhood as property values decrease... yet the people who collated this data chose to make a different assumption.
The more it correlates with a lower price, the more race issues.
It's seems like a weird choice. If there are 62% or 64% then the feature will yield the same value.
I think it would make more sense just to include percentage of households where at least one member of the household has a certain ethnicity.
I don't think it's a offensive to analyze demographic information in the aggregate. In fact this happens all the time.
b) It is not scikit-learn's responsibility to alter third party datasets.
IIUC, you're arguing that "whole-town" level aggregation is misleading. So if we get more granular, we could do it my neighborhood, street, building, apartment/car/shelter, bedroom, bed, bed @ time of day, etc.
Any one of those aggregation levels could hide interesting distinctions that could be made if only the data were reported with even more granularity.
So are you arguing against aggregation in general? Or just whole-town aggregation specifically?