Hacker News Reader

Comment by jacques_chester | Hacker News Reader

Before we jump to theories, has anyone eliminated the null hypothesis?

That is: assuming a totally random distribution of individuals into professions, purely by chance, some professions will have curious ratios. The first thing a statistician aims to do is to test the probability that the observed effect is due to chance.

A second thing to test is whether the ratios are stable across time.

This comment represents 20% of your RDA for nitpicky HN comments.

View on HN

I am not a gifted statistician, so I just made a monte carlo simulation to test. This is not too scientific because in reality different jobs have different numbers of people, and a thousand other variables, but anyway, here is the code and results:

    from random import random, randint

    p_white = .81

    n_occupations = 100
    n_workforce = 150000000

    occupations = [[] for i in xrange(n_occupations)]

    for i in xrange(n_workforce):
        occupation = randint(0,n_occupations-1)
        race = "white" if random() < p_white else "other"
        occupations[occupation].append(race)

    percentages = map(lambda x:x.count("white")/float(len(x)), occupations)
    print sorted(percentages)

    [0.8093594044124566, 0.8093716670190719, 0.8093837105400035, 0.8093941116532061, 0.8094080377696309, 0.809459598174709, 0.8094606757892058, 0.8094949877814243, 0.8095051091912103, 0.8095619154480623, 0.8095855138023271, 0.8096009184415862, 0.8096130265928078, 0.8096471477224493, 0.8096569834673563, 0.8096622916149537, 0.8096635533608949, 0.8096712436008832, 0.8096812597976153, 0.8096856369286581, 0.8096963532163073, 0.8097067434989362, 0.809721568598737, 0.8097215751436514, 0.8097328871910747, 0.8097572581628716, 0.809766392402407, 0.8097706403995997, 0.8097853662894858, 0.8097907743218179, 0.8098200848588418, 0.8098433273181519, 0.8098759161986114, 0.8098761662131717, 0.8098826202953597, 0.8098902227694705, 0.8098932712388633, 0.8099027774814941, 0.8099132209630405, 0.8099239110036907, 0.8099288512271164, 0.809935064025657, 0.8099377895998511, 0.8099442967886306, 0.8099576910748759, 0.8099684818490649, 0.8099723061883299, 0.8099772955347885, 0.8099790775887737, 0.8099951683577414, 0.8099964813792994, 0.8100114768250255, 0.8100264485053597, 0.8100326219727435, 0.8100426685467571, 0.8100538851385188, 0.8100624027564277, 0.8100772456940655, 0.8100799987206602, 0.8100878460787697, 0.8101051239951382, 0.8101238836646114, 0.8101259241335592, 0.8101299639808958, 0.8101333990942324, 0.8101444590699302, 0.810176363997584, 0.8101767343321185, 0.8101835910494692, 0.810185024612142, 0.8101881998783053, 0.8102189975744003, 0.8102348016041733, 0.810247294219311, 0.8102501899417512, 0.8102612850637286, 0.8102728907675896, 0.8102876025226937, 0.8102915035431886, 0.8103243498099062, 0.8103428626649917, 0.8103475670573029, 0.8103618541773706, 0.8103623158464366, 0.8103870909678774, 0.8104014543756358, 0.8104238667513951, 0.8104306535452401, 0.8104537503309478, 0.81046437557304, 0.8104696732444567, 0.8104846773483934, 0.8104918990629446, 0.810527481205214, 0.8105324346430826, 0.8105520454920258, 0.8106940187094273, 0.8107141451872244, 0.8107156884282282, 0.8108683817976586]

As we can see by the results, the min and max are around a tenth of a percent off from the expected value. Even if the job distribution is skewed (certain occupations have a larger workforce), if N is sufficiently large, numbers such as +15% should be completely unrealistic.

Please correct me if I have made a grave error that I did not mention (I know that this is very very hand wavy, unscientific).

No, that fits with the remarks made by others.

I wasn't sure whether the null hypothesis is approached by looking at the likelihood of sample variation from the general population of individuals, or whether by calculating how many job titles in the population of job titles would have such outlying numbers.

The consensus seems to be that most of these are not by chance. Roll on the debate.

I think you are making a grave error - near as I can tell, each of your simulated occupations has about 1.5 million people in it. Of the few professions listed that I checked, the biggest (farmer) has only 758k people in it:

Veterinarian 61k, farmer 758k, mining machine operators 21k, Speech language pathologist 123k.

The outcome doesn't change much (you go from 81.1% to 81.4% max), I'm just being a stats geek.

http://www.bls.gov/ooh/Healthcare/Veterinarians.htm http://www.bls.gov/ooh/farming-fishing-and-forestry/agricult... http://www.bls.gov/ooh/About/Data-for-Occupations-Not-Covere... http://www.bls.gov/ooh/Healthcare/Speech-language-pathologis...

Geography probably plays a big role in some of these. I expect that most veterinarians and farmers are located in farming and ranching states for example, and Iowa/Idaho are not exactly known for racial diversity.

Yes, I was aware of this, but in 5 minutes of googling I couldn't find a good answer to the total number of possible professions, so I went with an even 100. As I said, very hand-wavy and not too scientific. But yes, even for a workforce of 150,000 the results don't get close to breaching into the 90's.

A profession with 10K workers should randomly expect to be about 80-82% white.

From here: http://en.wikipedia.org/wiki/Binomial_proportion_confidence_...

p=0.81 n=10000 and z = 2 gives a 99% confidence interval

And 90% white is firmly in the "basically impossible by random chance" category.

I guess that depends on a uniform distribution of the location of the jobs to the racial characteristics in said location.

For instance I would imagine there would be more vets in an area with more animals, perhaps a country or town border region. At the same time those areas might have a higher concentration of white people than the average (with other races preferring a more urban environment).

They need EMTs just about everywhere (92% white).

but they need EMTs who live nearby, just about everywhere - and it does not take much for neighbourhoods and states to clear into distinct patterns by house-purchase preferences (game theoretic economist whose name cannot remember won a prize for this) basically all that is needed is a prefernce for a particular neighbourhood attribute (walk to shops, less black people, more black people, close to hiking routes) and for that preference to be available at that persons clearing price.

so, a dribbling knuckle dragging KKK member would pay a lot to avoid living in Harlem (or rather would sacrifice opportunity by staying in Fuckwatsr Alabama.)

Take a nice black professional couple - happy to live in well integrated area of Manhattan - till along comes baby. at this point a slightly larger house nearer good schools is desirable - and preferably closer to grandma. Alternatively there is a nice area of Queens (?) that is predominately white but has good schools. Now all it takes is a preference or a feeling of being more comfortable near grandma and near other young mothers "just like me".

Since grandma will likely be living in an area where there was traditionally a larger percentage of black families the comfortable preference is near grandma. and who would argue with an expectant mother wanting to be comfortable.

At a certain point the cost of being near grandma will exceed the amount they can afford without realising they are making an explicit racist choice. which is I think my preferred definition - not emotional preference but when a conscious decision is made with race as a factor.

All this shakes out to mean that levels of preference few if any of us would consider racist, lead to segregated neighbourhoods. luckily other preferences intrude masking the effect.

So to help with the discussion, this implies that if everyone was perfectly color blind we would expect to see a 81% ratio. if not what is the level, the error bar, that we accept as emotional preference that does not cross the line ?

And how do we tell that difference if the starting population is not 81% in that area ? and if the school district in the area is really bad / good? (nb nationally funded schools systems I would expect see nationally less segregated societies than the USA for this reason)

anyway my 2 cents

Doesn't your segregation hypothesis actually argue in favor of an EMT distribution closer to 81%?

Also I think the fact that you had to put a question mark after "Queens (?)" suggests that you might be speaking from a position of shelter.

I simply ran out of parts of New York I can name and associate with good / bad neighbourhoods. hence the ? as a sop to anyone who knows queens enough to laugh out loud at the idea there are any nice parts.

My guess would be that a large fraction of EMTs are in fire departments. Fire departments are notorious for being full of racists. They were pretty segregated until the mid 20th century, too.

As the system was taken over by the government, the pay got good, and it's one of the best jobs someone without college can get. So you can bet that the incumbents would use any weapon, including racism, to discourage competition for these jobs.

For some jobs listed, one could make the argument. For some others, like say - mining machine operator- you'd need a beefier formula that accounts for the fact of where the jobs are located. In many(but not all) areas, the most diverse are the bigger cities. Not going to find many mines in Chicago or New York.

That said, the veterinarian one surprised me a bit. I'd be fascinated to read a deeper dive into it. My suspected guess is that in rural areas, many veterinarians make 'house calls', and every resident has a -lot- of livestock, vs inner cities where one veterinarian can service a big chunk of the population. But, again, just a guess.

No. What you meant to say was that a random selection from a population of 10K white and black balls in the ratio 80% white balls & 20% black balls should be expected to be about 80% white.

People aren't black or white balls. Other factors (e.g., education, IQ, health, social class, and racism, etc.) are what make the difference. To imply that the difference is attributable purely to racism is naive and wrong-headed.

Isn't the difference between random chance and the complex realities of race and socioeconomic the entire point?

I don't think a definition of racism as historically and culturally naive as to be blind selection bias that exists only in the current moment is worthy of any serious consideration - only useful as a straw man for arguments such as yours here.

Any serious consideration of racism involves precisely the factors you mention. Education, nutrition, IQ, are all proxies for social class and I think you have to be quite biased to ignore how race relations throughout history have had direct impact on the current social status of race groups.

I think you must have misread the context here. The grandparent's hypothesis was that maybe the results seen here are merely the result of random chance (e.g. white and black balls being drawn from a bag). The parent was testing that hypothesis by checking the probability of getting those results purely by random chance. The answer was no, but no comment was being made about the source of that non-randomness. So yes, attributing all the differences to racism would be naive and wrongheaded (for most definitions of racism), but who's doing that?

> The grandparent's hypothesis was that maybe the results seen here are merely the result of random chance

Not necessarily my hypothesis per se. Rather, it's the default hypothesis that you're supposed to disprove before you go on to propose other hypotheses.

My problem is that I remember the existence and requirement to test the null hypothesis, but not the mechanics of how to do that. Luckily we have some people who are across basic statistics and so the conversation is able to move past "it's probably just chance" to speculate about whys.

Right, I didn't mean to imply that you believed the hypothesis, just that you were proposing it. The distinction doesn't matter much for my explanation anyway.

"I think you must have misread the context here. "

Yes, you are correct. My apology to all.

I think I'll try testing this statistically, even just as a cute example for Statistics Done Wrong. Thanks for the idea.

(If I get interesting results I'll post back here in a few days)

edit: I can't replicate the results in their charts using this BLS table:

http://www.bls.gov/cps/cpsaat11.htm

For example, for veterinarians I get 90.7% white, not 96.5%. Am I missing something?

The statistics done wrong bit is where they don't control for socioeconomic status.

Even ignoring those two chumps on the radio, the socioeconomic status of auto body repair (90.7% white) is not very high.

I don't think that's really true. That's generally something that requires at least tech school/community college. It's not manual labor.

For the individual occupations, presumably they all have enough participants to avoid a high chance of any one of them being that far off (for example, there are 100,000 veterinarians [1]) [Edit: and see sanskritabelt's comment below that seems to match my intuition.] Then the question is how many different occupations are being considered here, and it doesn't seem like it would be high enough to make numbers like 96% very likely. So if we were measuring the whole population, it seems--just eyeballing-- pretty unlikely.

But of course we're not measuring the whole population. The data from which this chart was actually derived might well be caused by random chance. The article's failure to cite a more specific source than the BLS generally is a bit annoying here. Edit: I guess the source is [2], but I'm still unclear on what was measured.

[1] https://www.avma.org/KB/Resources/Statistics/Pages/Market-re...

[2] http://www.bls.gov/cps/cpsaat11.htm

pg. 25 in this pdf (http://www.bls.gov/cps/cpsrs.pdf).

Found the source here: http://www.bls.gov/cps/cps_over.htm#methodology

Yeah I was thinking something like that, because as the article said 81% of all workers are white - simplified it means 8/10 are white - and these 33 jobs (out of how many?) have 9/10 ratio - does that really mean discrimination or is this story just a filler hit piece against perceived injustices that don't actually exist?

#1) This article was not a "hit piece", more a "isn't this curious?" piece

#2) There's already enough statistics out there that show injustice on a much larger scale - pretty much pick any statistic - life expectancy, wealth, employment, incarceration, etc and you'll see rank racial disparities -- in many cases racial disparities that haven't closed over the past 50 years.

Without knowing how many jobs there are total it doesn't sound fully like a "isn't this curious" piece. At best I'd put it with the "50 weird things about the 80's" type of piece. At worst it could detrimentally affect those professions in the name of progress.

Reply on news.ycombinator.com