Silicon Valley Is Using Trade Secrets to Hide Its Race Problem | Hacker News Reader

Silicon Valley Is Using Trade Secrets to Hide Its Race Problem

bloomberg.com

73 pointschollida1146 comments

For the millionth time. The distribution of populations not lining up with the general population distributions is not indicative of discrimination. Not to mention that the "anti-brown people" narrative absolutely dissolves in the face of the massive overrepresentation of Indians and other Asians. To say that Silicon Valley has a "race problem" and only include the employment statistics as they relate to the population at large is just pure insanity. What about the racial distribution of the hiring pool? It's less diverse by their own standards than the distributions in companies. What about the racial distribution of college graduates in engineering? Then what about mapping the racial distribution of the college graduates onto interview success rates, where the biased distribution of the graduates as produced by the universities' affirmative action policies must necessarily align with a similar policy at the company if their own filtering criteria is to produce the same results?

The thing that I find the most interesting about all of this is that there isn't an overt attempt to actually remove purported discrimination from the hiring process. Suggestions to anonymize the interview process typically produce results that are even less representative of their desired distribution, because almost every company is actively discriminating in favor of women and "marginalized communities." If their accusations were actually on the basis of correcting bias/discrimination, this is the sort of strategy you'd see advocated, but it's not. They want arbitrary quotas that are, given the available pool of candidates to even consider, mathematically impossible to attain.

The article cites data from the government on percentages of black, hispanic, and women in tech in Silicon Valley vs those same populations in tech nationwide. You would expect the percentages to be statistically equal.

You can also make any lie you want with statistics.

You can also use that argument to wave away any statistics you don't like the ramifications of.

Yes, but does this make my argument false?

It makes your argument specious, barring any supporting criticisms of the statistics in question.

It isn’t an argument, it’s a restated aphorism devoid of context. I think I’m not alone in wondering if you actually have an argument to make, one this involves making specific claims about the statistics I’m question and the methods by which you claim they’re lying.

That’s not even an argument, but I’d certainly be willing to see your analysis of these particular statistics and why you think they’re fraudulent.

No you probably wouldn't, as Silicon Valley attracts the top X% of the tech population due to the pay, the status of the companies hiring, VC money floating around, etc.

Are you saying the top X% is racially skewed? Because all things being equal, if the only difference between the two groups is color of their skin, I'd say that was a textbook definition of racial bias.

The question isn't whether there's racial bias anywhere in the pipeline from conception to employment, because there clearly is. It's whether there's racial bias in the SV companies themselves, or whether it happens exclusively in earlier stages.

He's explicitly rejecting the notion that the already skewed nationwide tech population is relevant. One would think, that if I was a large company recruiting nationwide, with brand name that's a resume magnet, I would be drawing from the skewed nationwide tech population, because that's what's entering the pipeline. If controlling for geographic factors, that's not what's coming out, something looks amiss. He explicitly is saying that "talent" is determining factor. That seems odd, since that would be conditionally independent.

All things are not equal. Black Americans as a group are poorer than other racial groups save perhaps native Americans, and have worse educational outcomes on average.

Is this the result of institutional racism? I’d say so, but it also can be a cause or underrepresentation in higher skilled employment, even if the employers are not biased in their selection processes.

Yes there are population biases in the hiring and funding data. No one denies that.

What he said was "the top x% of the tech population". The tech distribution as a whole is already skewed. If talent was independent of race, you'd expect an equally skewed distribution at whatever talent threshold you set. That's how conditional independence works. His statement is implying that is not true, and that if you set the talent threshold high, you get a different racial distribution. That's a very important distinction, if that is indeed what he's saying.

I really think this is what he's saying, because he's explicitly stating that large companies that recruit nationwide should be have a population equal to the skewed nationwide tech population, because they higher only "the top x% of the tech population". Not confounding geographic reasons (i.e. someone doesn't want to move away from their family), but specifically talent reasons.

This also plays a role. It would be important to note that the Bay Area also has a very different dynamic than let's say the south or the easy coast where there is also a much longer history of middle and upper middle class black families.

For example DC has a huge college educated black population because the government was one of the few places you could get a well paying as a black person with less discrimination. So lots of people decided to settle there. Atlanta has a lot of historically black colleges - creating a pool of educated workers.

In CA, most of the black population came over as blue collar workers in the military, shipyards, and manufacturing. Whereas most of the Asian population is made up of professional worker or affluent educated immigrants.

Yes I am generalizing here. We don't do a good job of carving out the differences for Asian groups. East Asians tend to be super educated, southeast asians tend to not be, and the outcomes are also widely different.

Ignoring the overt racism in this statement, it's just not true. I'll agree that a decent percentage of the top X percent are in the bay area due to pay, but nothing close to all of them. Saying so is an insult to developers not only nationwide, but worldwide...

"overt racism"? I didn't mention race at all in my statement, please calm down SJW. I never said that ALL the top talent is in SV, I said it attracts that kind of talent. Big difference.

If you want to discuss race, let's look at perhaps the best coding college in the United State, MIT. We can probably all agree MIT produces some top talent programmers, right? I know HN is typically anti-college, but I think we can agree on that. 30% Asian, 9% black, and 15% Hispanic. That's 5x higher Asian percentage than the general US population and slightly lower for both black and Hispanic. If that's the pool that companies in SV are pulling from, then of course their diversity numbers won't match the US population as a whole. Look at the colleges near SV too - Caltech (43% Asian), Stanford (22% Asian), Berkley (42%).

Dude, this statement -

No you probably wouldn't, as Silicon Valley attracts the top X% of the tech population

Could literally be summarized/rephrased as -

Silicon valley doesn't have a lot of black people, because it attracts top talent only.

I do presume you didn't mean it that way, which is why I didn't expand on it. But I want to point out that is the impression given.

One could argue that for a myriad of reasons, probably mostly money(again, for its own reasons), some minorities are underrepresented in the best tech schools and/or do not have the funds to up and move to the coast. That's an absolutely fair assessment. But I wouldn't assume that they are automatically not 'top talent' for those reasons. In my experience, the bay has a pretty low signal to noise ratio. Cheers.

Is the distribution of minorities static across the entire country? If you want to promote change you need to earn people's trust, skewing statistics to support your cause does not help that. I am on the side of removing discrimination, unfair wage practices, etc., but the diversity industry (speakers, journalists, special interest groups) often shoot themselves in the foot by making loud, bold, claims, based on gamed statistics...

> You would expect the percentages to be statistically equal.

Serious question: why? Why would you expect that? I’m seriously interested by a response to that, that would give way more weight to this affirmation.

No, you wouldn't, at least when it comes to the racial composition. SV (and surrounding area) has a very different demographic makeup than e.g. NY or Chicago or Houston. I'm not saying this explains all the differences between SV and nationwide, but it surely accounts for some.

PS: E.g. google tells me that the Santa Clara population is 2.5% black, while it's 12.6% nationwide (2010 census). The article now says nationwide the tech industry is 14.5% black; 14.5 / 12.6 = 1.15, 2.5 * 1.15 = 2.9; So adjusted for demographics I'd expect the SV number to be 2.9% black employees in tech after adjusting for demographics of the area, compared to the 2.6% the article states.

However, it might be that the low black population in that area is highly correlated to the hiring practices of the tech companies in that area, or it may not be.

Except when looking at the EEOC report, the numbers they are comparing are percentages of black, hispanic, and women working in tech in Silicon Valley vs. the percentages of those same groups working in all industries (not just tech) in the US.

In reality, the tech-tech participation comparison between SV and the US is closer than the article makes it seem. Look at the chart "INDUSTRY PARTICIPATION BY GENDER SEX AND RACE GROUP - HIGH TECH VS. ALL PRIVATE INDUSTRIES" in the EEOC report [1], and then compare to the numbers listed on the graphs for SV.

The proper corresponding stats for representation in tech between SV and US would be:

Black: 2.6% SV vs. 7.4% US

Hispanic: 6.4% SV vs. 8.0% US

Women: 32.3% SV vs. 35.7% US

So one could still point to the differences above for looking at the group disparities, but the article's statistics are comparing apples to oranges.

Not to mention location demographics and other factors attribute to differences between SV as a data point vs. the aggregate data for the US.

The pick-and-choose implementation of statistics is looking through a narrow lens.

[1] https://www.eeoc.gov/eeoc/statistics/reports/hightech/

The Bloomberg article is either intentionally misleading on where they pulled their stats from OR they used a different table. Regardless the stats don't match the chart you referenced so I am guessing it is a different source. I don't find a straight map for your numbers either.. can you clarify how you got those numbers?

The US numbers I pulled directly from a table in the EEOC report. The EEOC report says the data is from 2014, and the article's charts cite "Latest comparable data as of 2014", so I am not sure why there are minor discrepancies in the article's US numbers unless they didn't use 2014 data.

Which table? Thanks.

I can't find your numbers in the link you provided. Can you point me to the table you referenced?

Figure 5.

The operational principle of political correctness is we imagine a way we would like the world to be and then we act like it's that way. When people dispute with empirical data that is not that way, they are destroying the magical thinking that will make it that way. They are a heretic that must be persecuted. So people hide that the empirical reality and the imagined perfect world are not matching up.

This is much like Soviet plant managers would lie about their production numbers to avoid getting sent to Siberia because the politically correct idea in those days was that communist production was at least equal or better than capitalist production. Similarly, in order to give praise to the imagined perfect world, we offer up a regular sacrifice to the imagined perfect world in the form of a persecution of non-believers, some would even call them saboteurs of the imagined world when they point out its lack of empirical observability, even though after the persecution the empirical dissonance remains and doesn't go away.

I once went on an interview to Uber in Palo Alto. At lunchtime, when we went to the cafeteria, I saw most of the 200+ workers were either Chinese or Indian, sitting in their prospective table. Same with LinkedIn and workday in Santa Clara. I don’t see any racisms in Silicon Valley

If most were either Chinese or Indian, then they were greatly over-represented compared to US demographics. If you replace Chinese with white, it'd be taken as a sign of racism and privilege. So why isn't it racism in this case?

I remember a while ago there was an article about a company that openly promotes women and discriminates against males. The article was in a positive light. It seems that only whites are guilty of racism and and only males are guilty of gender discrimination.

More and more people are adopting new definitions for racism and sexism -- "prejudice (against a certain race/sex) plus power." Combined with the concept of white male privilege, these new definitions imply that only whites can be racist, and only males can be sexist.

Approximately 1/3 of the world population is either Chinese or Indian. Probably even more if you consider only countries with a certain minimum level of economic development.

Do you mean to imply we should be using world population statistics to judge if US companies are sufficiently diverse? Because by that standard, we could judge the US itself, by not reflecting global demographics in its population, to not be doing enough to advance minorities.

Minorities by US standards, since otherwise white people would also be a minority, and not even the largest one, depending on how you define 'white'.

> Do you mean to imply we should be using world population statistics to judge if US companies are sufficiently diverse?

Nah, I'm just saying that the tech industry hires from a global pool of candidates, and it's not entirely surprising that a bunch of them are from China and India. If a bunch of them were from Malta and Andorra, that might be weird, but it isn't so.

You realize racism doesn't necessarily mean discriminating against _all_ races, right?

To me the issue is not discrimination but rather representation. I will give you a concrete example. Our team (working in medicine) had very few Latinos for a long time. As a result of this we completely avoided gathering training data from Latin America because no one was well versed in how to conduct business there, yet we had data/deals with US, Europe and Asia. As a result our algorithms will likely not help Latin America and Latino Americans as much. This then perpetuates marginalization because it could lead to worse health outcomes for Latinos.

Why the dimension of race and not other dimensions? It is a real dimension with cultural and medical significance. Cancer presents differently for different races for example.

When you have large pseudo monopolies it is important to have diverse groups of people running them to avoid effects like this that can make marginalization worse.

Another classic example is voice recognition and accents. By not having more Spanish speakers at a company, if your company makes a voice recognition product it is easy to have blind spots. Your product may fail for those with an accent, and you won’t notice or think to measure along those dimensions.

I agree, however, that strictly proportional representation is not important. However promoting diversity is important and when you have numbers like 3 or 7 percent it’s easy to end up with large, important sub teams with no representation for historically marginalized subgroups. And this can lead to worse products, especially when looked at from the point of view of marginalized subgroups.

The main point of the article is that it is pretty sketchy to be hiding these numbers

The example you gave is a diversity of skills, and that diversity of skills was actually a requirement for your business. If the inclusion of individuals possessing those additional skills (ability to speak Spanish, ability to network effectively in a given foreign country/culture) happens to map on to a racial or ethnic diversity, great, but that's not what's being advocated for here.

This also brings up an interesting point of contention. The diversity dance has 2 arguments at odds with themselves.

Claim 1: All populations of people are inherently equivalent and should therefore have proportional representations in all things.

Claim 2: Diversity is a boon because populations are inherently unique and provide a unique set of characteristics to the table.

Pick one. You cannot make both of those claims simultaneously.

You are advocating in favor of Claim 2, which is fine, but what if the position in question doesn't actually benefit from a diverse set of skills? What if it's a low level circuit designer that tends to favor a particular skillset and personality type? Is it not possible that attributes more commonly represented in a certain population might be overrepresented in sectors that require those characteristics?

In contrast, if you advocate instead for Claim 1, then there should be no discernible advantage to diverse hiring, it's just that a failure to do so would be indicative of bias.

I don't know anyone who believes in Claim 1 as you've presented it. There is an alternate version that I do believe

"All populations of people are inherently valuable and therefore should not be left behind by advancements in technology"

The examples I gave were both cases where it was not obvious at all to predict that the skills would be needed.

For example, while the low level circuit designer probably won't benefit from cultural and racial awareness, he may not always be a low level circuit designer in the company. He may rise in the company and start making decisions about how/where/which circuits are built. And at some point having cultural and racial awareness would be a benefit.

All populations are inherently valuable, to whom? Every business values every population intrinsically? How do you decide how valuable they are? Do you need to hire as many Maori from New Zealand and as many Inuit from Alaska as their world populations would dictate, or are their intrinsic values allowed to differ in any capacity?

Let me rephrase

Individuals are valuable regardless of group memberships. Therefore it makes sense to avoid leaving large groups of people behind, regardless of which group it is.

Our team (working in medicine) had very few Latinos for a long time. As a result of this we completely avoided gathering training data from Latin America because no one was well versed in how to conduct business there

Isn't that more diversity of experience versus diversity of racial makeup?

You could have a white guy who grew up in Latin America provide you with the insights. A Latino who grew up in the US might have absolutely nothing to contribute.

The point is that without diversity, even awareness of these problems can be missing. Obviously once you understand what the skill/experience gap is, you can go out and try to fill it. But a team lacking in diversity will also end up with these sorts of blind spots. Latin Americans can (and did :) ) notice this gap and now it is beginning to be filled.

But my point is that it's diversity of experience/knowledge, not racial diversity that solves those problems.

My point is not that the best way to solve these problems is through racial diversity. My point is that they inevitably crop up due a lack of awareness when you do not have diversity.

Then we disagree. I'd much rather filter based on diversity of thought/experience/knowledge if I'm looking to cover all my gaps, then base it on race where there is a chance you'll end up with group think because they all have similar thoughts/experience/knowledge.

As a practical matter I think it's pretty hard to get diversity of thought/experience/knowledge without also getting racial diversity. The two are extremely correlated, because social groups are often racially homogenous. In addition, filtering based on diversity of thought is pretty difficult. It's already so hard to interview people well...

https://www.washingtonpost.com/news/wonk/wp/2014/08/25/three...

I agree that we all have blind spots and that diversity in decision-makers can help correct these. If this principle was applied consistently across any dimension where such blind spots are likely to exist, I would be on board. However when some axes of diversity are elevated over others, based not on likeliness of blind spots but rather on unrelated grievances ("historically marginalized subgroups"), the whole exercise takes on a character of social activism. Social activism is important, but has blind spots of its own, and the culture of activism these days seems increasingly insistent on not recognizing them.

I think gender, race, class, and sexual orientation are probably the most important dimensions. I gave very specific reasons why race is important in my field of medicine, and I don't know why you think I am highlighting it as an important dimension because of historical marginalization. What are other ones that should be as elevated?

I think I drew that conclusion because you mentioned marginalization four times.

I think political orientation leads to lots of blind spots. If it is important that businesses have representation of their user-base, I think diversity of political opinion is a natural axis to consider.

I think I agree with this. But I don’t think you need proportional representation, only representation. I think the percentage of people who vote republican is probably well north of 10% for most of GAFA, and if it’s not it’s worth looking into why.

When you say "Latino", do you mean someone who actually hails from Latin America? Because I'm Latino from the United States and I'd probably be just as lost when it comes to dealing with Latin America as a non-Latino from the United States.

Yes, I’m in the same boat. But I think you’d be a lot more likely to notice that there was something missing.

> Cancer presents differently for different races for example.

Ok, but ... you've been talking about Latino as if it is a race. Does the word Latino describe a race or a culture?

That’s a complicated question I don’t have a good answer for. There is definitely a racial component to it though.

And which race would Latino represent?

If you're Latino you are much much more likely to be mestizo. Which basically means it's a confusing mix of Spanish, indigenous american, and African.

https://en.wikipedia.org/wiki/Mestizo

And I'm not sure it's easy to tell to what degree/how much of a mixture you are.

Judging by [0], the whites are most prevalent non mixed ethnicity in South America and if you also consider that Mestizos and Mulatos are half white that also means that means that large majority of the population is white to some degree.

[0] https://en.wikipedia.org/wiki/Demographics_of_South_America#...

Two things

1) if the first table is self identified data I would strongly expect it to underrepresent how many people are mixed. Unless you have gotten genetic testing, it’s hard to know like I mentioned. And there is social pressure to identify as white. 2) Even then, whites are still pretty much a minority in most countries, especially the ones that have more significant immigration to the US.

hammer on nail. I don't know why the demand for diversity in consumer & people oriented businesses upsets so many people.

It's simply good strategy to hire a diverse workforce so your products/services can spread to all types of markets.

Diversity in medical training data is important, through it is much more complex than getting "US, Europe, Asia and Latin America". A EU citizen in the south will have complete different set of environment to a citizen of the far north. Using climate zone classification, EU has both Temperate (mesothermal) climate and on the opposite end Polar and alpine (montane) climates. One place where the sun never goes up for part of the year, and an other where people go to beaches for vacations in the sun. Medicine that involve D-vitamin, sleep, cancer risks and a lot more is greatly effected by the latitude, which is part of the combination that is genes and environment. In the end you need a multidimensional matrix with everything from environment to genes, such as ethnic groups, gender, climate, culture and so on. If you are doing something which is similar to scientific studies then there is also an error margin in the data collection, such as in ancestry, and the error rate differ from country to country. Treating EU as a single group is plain terrible from a representation perspective.

https://upload.wikimedia.org/wikipedia/commons/d/d5/K%C3%B6p...

Agreed. The general population argument also has troubling implications. For example, why don't we talk about the lack of representation in tech by whites?

After all, they're 3/4 of the U.S. population. At Google, this means they're already underrepresented at 53.1% of all employees in 2018. That number does not separately identify whites as non-Hispanic, or more significantly (and most troubling to anyone who's seriously examined the implications of this type of argument), non-Semitic.

There are a few things at work - Silicon Valley companies do not necessarily match the local demographics either. And even if you look at county wide stats, it doesn't account for displacement to neighboring counties. And these neighboring counties also represent the "silicon valley" workforce.

There are roughly half as many black people in the "Bay Area" compared to 20 years ago. But the black population increased in neighboring counties significantly.

While Asians are "over-represented" in tech, they are under represented in leadership and senior roles - compared to their representation in the lower ranks.

While there are a lot of technical roles at tech companies, the results aren't any different for those non-technical roles either.

And let's not even discuss how hiring and recruiting practices lead to a very limited pool. If most black engineers come from HBCUs and tech companies never recruit at HBCUs - well they are missing out on a large pool. If most hires come from internal referrals and the average person doesn't have a diverse friend pool - it keeps the status quo.

So diversity is broken on multiple levels, and it is easy to hide.

1. tech workers don't match the percentages in the larger regional job force 2. non-white people are not represented in leadership, not even commiserate to their ratios in the larger tech workforce 3. There are a lot of non-engineering jobs that should have more diverse pools (sales, marketing, administration, etc) and these roles still have the race problem. 4. Recruiting and hiring processes reinforce the status quo on "diversity"

Something is broken, but we spend a lot of time trying to obscure that fact.

> Suggestions to anonymize the interview process typically produce results that are even less representative of their desired distribution, because almost every company is actively discriminating in favor of women and "marginalized communities."

Two citations needed.

I've been on hiring committees for years, at both very large, and very small companies. I've hired and rejected both men and women. I've never seen a "diversity hire" done.

Also, you are the first person I've seen say that blind interviews decrease diversity. In fact, the study that gets cited the most involving orchestras showed the exact opposite. [0] I would love to read where you got your information.

Thanks.

[0] http://gap.hks.harvard.edu/orchestrating-impartiality-impact...

If we're doing anecdotal, I have been in many hiring processes and have been told to favor diversity.

If you want the most recent one from memory:

https://www.wsj.com/articles/youtube-hiring-for-some-positio...

https://www.theverge.com/2018/3/2/17070624/google-youtube-wi...

http://fortune.com/2018/09/25/brainstorm-reinvent-mellody-ho...

Blind/anonymized hiring (which I am in favor of and would love to see advocated) has examples going in either direction. The orchestra one you referenced is the go to example of it doing its job well, but if it generally produced the results they were looking for, it would be a noncontroversial strategy being pushed, which it is not.

http://behaviouraleconomics.pmc.gov.au/projects/going-blind-...

All you need for a political controversy is for a group to perceive themselves as losers. The merits, are completely immaterial to whether a controversy exists or not. Some guy gets passed over for promotion, and he looks for a scapegoat, suddenly there's a controversy.

As far as a lawsuit being filed, I'll reserve judgement until the case plays out, thanks.

>massive overrepresentation of Indians and other Asians

As indicated in the article, there's evidence [0] that there is profound underrepresentation at the leadership level relative to junior representation.

[0] https://c.ymcdn.com/sites/www.ascendleadership.org/resource/...

Yes, but again, the assumption is racial discrimination. That is the de facto assumption for every possible case. What about language, or culture, confidence, etc? If you are a foreigner to a country, of which many Asians are either first or second generation, you might be comfortable performing culturally agnostic technical feats, but not at all confident taking a leadership role. Or if we want to entertain that there is discrimination at play, how does cultural discrimination map on to racism. You discriminate culturally when choosing who to be friends with. If you like bros, hipsters, nerds, outdoorsie types, etc, you're making a cultural discrimination. If there is a degree of correlation between culture and race/ethnicity, which there is, then how might that come into play regarding the roles people assume in an organization? The presumption of racial discrimination in a vacuum is a refusal to content with any level of nuance or complexity.

You're dismissing my point and deflecting with something that I neither referenced nor disputed.

You made this claim:

>the "anti-brown people" narrative absolutely dissolves in the face of the massive overrepresentation of Indians and other Asians.

Nothing you have said constitutes absolute dissolution of this narrative. The one measurable claim you made was overrepresentation of Indians and other Asians, which holds no water in light of drastic underrepresentation at the senior level. After I pointed that out, you responded with why you believe that discrepancy could be justified. How does that constitute irrefutable evidence that the narrative is misguided?

Does that take into account that China graduates 7 times as many people today as 20 years ago? It makes sense that there are few Asians in leadership positions simply because they barely have any educated people with lots of experience.

https://assets.weforum.org/editor/PSSvl1XRKP4L2tFL2PNtPufC-v...

Your more general point is argued a little more strongly than I would argue it, but there is a stronger and more limited point against demographic comparisons between tech industry workers and the US population as a whole. That argument is that tech industry workers are hired from a global pool, not a national pool. If the Bay Area tech sector hires 200 Czech engineers in a given month, they increase the number of Czech engineers in the Bay Area tech industry by 200, and they also increase the population of Czech people in the United States by 200. Same numerator, but a different denominator. Purely by mathematical necessity, this directly causes demographic disparities.

My beliefs aside [0], you do bring up a good point. I can't find the HN thread, but there was a good TEDx presentation on it as well:

Given two curves on the same graph, you are not guaranteed to be able to min/max them simultaneously.

For example: say you are a mortgage broker at a bank and you have to give out mortgages to people in your community. Obviously, the people in your community are diverse. There are men, women, white people, black people, homosexuals, heterosexuals, etc. all wanting mortgages. Say you have some historical data that you know to be accurate, namely the credit score of successful applicants, the associated foreclosure rate, and some demographic data like race and sex. Also say that you want the bank to remain solvent and competitive, handing out loans to the best applicants and also garnering a reputation for being a 'fair' community bank that people will actually be able to get loans from. Now, the question is, what is the cutoff for credit-scores that will cause an applicant to not get a mortgage [1]?

Say, on the y-axis is the foreclosure rate, and on the x-axis is an applicant's credit score. You have some curves for white women, black women, white men, black men, and many other permutations of the human rainbow. These curves are all different from each other.

Now, if you set an arbitrary cut-off score, you may be setting a score that says that white people are less likely to get a mortgage than black people. That's obviously disenfranchising white people in your community, and you should change the cut-off to be more equitable to the people around you.

However, now that you have changed the cut-off for approval of a mortgage to be racial equitable, you have now made it such that men are less likely to get a mortgage than women are. This is also not good, and you should change the cut-off again to be fair to all sexes and genders. However, now you are back to being unfair to white people, again.

Given the data you have, you are in a situation where, no matter what you do, you are discriminatory. In fact, I'd say that this is the most likely situation to be in. That all the curves and graphs would align just so and that you could be non-discriminatory under ALL scenarios is extremely unlikely. Honestly, you do have to pick and choose who you will not discriminate against.

This may seem disheartening, and, yeah, it is [2]. However, that does not mean that we shouldn't try to change things. If anything, understanding that you will very likely be discriminatory no-matter-what, is helpful. You now have a better view of what you can change and how that may affect things. You can choose where to set your parameters with better clarity towards your fellow humans. Maybe you oscillate between gender-parity and height-parity. Maybe you choose to focus on income-inequality for 5 years and then switch to racial-inequality for a focus. Whatever your thesis is on how to gain better equity in your community, maybe you now have a better understanding of the mechanics of the system and you can better affect it positively.

[0] In talking privately to friends that are also considered 'minorities', discrimination is occurring and is systemic. Though this is personal anecdata.

[1] I'm trying to simplify this as much as possible. Obviously, real applications are VERY nuanced and complicated.

[2] Unless you are a journalist. Then, well, this means you will always have a lot to write about!

If the bank does not take race or gender into its decision-making process, it is not subject to race/gender discrimination. Period. If the outcome produced is that people of a certain race or gender receive more or less loans, that does not mean that there is anything wrong with that system. The banks isolated system is not at all discriminatory, it just receives different inputs that happen to differ, on average, between population groups.

If you're concerned for the capability of a given arbitrary population group to have a different outcome, then the goal posts need to necessarily be moved back. Why did that population have lower credit score, lower income, lower wealth? Historical racism? That certainly has some role, but that has no bearing whatsoever on whether or not the present system is itself racially biased. Is it the code or is it the data?

Regarding your anecdata, consider an interesting thing I saw recently:

https://www.youtube.com/watch?v=lk3tVSJSkQM

This was meant to highlight injustice, but really just shows how differently people can interpret the exact same events occurring to them on the basis of their preconceptions.

I mean, this is kinda the point.

No matter what you do, you'll be discriminatory (to a very high likelihood). Even if you don't look at the break-downs of your data, you will be discriminating against some group or another. It is incredibly unlikely that you can max/min two curves at once, let alone for all the classes, races, sexes, castes, genders, etc. that exist.

I agree that root causes should be addressed first and is the best use of time and resources. However, that's not how journalists see it. They see a red-meat story and they go with it.

Federal investigators, journalists, community organizers, etc don't really care what metrics the banks are using/not-using, they care about the results. And I'm saying that there will very nearly always be discrimination if you use nearly any metric. And that's not a bad thing. Being in a Catch-22 is part of life. Trying to get out of one is the thing that matters.

More like journalism is using race to hide its jealousy problem. Tech now has the power, influence, and compensation that journalist used to have, and they're mad about it. Most of these companies go way out of their way to recruit minorities and women. If you believe that not having enough of them in tech companies is a problem, you're going to have to look further down the funnel, to college major choices and honestly probably much earlier. These groups are simply not graduating with CS degrees or programming skills in the numbers necessary to be equally represented as employees at these companies. Whether that's because of discrimination at earlier stages or simple preference, fixing it at the level of hiring for tech companies is way way too late to improve anything. By then it's a completely zero-sum problem. There are only so many viable candidates from these groups with the requisite skills, if Google hires one, that's one less that Palantir can take. It's just a silly game of musical chairs until we fix the pipeline problem, which is that there are not enough viable candidates from these groups. Once that is fixed, then if companies are still not hiring them, let's talk about that, but I very much doubt that will be the case.

Would you be willing to share your ideas on what can be done at the beginning of the pipeline?

Good primary and secondary education systems and affordable colleges.

Countering American anti-intellectualism is needed as well.

Yep. America hasn't even started to reckon with the attitudes encapsulated in the n word (no, not that one, the other one).

Step one (and really the only step) is that people have to want to become developers/scientists, sit in front of a computer all day, read and learn, all the time.

There are a million different reasons someone may not want to do that or may be encouraged against it, but you'd have to look at every individual and culture to find out why. It may not even turn out to be a bad thing. What if someone wants to become a chef or athlete instead, is that a problem?

For choosing a major and career I think step makes sense but I think that is near the end of the pipeline.

What about exposure to the field and introducing computational thinking into earlier pars of the education system? Not necessary making kids program as their first exposure, but exposing them to computational concepts (maybe start with power of twos when learning multiplication, or control-flow-logic for writing a story....).

There are so many hours in a day. Exposure to computation as opposed to not exposing to what? What would you trade this with?

I think that's a great idea.

Honestly I don't think any culture in the west has ever been truly supportive of becoming a STEM person. Until very recently is was extremely derided in even white American culture as being "for nerds".

A more positive exposure to technology at an early age can only be a good thing. I think we'll still find very few people want to become engineers due to the required workload and that will be ok.

I think even more helpful would be framing how this actually translates into the real world - what does this sort of thing enable in the real world.

I loved the show Numbers because it was a show about how math helped detectives solve crimes. Way more interesting than just doing algebra about speeding trains.

I think sometimes there is an idea that everyone has the same problem and your problem is universal.

For example I heard about a hackathon for black and latino kids. And the ideas that came out of it were really different. One app was a social networking platform for private school kids to connect with other black private school kids to talk about their experiences and vent. Another idea was a payment app to send money to Mexico without having to go to Western Union or whatever.

There is a startup in Oakland that has an app to help people who are caught up in the criminal justice system to keep track of court appearances and hearings and stuff so they don't need to go to jail for bogus stuff like missing a court appearance.

Technology has the opportunity to enable and transform - if we make sure everyone has access.

Young girls/women have much more sexual value than young guys in general, so it makes sense for them to invest more in their social life and looks, and less in their future earning potential.

Still, it can be changed: in China studying is increasing social value much more than in Europe / US for example.

In my experience the theme of what someone wants to do as a career often forms in high school. Programming/software is equated to the more introverted gamer/robotics types which often confers lower popularity and social status then other extra-curricular activities. Men and women alike choose to opt out of programming for this reason, although the absence of women is much more heavily felt.

Sure, instill discipline in schools. Go to a school of 'underrepresented minorities' and you'll see an incredible amount of misbehavior. Anyone who struggles to listen and not gratify their most immediate desires will probably fail to develop technical skills.

Also, immediate suspension for kids who bully academically-inclined students for 'acting white'. Not only is it a racially charged statement that probably has no place in school (or so progs would tell us), but it seems levied exclusively against the kids who need education the most.

Maybe have a little empathy here. In a low-income school, kids are walking in the door with way more problems than your upper middle class school where most kids have their basic needs met.

Also, it is time to retire this stereotype that every non-white kid who is academically inclined is being bullied for acting white at school all the time. I get the most grief from white kids about "acting white" growing up. From fellow black kids it was pretty rare - 1-2 times. Mostly other kids were more inclined to help uplift me.

> upper middle class

The US is a long way from having its entire white population in the upper middle class: http://www.nccp.org/media/releases/release_34.html

You've italicised "pipeline" as if you believe this is a novel insight, but actually it's very common to dismiss diversity and inclusion efforts by redirecting attention to "the pipeline problem". Unfortunately there is evidence that the pipeline is far from the only problem, and even after successful interventions to swell the pool of qualified diverse candidates, companies' representation is still extremely skewed, and those candidates still encounter hiring barriers that other demographics do not.

Here's one thread from an organisation that spent 10 years working on the pipeline and found many companies wouldn't hire their candidates: https://twitter.com/Code2040/status/1092853501766467585?s=19

(Edit: gah, pasted the same link twice instead of the Twitter thread I meant to post)

And some more links on the "pipeline problem": http://isitapipelineproblem.com

> You've italicised "pipeline" as if you believe this is a novel insight, but actually it's very common to dismiss diversity and inclusion efforts by redirecting attention to "the pipeline problem".

I'm not dismissing diversity and inclusion efforts. They're being made, and i'm glad they're being made.

> Here's one thread from an organisation that spent 10 years working on the pipeline and found many companies wouldn't hire their candidates: https://twitter.com/Code2040/status/1092853501766467585?s=19

> (Edit: gah, pasted the same link twice instead of the Twitter thread I meant to post)

> And some more links on the "pipeline problem": http://isitapipelineproblem.com

I'm certainly open to arguments of this form, though the links you've pasted here aren't very convincing to me. The Code2040 people say the companies said their candidates weren't qualified. I see no attempt at rebutting that (by e.g. trying to objectively measure 'qualifiedness' and then showing that these companies were preferring equally qualified white candidates).

Similarly, for the attrition rates for women, I think that very same data can be used to show a preference difference rather than an inclusion problem. If you wanted to make the case that it was an inclusion problem, you'd have to believe that fields like medicine and law, which women have become quite well represented in, were substantially less male-biased than tech. If you drill down in medicine though, you'll see that women sort themselves into sub-specialties that deal with human contact, where men sort themselves into sub-specialties that do not, on average. Again, pointing to a fundamental preferential difference.

In my own personal experience, i've known a few women who are excellent CS students and programmers. They had no shortage of capability relative to the men I knew. However, without exception, every single one of them aspired to be a project manager of some kind, rather than an actual engineer. They were more than capable of doing the engineering, and they did do it. However, their end goal was managing people and process. Most of the men I know who are engineers, hope to be writing code until they die. This is anecdotal of course, but it's an experience I see echoed a lot.

One look at the website of Seattle's Garfield High School (34% black/latino) CS Club website pretty much sums it up: http://www.garfieldcs.com/

The makeup is similar at the college level: sitting in on CS courses at UW you see a sea of white, east asian, and south asian faces. I can count the number of black or latino students (out of the several hundred enrolled students I saw) on two hands.

It's a pipeline problem, full stop.

And corporate diversity goals are based on the pipeline, not the general population. Lots of folks in this conversation seem to be making utterly incorrect assumptions about programs that they seem to have deep contempt for. Unsurprising.

Well, right. Corporations internally work towards their diversity goals, which are sensibly based on the pipeline and aim to eliminate bias in hiring. That's why they don't want to publicize their numbers; they know people like the Bloomberg reporter who wrote this article will incorrectly try to measure their efforts against the total number of minorities in the US.

The hiring pipeline is discussed explicitly in the article, and comparison to national demographics is legitimate.

Comparing hiring to national demographics is not legitimate. You should compare hiring to national graduates with CS degrees, controlling for institution. That will tell you if there is discrimination in the hiring step. If you want to see if there is discrimination in entrance to CS programs, you go a step earlier, etc. And you keep walking back until you find the source of the discrepancy. Where do women get off the CS pipeline? Is it at the hiring filter? The college major filter? Somewhere in high school? Even earlier? It's disingenuous to place the blame for the entire pipeline at the feet of the entities at the very end. Somewhere there is a bias, but it isn't necessarily at the corporate level.

It's certainly true that some minorities are underrepresented in tech. A lot of the irritation you're seeing comes from the repeated accusations and implications that this is due to some unspoken policy of discrimination at tech companies.

Companies can and should make an effort to ensure they're hiring a diverse range of people. However, it needs to be done with the bigger picture in mind - otherwise it becomes a defacto quota system.

So policies that result in under-representation of POC are neutral, but changing hiring & retention practices to improve diversity is a defacto quota system?

If it is literally a quota system...then yes, it is a quota system.

What if there isn't a pipeline problem? What if there is an actual difference in preference? For example, looking at gender and STEM it seems that [0]:

a. Countries that push for more gender equality see less women in STEM.

b. Girls in school perform just as well as boys in STEM.

Also, men are also highly over-represented in low-end jobs such as drivers, construction, mechanics etc [1]. Is that also a sign to a bias/pipeline problem? Should we attempt to strive for diversity in those fields too?

[0] https://www.theatlantic.com/science/archive/2018/02/the-more...

[1] https://www.bostonglobe.com/metro/2017/03/06/chart-the-perce...

> What if there isn't a pipeline problem? What if there is an actual difference in preference?

To be clear, i'm including 'preference' as a sub-component of 'pipeline problem'. I think it's a cultural open question whether or not we consider this a 'problem'. I think the argument for preference is much stronger on the gender question though, than on the minority question. There's strong psych research indicating preferences against programming for women, but i'm not aware of any for say, African Americans. In the minority case, I suspect it's more of an access/opportunity issue early on, but i'm open to many views of these issues.

Given the link between tech work and depression, and more generally the link between depression and careers with very little physical activity, I think it's unconscionable to modify the preferences of people who would otherwise prefer to not work in tech.

By all means, help people who desire it. By why 'correct' people who don't?

Ya, i'm not a huge fan of 'correcting' people's preferences. I do think it's possible that part of the issue is how women are socialized, and it's also possible part of the issue is how CS is taught. However, I think the lion's share of the variance is explained by biologically innate preferences, and while there could be good arguments for changing those, I don't see any being made at the moment.

At a company I worked at they hired an young female intern in her late high school years, who made it obvious she didn't really want to be there. Our project manager made sure to tell us to "make programming seem as interesting as possible to her". When I was 17, I would have killed for an opportunity to work at a software company, and so would many other young kids.

It's time for the world to admit, there's reasons why some people are naturally pulled towards one discipline or another.

I have a 3 year old daughter, and I'd love to teach her math and programming. I sat her in front of the computer with me to get her interested and devised lots of fun math based games to play with her. But, no matter how fun I make the material, she's just not into counting things. "daddy, I don't want to count anymore" (after just a few tries) Stories about princesses and reading/writing however, this she can listen to all day long. I'm certainly not giving up on teaching her programming but you have to admit it when certain types make it clear their not into it.

We need to stop trying to attribute that to gender. I loved math as a kid, I thought it was fun. I was on the math team for middle school and high school. I had a computer at home from a young age, and played the math games my parents forced on me. In addition to the game, games. (I am 40 - so things were pretty different as I was coming up. But my family was an early adopter of technology - we had a computer when I was 5. And if I would have had different upbringing the outcome could have been quite different.)

Being a programmer wasn't something that was on my radar at all. I didn't know any. I didn't know it was a job that existed. As I got older it seemed like "computer people" only liked video games, magic cards, scifi books and didn't like to hang out with people. It seemed boring and antisocial - and not like my sort of people.

I probably would have been a great engineer. And while I work in tech, I am a marketer and have worked in more technical roles around software implementation, operations, pre-sales in addition to all of the marketing jobs.

So let's ask ourselves something else - how many people do we lose along the way because of exposure or the way we have socialized what an engineer is, does and likes.

The percentage of women that likes video-games/programming is lower than the percentage of men. That's just a fact. You can try to correct that with lots of outreach and exposure but it's only going to work for some people.

Anecdotally. I've worked at a (hard core strategy) gaming company. Even there, Most of women that worked there, that I knew, did not like gaming at all (if we're being honest).

I’m a current CS student. When I walked into my school’s CS computer lab, there were about an equal number of female and male students.

I’m also a transfer student. At my previous university, at which I was also a CS major, there were maybe like 5 women to every 15 men in a class.

This isn’t reflected at the tech companies I’ve worked at. There are far fewer women. This article, and articles like these: https://hbr.org/2016/08/why-do-so-many-women-who-study-engin..., point out that the number of women graduating do not match the number of women who choose to go into the tech sector, or who remain there.

The whole “diversity industry” (and it is a huge industry) is based on a lie... that “diversity” improves a company’s performance. The research on this is so shaky with so much empirical evidence to the contrary, and everyone knows it, which is why companies are trying to do nothing more than “lip service.”

> that “diversity” improves a company’s performance.

Hm - well, I agree that there's a lie floating around, but I don't think that's quite the lie in play. More often what they say is that you're hurting yourself by turning away qualified candidates based on their race/geneder/sexual orientation/whatever they're not seeing as many of as they think they ought to be. The problem is that, while this is strictly true, there's no evidence that qualified people of _x_ race/gender/sexual orientation are actually being turned away. For all the digital ink that's been virtually spilled on tech and diversity, I have yet to see _anybody_ claim, even anecdotally, that they, themselves are qualified members of diversity group _x_ but are unable to find employment. Instead, they express bewilderment that although they have no trouble finding employment, they don't see enough other members of group _x_... which they presume must be because all of the other ones are being turned away by prejudiced hiring managers.

This is something I've wondered about a lot as well. Look at any large company in the world, then look at how diverse they are. Probably isn't much of a correlation between one and the other.

Looking at companies like Google and Amazon and Facebook and what not, you'd suspect any effects from having a diverse workforce are massively outweighed by other factors like selling a product/service more people want, good product design, price, marketing efforts, etc.

I don't really see how diversity is tech's problem any more than it's hospitals' problem. If you want more engineers from more underrepresented groups you need to educate them. If there aren't enough doctors of group X it's not the fault of hospitals. It's the fault of society for not facilitating more of them going into medicine. It seems to me that the case for tech workers is analogous.

100% agree. Look at the diversity of CS courses at top universities. If minorities and females are not represented there then how can you expect companies who hire top talent to have them?

I find it a little absurd to compare the SV demographics to the US demographics, and then claiming that the problem is with SV's hiring.

It's a US problem that its society has failed to push the minorities up with better welfare and education; it just manifests itself as lower representation in SV.

I feel like articles like this are major trend of racism towards Indians and Chinese.

Lately, as American-born Indian-background person, I have been feeling discriminated during job interviews. My friends and I are not really looking to changes jobs but are always interviewing passively.

My black friends have recruiters hounding them on LinkedIn. They get more job offers, have easier interview questions, and even higher salarier.

While my experience is usually average. But when we compare type of questions we answered at the same company for same role, it is pretty obvious that I was asked harder questions.

On other hand, I can tell when my interviewers are mostly Indian, they discriminate in my favor.

This push towards intellectual and cultural uniformity is so incredibly anti-diversity. "All cultures must have the same interests as white people" is essentially what is being said after you account for all the various "systemic privileges."

EDIT>> to put it another way, white people consider something "wrong" if different demographics aren't interested in the same subjects as they are, at the same rate. It's the ultimate self-centered worldview.

That's a very interesting take, I've never considered it that way before. I guess that goes to show I need to work on my empathy.

Mathematically, not all companies can have a higher proportion of $GROUP than the industry as a whole. So if companies advertise and compete on having a higher-than-industry-average proportion of $GROUP, they paint a target on their back.

Since this has become yet another discussion about "the pipeline problem" instead of a discussion about a tactic major tech companies are using to duck accountability, this Twitter thread might be informative: https://twitter.com/Code2040/status/1092853501766467585?s=19

It's from an organisation (Code2040) that spent 10 years working to build a pipeline of qualified Black and Latinx candidates, only to find many companies had hiring processes that wouldn't hire their candidates anyway.

It's a series of opinions in tweets. no facts. no scientific anything.

From a person who (after looking at their LinkedIn bio) is in no way qualified to tell companies what they need _at all_

From what I see of the company & CEO, it's a self serving & self preserving job role & company.

Their facts are based the experiences of trying to place their program graduates and the experiences of those people. Code2040 had a very extensive program pairing young people with tech worker mentors, getting them the right education and helping to place them in tech jobs. And after doing that for 10 years they found without companies that were very very committed to retention, even with all of that preparation, it still didn't work. Hence the pivot by that organization.

They started trying to solve the "pipeline problem" and it turned out that wasn't really the problem.

I am very interested what the other half of black engineers do and why. Do they stay unemployed, or choose a different career?

I have a hard time imagining that so much educated engineers who can be effetively underpaid are not being capitalized on. That sounds like a wasted business opportunity.

Businesses outsource work to india or east europe, why not exploit locally? I've read they also import workers on hb1 because they can pay less for same talent, so this doesn't add up for me. I can't belive racism vs black people specifically wins vs money on such large scale, since it doesn't vs other races.

Interesting.

Who is funding these diversity studies? That should give a clue to what the true reason is. I am doubtful companies really care about correcting discrimination in hiring.

This seems like it's using Palintir and Oracle to tar all of Silicon Valley. Aren't Google, Apple, MS, Facebook, etc. leaders on diversity transparency across the entire business community, tech and non-tech?

>The vast majority of companies on the list that report their full diversity numbers are in the tech sector. Companies like Google, Facebook, and Apple have released reports on the demographics of their workforce in recent years, so it may be unsurprising that 75% of Fortune 500 firms publishing their numbers are in the tech space.

http://fortune.com/2017/06/07/fortune-500-diversity/

I am very interested what the other half of black engineers do and why. Do they stay unemployed, or choose a different career?

I have a hard time imagining that so much educated engineers who can be effetively underpaid are not being capitalized on. That sounds like a wasted business opportunity.

Businesses outsource work to india or east europe, why not exploit locally? I've read they also import workers on hb1 because they can pay less for same talent, so this doesn't add up for me. I can't belive racism vs black people specifically wins vs cash on such large scale, since it doesn't vs other races.

Interesting.

So, are companies less profitable and successful than they might be by not hiring on a totally meritocratic basis? In other words, do tech companies really reject potential employees for reasons other than pure merit?

maybe I am clueless but how can anonymous data turned into the government be trade secret protectable?

It's likely not. Even the premise that employment demographics are trade secret at all is malarkey.

More like hiding its h2b fetish.

"And as criticism grows over Silicon Valley’s bro culture and its lack of minority representation"

Isn't 40% of the tech industry made up of east and south asians? Do they not count as minorities? If anything, you'd think SV would be praised for being one of the most diverse industries.

"A growing body of research shows diverse perspectives bring tangible benefits to firms that support them"

You see this so much in these types of articles. "Diverse perspectives" doesn't necessarily mean "diverse ethnicity". A group of 3 white men ( CEO, factory work and homeless man ) will have greater diverse perspectives than 3 female CEOs ( black, white and asian ) who went to the same schools, hung out with the same crowd and live in the same area. Also if her assertions were correct, wouldn't more diverse firms naturally have beaten out less diverse firms and we wouldn't need to legislative or force diversity in these companies?

If what is being stated in this article is true, why doesn't YC or any VC firm gather the most diverse set of founders to start a new Google, Facebook, Apple, etc? "Research" has shown that the new diverse google, facebook, etc have tangible benefits which must translate into success. Right?

Instead of complaining about it, why not start a diverse company and take hundreds of billions from the less diverse companies like Apple, Google, Facebook?

Also, does this apply just to tech companies only? What about another highly successful firm in the silicon valley area - the Golden State Warriors. Instead of having 5 african american starters, why not diversify the team since "research" has shown that has tangible benefits.

I'm against racism of all kinds whether it benefits whites,blacks,asians,etc. I'm against sexism as well - whether it benefits males or females. If there are systematic obstacles, lets remove those rather than creating more racism and racial obstacles.

You realize the "diversity" problem is also impacted by the funding problems. It is really hard for people of color and women to get funding - from any source. Banks, VCs, Angel investors and so on. Combine that with the wealth gap and it makes it a lot harder to get started.

If lack of diversity isn't a problem, why aren't these companies open with their figures? Why go to such extents? There's also research to suggest women and minorities are underpaid relative to their male, white counterparts, shouldn't companies fix that?

Is it a crazy idea the reason is that the figures themselves are irrelevant?

Why are they irrelevant? Why can't the data be used to address this disparity? If certain groups shy away from STEM careers, why not try to do something about it?

All of the companies named in the article are indeed trying to do something about it, and almost certainly use the data internally to address it.

Because the media coverage would turn into a "problem" that would be a PR issue.

Doubtful it can get worse than it already is, lack of diversity in tech is a widespread issue and known all too well.

Even the general public is in a sense aware of it, when people picture stereotypical programmers, I doubt they have a black Muslim woman in mind.

Transparency is important.

You're lumping in two different issues: (perceived) lack of diversity on the gender and/or minority axis (or black/Hispanics really, as Asian minorities are well represented it seems), and companies underpaying certain groups.

Actually, if you can pay women and minorities less for the same job and get away with it so far, and the abusive companies do just that, I'd guess those abusive companies would hire more, not less, people of those groups to "save" even more money.

My point was that certain underrepresented groups are treated differently in STEM fields (particularly tech), they either face obstacles to being hired, or, once hired, are not paid proportionately relative to their white, male colleagues. Research exists to support both of these points.

Reply on news.ycombinator.com