Hacker News Reader

Wouldn't the weakest male and weakest female founders be underperforming more due to individual factors than bias?

Maybe a mean of the lower quartile would remove some the the noise.

Suppose the cutoff for men is C but for women is C+K. Then the weakest man can be expected to have quality C+epsilon, while the weakest woman will have quality C+K+epsilon.

Here epsilon is how close a typical candidate will be to the cutoff, and is mainly a function of the sample size. I don't know the behavior of epsilon off the top of my head.

I think MattHeard understood this and his point was that the weakest man will realistically have quality C + epsilon + random, and the weakest woman will have quality C + K + epsilon + random. The random term arises because no evaluation process is going to perfectly tell you how people are going to end up performing. But yeah, this seems fixable also, by averaging some number of the lowest performing members of each group, as MattHeard suggested.

I'm not 100% sure how to deal with noise + extremal statistics. But I've got a 12 hour plane ride ahead of me tomorrow, so I can probably work out a fix.

I suspect the math is easier if you reframe the hypothesis to be about the derivatives: adding one more person from either group should have the same marginal effect on outcomes. As you get more observations you get more in a neighborhood of the cutoff which gives a consistent estimator of the derivative.

> In practice, measure the weakest male and weakest female to be accepted in your sample set, or some similar approximation.

In practice, where do you get these definitive unbiased measurements of the expected return of investing in an individual (or graduation probability or some other score)?

This test can be used after the fact, so you're not measuring expected return but actual return.

It's pretty easy to measure actual return (or actual graduation rate, or actual GPA, etc.).

What test? We're talking about computing the minimum of what before-the-fact EV should be (based on what set of information?) here, not computing what the mean EV should be (for which you can add up the outcomes and divide by N).

> What test?

The test for bias proposed by pg and modified by yummyfajitas.

> before-the-fact EV

Why are you bringing in before-the-fact EV at all? That's not a component.

The test is based on comparing the minimums, yes. So, for example, what is GPA of the worst female and male students. What part of that requires expectations?

Before-the-fact EV is what the entire question of bias is about. In the case of investments, your minimum outcome is that you shut down the company. Comparing this for male/female sets is useless. Also, looking at other aspects of the distribution (e.g. how many $10M exits, $100M exits) isn't useful because for investing, you want to maximize mean return, and it's not like looking at 20-80%tile outcomes numerically tells you much about that -- are these people saving dead companies or are they derisking $10B companies?

With students there is more of a connection between before-the-fact EV and outcomes, and much more information exposed (grades and course registrations on a semester-by-semester basis). Without that information (if you just looked at who graduated, who didn't) you can't really say as much. (You could run a Netflix-like competition to see who has the best graduation prediction engine and use its estimates as your definitive answer to what an unbiased EV predictor would say about college applicants.) With GPA information, let's say you look at the 10th percentile GPA's in each set and below, for boys it's 0.1-1.1 and for girls it's 0.1-1.5. What does that tell you about before-the-fact EV cutoffs? It doesn't tell you much, because one set of students, when they fail, could be more likely to fail harder than the other. It takes a lot more work than just that.

Isn't this not as meaningful as the mean, because the minimum doesnt really tell you about true capability? I can also easily construct an institution that satisfies your parameters, the minimums of two groups match, as in they have a token member at C for every group, but the characteristic performance of all members is different?

This in my eyes amounts to a simplification that is neat mathematically, but removes most of the useful information. I admit though that if the minimums did not match very well, that is an obvious sign of bias.

Edit: I suppose what I'm essentially saying is, what if minimums do match, then we cannot rule out bias.

If you think you can construct examples that break my test, go ahead and do it. I'm curious to see what you can come up with.

However, your claim that you can have "a token member at C for every group" is merely the claim that adversarial sampling can destroy any statistical procedure. So what? This is boring mathematically. Nor is it relevant to the problem at hand unless you want to claim that First Round Capital is actively conspiring to both a) be biased and earn less money and b) waste more money hiding that bias by funding tokens.

In a grandchild comment, you note that how close the min of each group is to the cutoff depends on sample size. This is true, but I think it's deal-breakingly true.

Suppose we have to input groups, A and B. Members of each group are distributed as Exp(1), ie same underlying distribution. Our selection procedure is totally fair as well: we take everyone without question.

However, there are 9x as many people in group A as group B. So the min of group A accepted (= the min of all group A) will be distributed as Exp(9 * |B|) and the min of group B accepted will be distributed as Exp(|B|).

So in expectation the min from group A will be smaller than the min from B, and indeed this happens 90% of the time. (Aren't exponentials nice?)

Of course in this case we can note that this effect is from differences in sample size and exactly correct for it. But normally we will not know how to do this correction properly because we don't know the true underlying distribution or the acceptance criteria.

First of all, making a group 9x larger doesn't change the shape of the distribution. The distribution would look like 9 exp(|x|), not exp(9|x|).

Secondly, I calculate a p-value for this test here:

https://news.ycombinator.com/item?id=10484309

It's proportional to the smaller of the sample sizes. I'll have a more detailed writeup soon. Also, this test is non-parametric, so you don't actually need to know either C or the distributions f and g - all you need is a certain level of uniform regularity in f and g.

A note: The original post is correct in what it says. It is saying if one draws N independent samples from an exp(1) distribution then the minimum is distributed as exp(N). This is true. Because of this, for any finite sample there is a systematic 'bias' by your metric if the sample size differs between the two groups. However, you are correct in that this difference vanishes as N becomes large and further that one can make controlled statements about the expected size of this difference, given the distribution.

Oops - I did misread what he wrote. He was discussing the distribution of the min of the sample, not the distribution itself.

Also, there may be a bias in the true p-value, but there is none in the p-value bound based on h(d) and min(N1, N2).

I like your proposal in theory but I doubt it works in practice. I'll try to explain: in practice what First Round has done is measuring post investment performance. I'm pretty sure that if the sample is big enough you will have an "all male" startup and a "with female" startup that failed thus the min being zero in both case. This doesn't mean there's no bias.

The point is that when you decide which company you finance (the source of bias) you make an estimation of future potential. But your test (and PG's one) measure ex-post results. Since there's a lot of uncertainty between the ex-ante and the ex-post measure, your test doesn't work.

Let me put in another way. The measure you are able to perform is not H(x-C)f(x) but a * H(x-C)f(x) + (1-a) * random where random is a random number and a is the weight of f(x) on the final outcome. You are right if a is 1, you and pg are wrong if a is 0 (but it will be a problem for the VCs), for everything in between you have to make assumption on the distribution of random.

I discuss in a different comment the effect of additive noise. I admit I haven't worked out the details of the min statistic and additive noise yet. But I'm actually fairly bullish on it - this problem is mostly just simple statistics.

I also suspect that first rounds data is useless. PG is being too credible about it - I suspect he is just being political and making his post more PC, since he doesnt seem that statistically naive. (Taking a lesson from Sam Altman most likely.)

There's also the fact that the effect size is huge — 63%. Most real world population differences are not weird enough to generate that large an effect without bias. The bigger questions are how much would the effect size change without removing outliers (Uber as pg mentions but presumably others too) and how significant is the result.