Heh, Peter Thiel did, too.

I would be curious about startups you do not fund

I would be interested in the valuation of the group of companies that you "almost" funded.

Let's say you ranked startups during admission on a 1 to 100 scale, and took the top 20 scoring startups. (I don't know if this is the metric you used, but I imagine the following protocol could be adapted regardless; as long as you kept the data.)

Take the bottom 5 scoring startups that you funded - i.e. the ones that just made it. Now compare their outcomes to the next 5 --- i.e. the top 5 scoring startups that didn't get funded.

If you see a significant difference, that could show that being a part of YC adds tremendous value to a company. On the other hand, this sort of analysis is really hard because the data points are so few, and the noise in the data so high (the volatility is just too much to make meaningful conclusions).

It's still an interesting experiment to run, because there is a chance that being a part of YC adds just such a large amount of value that it still comes out in the data.

This sounds like a good instrumental-variable exercise for finding out whether the YC branding adds a significant amount of value to investors.

My understanding is that the person who asked that question wants to measure the value YC is adding, separate from YC's ability to select promising teams. Obviously there's no way to separate those things, so s/he suggested an absurd experimental setup in which half of the accepted startups, chosen at random, would be excluded from YC.

The control group would consist of the hypothetical startups that you didn't fund (but were qualitively equal to the ones you did fund).

EDIT: gabrielroth got it.

He's saying there should have been a control group. We can't directly compare startups that YC funded with startups of equal quality that YC did not fund, because the latter does not exist (assuming YC is correctly selecting for quality.) To isolate YC's influence, you'd need YC to pick a set twice as large and fund only half of them at random.