You're excluding brand dilution as a function of class size. Knowing you are going to be one of 100+ companies going through the program each semester makes you less enthusiastic to join if you already have a very promising or successful company, and it makes it harder for VCs to determine signal; "It's from YC" stops being good enough.
A big reason for the class size explosion is almost certainly for network effects, a fleet of ready customers who support YC and drive growth out of the gates.
It creates the space for another brilliant investor to launch a rival YC that caters to a small size of elite founders. Isn't that almost inevitable?