Why does everything need to scale? Why couldn't they stick to taking a few high quality companies every year? There was enough money to be made that way. Seems the belief is that everything needs to scale until it doesn't work anymore.
Keep in mind, YC is very early stage. Asking them to pick only winners based on an application and short interview process would be a fool's errand. I believe their acceptance rates tend to be around 2%, so it's not like they aren't being selective up front—it just takes time to really get signal on who is executing and finding traction. Startup investment is high risk and justified by the outliers, so they have every incentive to find them early, it only makes sense that they would give more support to their winners as they emerge.
And knowing which companies are high quality is almost impossible at the early stages.
Because they don't actually have the ability to pick high-quality companies.
If you don’t eat the pie, someone else will. If someone else eats more pie than you, they will be stronger than you and eat your pie.
Switching analogies, unless you have a defensible niche with a wide moat, your position of power is an unstable equilibrium; in the open market only a monopoly is a stable attractor state.
The Peter Principle is pointing out that if you keep getting promoted by doing an exceptional job, and you stop getting promoted when you stop, that you have ended up leaving a job you were exceptional at, and landing in a job you are not good at. In a certain sense it's saying "you should be demoted back to your last job that you were exceptional at"; it's not suggesting that, it's suggesting a sort of snarky "this whole system is set up to fail" because it's from a "isn't capitalism grubby" perspective.
None of this flavor is in the general YC scaling case, though you may be able to find it in various places within that.