I think we need to look at the populations of these colleges not as solid blocks, but as distributions.
In the simplest case, we could plot the population based on objective talent vs. financial access.
And it's likely that with Stanford, you'll find some people with both high objective talent and medium financial access, and others with medium objective talent but high financial access. Even the scholarship recipients are going to need the freedom to ship off to college and pay for incidental expenses, rather than remain at home to support their families.
On the flip side, at less prestigious colleges, it might be true that in the aggregate the objective talent level is a bit lower, but there are going to be some students with objectively high talent who simply lack the financial access to a place like Stanford, even if scholarships were on the table.
From a recruiting standpoint, is it easier for a company like Google to just ignore students from those less prestigious colleges? Yes. But will it create a bias toward people with more wealth and privilege, and undermine the goal of diversity initiatives? Also yes.