The vast, vast, vast majority of people in that 55% are not even remotely poor. Most are of upper-middle income and of even higher socioeconomic status (e.g. progeny of diplomats, famous art gallery owners, and esteemed professors who make a merely upper-middle income but are higher in social status).
Yale spends nearly $100 million annually on financial aid. And this is up from roughly $32 million in 2001. This is not just a "token" expenditure, nor is it a token growth in spending.
It "spends" that money by giving aid packages that can only be redeemed by purchasing their extremely expensive product. Don't get me wrong; I think it's better for universities to price discriminate in this case than for them not to do so. However, it's not accurate to claim that they're bending over backward to provide equal access to the poor, as they're demonstrably not doing this.